regex,awk,sed , How to filter data from flat file with multiply lines pattern using awk or sed tool?

How to filter data from flat file with multiply lines pattern using awk or sed tool?


Tag: regex,awk,sed

This is my first post on this site. I have probably not very easy problem with awk or sed language. In my file are data like this:


And I need following result:

[Open]D E F[/Close]
[Open]H I J[/Close]

For now I have not working awk code:


/[\/End]/ {
    if(step==3) print "[/Close]"

step==2 {


/[Start]/ {
    print "[Begin]"


Many thanks for yours answers. I hope to stay here a little bit longer. Cheers! P.


This awk will do most of it, but will leave space before the [\Close]

awk '/Start/{ORS=FS} /End/{ORS=RS} sub(/Start/,"Open") sub(/End/,"Close") 1' file

It's easy to trim that in another pass (pipe previous output to this script)

awk 'sub(/ \[/,"\[") 1'


Validate part of mail suffix

I'm validating mail addresses that could look something like this: [email protected] but also [email protected] I would like to check whether if xx is present or not. The first thing that came to mind was String.Contains, but it would obviously match any occurrences of xx in the string. Is regexp the...

AWK write to new column base on if else of other column

I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Regex not working in HTML5 pattern

So I have this regex intended to let pass all text but those that contain as initial chars the "34" sequence: ^(?!34)(?=([\w]+)) The regex is working fine for me in , check the tests to see the intended behavior. Any Idea why it isn't working in my form? <form>...

Store regex pattern as a string in PHP when regex pattern contains both single and double quotes

UPDATE: It turned out that what broke my reg expressions after escaping the quotes was the / deliminator terminating the expression early. I'm attempting to store a regex pattern as a string like so: $link_pattern = "/<a\s+href=(?:"([^"]+)"|'([^']+)').*?>(.*?)</a>/"; However I'm unable to properly encapsulate it since the regex pattern contains both...

Please can someone help me understand the exec method for regular expressions?

The best place I have found for the exec method is Eloquent Javascript Chapter 9: "Regular expressions also have an exec (execute) method that will return null if no match was found and return an object with information about the match otherwise. An object returned from exec has an index...

Regex pass dynamic values with boundry

I'm trying to pass a dynamic value at runtime with a boundary \b to a Regex function. My code is: static void Main(string[] args) { string sent = "Accelerometer, gyro, proximity, compass, barometer, gesture, heart rate"; string match = "gyro"; string bound = @"\b"; if (Regex.IsMatch(sent, @"\bgyro", RegexOptions.IgnoreCase)) { Console.WriteLine("match...

How to write RegEx for inserting line break for line length more than 30 characters?

I am using a text editor which lets use regular expression to find / replace text. I have a large text file. I want to insert new line in each lines which are more than 30 characters. I want the line to break after 30th character (doesnt matter if a...

I need to make sure that only certain characters are in a list?

I have this to get input and put it in a list: def start(): move_order=[raw_input("Enter your moves: ").split()] And I only want the characters A, D, S, C, H (it's for a game >_>) to be allowed. I've tried using the regular expressions stuff: if re.match('[ADSCH]+', [move_order]) is False: print...

Identify that a string could be a datetime object

If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

BeautifulSoup: Parsing bad Wordpress HTML

So I need to scrape some a site using Python but the problem is that the markup is random, unstructured, and proving hard to work with. For example <p style='font-size: 24px;'> <strong>Title A</strong> </p> <p> <strong> First Subtitle of Title A </strong> "Text for first subtitle" </p> Then it will...

Matching string inside file and returning result

I've got a few peculiar issues with trying to search for a string inside of a .db file. The way I tried was by using grep, which does apparently find the string(s), although this is the output: $ grep "ext" *.db Binary file enormous.db matches There are a couple problems...

Python match whole file name, not just extension

Objective: Use Python regex to return a list of filenames have certain file extensions. List of files: X8SIA9.ROM X8SIA0.rom X8SIA0.928 Regex to find files that end in *.ROM or 3x numbers: bios_file = re.findall(r'.*\.(rom|[0-9]{3})+', name, re.I) Issue: bios_file is returning ['ROM'], ['rom'], [928]. bios_file should be returning the entire filename...

Regex to remove `.` from a sub-string enclosed in square brackets

I have this regex in C#: \[.+?\] This regex extracts the sub-strings enclosed between square brackets. But before doing that I want to remove . inside these sub-strings. For example, the string hello,[how are yo.u?]There are [300.2] billion stars in [Milkyw.?ay]. should become hello,[how are you?]There are [3002] billion stars...

MySQL substring match using regular expression; substring contain 'man' not 'woman'

I have an issue while I fetch data from database using regular expression. While I search for 'man' in tags it returns tags contains 'woman' too; because its substring. SELECT '#hellowomanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 correct, it contains 'woman' SELECT '#helloowmanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 incorrect, it can...

Ignore first few lines and last few lines in a file Linux

I have a file like this and would like to print $0 except the first two and last three lines in linux. Tried awk command but no luck, is there any options I am using the following command - I suppose I am doing something wrong, but not able to...

Reg ex matching a word

I need to match only first two files, out of four files listed below: ABD_DEF_GHIJ_20150611 ABD_DEF_GHIJ ABD_DEF_GHIJ_FX_20150611 ABD_DEF_GHIJ_FX I am using reg ex - ABD_DEF_GHIJ(_\d{8}|\b) and it's working fine. I would like to know if my solution is ok or there is any better alternate solution....

Convert AWK command to sqlite query

I have to parse very big (abt 40Gb) text files(logs) very often. Usually AWK/grep is enough for my needs, but logs are growing and now I'm curious whether sqlite3 will allow me to do the same things more efficiently. I chosen sqlite for my tests as it installed out-of-the-box (SQLite...

Split by a comma that is not inside parentheses, skipping anything inside them

I know it might be another topic about regexes, but despite I searched it, I couldn't get the clear answer. So here is my problem- I have a string like this: {1,2,{3,{4},5},{5,6}} I'm removing the most outside parentheses (they are there from input, and I don't need them), so now...

Pharo punctuation marks [duplicate]

This question already has an answer here: Learning Regular Expressions [closed] 1 answer Is there any way to capture all the places in a String, where there are punctuation marks like '.,:;!?' Basically, am looking for a regex that captures all the above....

Regex that allow void fractional part of number

@"[+-]?\d+(\.\d+)?" -this is a regex I have wrote for numbers it allows [+-] minus before the number digits before and digits after the point the question is how to change this to allow "not finished" values so that input of "5." - is fine too ?...

Ruby gsub group parameters do not work when preceded by escaped slashes

I am trying to perform a trivial substitution, that in any other language I have come across, work as per the documentation. However, my substitution fails for some reason. The documentation examples list: "hello".gsub(/[aeiou]/, '*') #=> "h*ll*" "hello".gsub(/([aeiou])/, '<\1>') #=> "h<e>ll<o>" "hello".gsub(/./) {|s| s.ord.to_s + ' '} #=> "104 101...

Warning: preg_match_all(): Unknown modifier '\' [duplicate]

This question already has an answer here: PHP - preg_match and “Unknown modifier” error 3 answers I know there are tons of duplicate question like this. But I am stuck here. Can anyone help me out? Here's the code: <?php function count_images($string){ preg_match_all('/[^/\s]+/\S+\.(jpg|png|gif)/', $string, $results); return count($results[1]); } ?>...

How many characters are visible like a space, but are not space characters?

If I want to discover the hexadecimal equivalent of a space in PHP I can play with bin2hex: php > echo var_dump(bin2hex(" ")); string(2) "20" I can also obtain space character from "20" php > echo var_dump(hex2bin("20")); string(1) " " But there exist Unicode versions of a "visible" space: php...

How to Match a string with the format: “20959WC-01” in php?

i want to restrict a user to enter a value which is similar to the value "20959WC-01", means it must contains 5 integers followed by two character, a '-' and two integers, can anyone please give me a solution to sort out this problem. Thanks in advance :) ...

Java - Enforce TextField Format - UX - 00:00:00;00

Please see below image of the UI of my current application Currently, I am enforcing Data Format of the text fields in the back end. If the RegEx does match (00:00:00;00), it will throw an error. My question is from a UX perspective, is it possible to make the input...

Patterns (preferably java) find floor space in sq ft

I have the following data that needs parsing. The pattern could be (1) approx 1,000 sq ft (2)c. 500sqft (3) 2,100 sq ft This is my code to find a digit but I need the above... in java Pattern regex = Pattern.compile("\\d[\\d,\\.]+"); Matcher finder = regex.matcher(price); if(finder.find()){ try { String...

Match a pattern preceded by a specific pattern without using a lookbehind

Is there a way to match a B only if preceded by an A? The A can be at any position behind the B, with any amount of characters between. Examples: A_B (Matches `B`) C_B (No match) I've tried: (?=A)[^B]*B But it matches all the characters preceeding B as well....

match line break except line begin with spcific word or blank line

If I have text that the line breaks is broken: Chapter 1 Lorem ipsum dolor sit amet, consectetur adipisci ng elit, sed do eiusmod tempor incididunt ut la bore et dolore magna aliqua. Ut enim ad minim ve niam, quis nostrud exercitation ullamco labo ris nisi ut aliquip ex ea...

Regex with whitespaces and preceding zeros

I want to match the string 11 with a regular Expression in SAS. The 11 can be preceded by zero or more 0 and/or by white spaces. Any other character is not allowed. Likewise, if anything there should only be white spaces following the 11. Examples: Match: 0000011 11 11<space><space>...

Bash modify CSV to change a field

I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

Finding embeded xpaths in a String

I have a string where I have the user should be able to specify xpaths that will be evaluated at runtime. I was thinking about having a the following way to specify it. String = "Hi my name is (/message/user) how can i help you with (/message/message) "; How can...

Swing regular expression for phone number validation

I want to validate phone number field in swing, so I am writing code to allow user to enter only digits, comma, spaces. For this I am using regular expression, when user enter characters or other than the pattern text field will consume. My code is not working. Can anyone...

RegExp to check if file is image

Is there a way to know if the file extension is an image? i got this. image/png Already try with var imageReg = /\.(gif|jpg|jpeg|tiff|png)$/i; string = "image/png" imageReg.test(string) But this return false;...

Regular Expression for whole world

First of all, I use C# 4.0 to parse the code of a VB6 application. I have some old VB6 code and about 500+ copies of it. And I use a regular expression to grab all kinds of global variables from the code. The code is described as "Yuck" and...

AWK count number of times a term appear with respect to other columns

Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

PHP Regular Expressions Counting starting consonants in a string

I need to find out how many starting consonants a word has. The number is used later in the program. The code below does work, I am wondering if it is possible to do this with a regular expression. $mystring ="SomeStringExample"; $mystring2 =("bcdfghjklmnpqrstvwxyzABCDFGHJKLMNPQRSTWVXYZ"); $var = strspn($mystring, $mystring2); Using a regular...

javascript replace dot (not period) character

I have some text being rendered that has an inline list with the bullet points notated by the "•" character, all on one line of text. It's ugly, but I don't want to change the source data. So I thought I'd just do a quick javascript replace to insert breaks...

Get number from string

I am trying to get the enclosed number between two slashes in a URL using regex. The code regex I have is not working, I am fairly new to regex and don't really understand it. The regex: http:\/\/?www\.?example\.com\/g\/(^\d$)\/\w The URL: Trying to get the "1337", which is the PlaceId....

like and regexp_like

For like we have %. for ex: if we give ad% it ll fetch all the records which starts with ad but i should use regexp_like. for regexp_like what can be used so that it acts as % for like. i cant use ^ad because from UI we ll give...

How to match words in 2 list against another string of words without sub-string matching in Python?

I have 2 lists with keywords in them: slangNames = [Vikes, Demmies, D, MS Contin] riskNames = [enough, pop, final, stress, trade] i also have a dictionary called overallDict, that contains tweets. The key value pairs are {ID: Tweet text) For eg: {1:"Vikes is not enough for me", 2:"Demmies is...

jQuery / Regex: How to compare string against several substrings

I have a special validation need where I need to check if a string contains ANY out of several substrings like the following list: 12 23 34 45 56 67 78 89 90 01 I am new to jQuery and the only thing I could think of here is the...

Get all prices with $ from string into an array in Javascript

var string = 'Our Prices are $355.00 and $550, down form $999.00'; How can I get those 3 prices into an array?...

Does there exist an algorithm for iterating through all strings that conform to a particular regex?

I'm making a script to try and hack into an account whose login password is at least 8 characters long and includes at least 1 number, 1 special character and 1 capital letter. I will use brute force. Is there a compact, elegant and efficient way to iterate through every...

Python regular expression, matching the last word

I've the following problem. I'm looking to find all words in a string that typically looks like so HelloWorldToYou Notice, each word is capitalized as a start followed by the next word and so on. I'm looking to create a list of words from it. So the final expected output...

How to create the javascript regular expression for number with some special symbols

what can be the java-script regular expression which gives the numbers with some symbols For example following condition must be pass. Number can start with $ Can have the . or , : symbols between and % sign at the send. Passing valus: $233 48.23% 278 22.33 45:23 10,000 Number...

Extracting strings from HTML with Python wont work with regex or BeautifulSoup

Im using Python 2.7, BeautifulSoup4, regex, and requests on windows 7. I've scraped some code from a website and I am having problems parsing and extracting the bits I want and storing them in a dictionary. What I'm after is text that is presented as follows in the code: @CAD_DTA\">I...

REGEX python find previous string

I'm trying to find if the last word of the string is followed by a space or a special char, and if yes return the string without this space/special char For example : "do you love dogs ?" ==> return "do you love dogs" "i love my dog " (space...

How do I isolate the text between 2 delimiters on the left and 7 delimiters on the right in Python?

I have a string: string = ""7807161604","Sat Jan 16 00:00:57 +0000 2010","Global focus begins tonight. Pretty interested to hear more about it.","Madison Alabama","al","17428434","81","51","Sun Nov 16 21:46:24 +0000 2008","243" I only want the text: "Global focus begins tonight. Pretty interested to hear more about it."" which is between the 2nd and...

Remove plus sign from url using .htaccess

My current url is: http://localhost/mywbsite/search/one+piece/149 I want to change link to like this http://localhost/mywbsite/search/one-piece/149 I have already .htaccess file with this code Options -MultiViews RewriteEngine On RewriteBase /indianrealitybytes/ RewriteCond %{THE_REQUEST} /search_advance\.php\?keywords=([^&]+)&f=([^\s&]+) [NC] RewriteRule ^ search/%1/%2? [R=301,L] RewriteRule ^search/([^/]+)/([^/]+)/?$ search_advance.php?keywords=$1&f=$2 [QSA,L,NC] I don't know where to change to remove plus(+) sign...

regex - Match filename with or without extension

Need a regex pattern to match all of the following: hello hello. I tried \b\w+\.?\w+?\b, but this doesn't match "hello." (the second string mentioned above)....