FAQ Database Discussion Community


find the lines where sequence/sorting is broken in linux

linux,sorting,awk,scripting
I have a log file which contains lot of logs and first column contains the epoch time stamp, Ideally they should be in sequence/sorted. however if something goes wrong in system timestamp resets to some default value and sequence breaks and restarts. I am trying to find the lines where...

How to create a new file where each new line contains some text and quoted line/step number?

shell,unix,command-line,awk
I want to use a loop while creating a new file where each new line is created by one/single step: some text, then quoted step number, and then something else. I started with this code, but it does not work: gawk ' { for (i = 1; i <= 1000;...

Wrapper for the matlab to read sparse data file

matlab,sorting,text,awk,libsvm
Libsvm can read the following datafile and convert it into sparse data structure in matlab (using libsvmread). -1 3:1 11:1 14:1 19:1 39:1 42:1 55:1 64:1 67:1 73:1 75:1 76:1 80:1 83:1 -1 3:1 6:1 17:1 27:1 35:1 40:1 57:1 63:1 69:1 73:1 74:1 76:1 81:1 103:1 First column is...

awk regex start of line anchor matches whitespace

regex,bash,awk
Parsing an input file through awk I ran into an issue with anchors in awk. Given the following file: 2015 2015 test test Output with awk $ awk '$1 ~ /^[0-9]/' file 2015 2015 Output with sed $ sed -n '/^[0-9]/p' file 2015 Can somebody explain the behaviour I'm seeing...

grep/sed replace no match with blank space

bash,awk,sed,grep
Example of a file I'm greping information in: name : server1 description : webserver memory : 32gb name : server2 memory : 128gb name : server3 description : appserver I'm doing something like this : cat myfile | egrep -w "name|description|memory" | awk -F" " '{print $3}' >> myfile2 In...

BASH - conditional sum of columns and rows in csv file

linux,bash,csv,awk
i have CSV file with some database benchmark results here is the example: Date;dbms;type;description;W;D;S;results;time;id Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;570;265;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;420;215;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;500;365;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;530;255;50 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;870;265;99 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;620;215;99...

netstat: parse (awk?) output to return “Program name” not “PID/Program name”

bash,ubuntu,awk,netstat
UBUNTU 14.04 netstat -p outputs both "PID/Program name" in the same column. I just want "Program name" in that column. What's the easiest way to do this? Actual Output [email protected]:~# netstat -tlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0...

How to filter data from flat file with multiply lines pattern using awk or sed tool?

regex,awk,sed
This is my first post on this site. I have probably not very easy problem with awk or sed language. In my file are data like this: A B C [Start]D E F [/End] G ... [Start]H I J [/End] ... K And I need following result: A B C...

Print the last 1,2,3..Nth or first 1,2,3…Nth matching block pattern using awk or sed

bash,awk,sed
pattern1 a b pattern2 cd pattern1 re pattern2 gh pattern1 ef pattern2 qw e I can show all matching pattern by sed -n '/pattern1/,/pattern2/p' Choose the second matching pattern or any Nth by awk -vM=2 '(x+=/pattern1/)==M&&x+=/pattern2/' file pattern1 re pattern2 Print only last matching pattern by awk 'x+=/pattern1|pattern2/{!y++&&B="";B=B?B"\n"$0:$0;x==2&&y=x=0}END{print B}' file...

Extracting columns within a range AWK

unix,awk
I've been trying to extract lines where a number in one columns falls within a particular range. Lets say I have a file that looks as so: chrom prediction chrom1 0.75 chrom2 0.5 chrom4 0.76 If I wanted to print lines where the prediction value was in the range from...

Repeating the format specifiers in awk

awk,printf,gawk
I am trying to format the output of the AWK's printf() function. More precisely, I am trying to print a matrix with very long rows and I would like to wrap them and continue on the next line. What I am trying to do is best illustrated using Fortran. Consider...

Extract lines from File2 already found File1

linux,text,awk,sed,grep
Using linux commandline, i need to output the lines from text file2 that are already found in file1. File1: C A G E B D H F File2: N I H J K M D L A Output: A D H Thanks!...

Test if pattern in file exists, evaluate if

bash,awk
I want to test if a pattern exists in a file. If the pattern exists (In a file among a list), do a replacement and update the corresponded file. If the pattern doesn't exists, do nothing. So far I have done: if [ -f $((`awk '/Pattern1/' $FILE`))];then sed 's/Pattern1/\nWORD/g' $FILE...

AWK count number of times a term appear with respect to other columns

linux,shell,command-line,awk,sed
Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

awk script to eliminate special characters

awk
My output looks like below judi# *diff -C 0 $* mountYday mountTday *** mountYday Sat Jun 13 02:57:09 2015 --- mountTday Sat Jun 13 02:59:49 2015 *************** *** 20 **** - /test on /dev/vx/dsk/test/test read/write/setuid/devices/intr/largefiles/logging/xattr/onerror=panic/dev=48986a5 on Wed Apr 22 22:28:04 2015 --- 19 ---- judi# I need to get only...

Calculate variance in bash

linux,bash,awk,statistics,variance
I want to compute the variance of an input txt file like this one: 1, 5 2, 5 3, 5 4, 10 And I want the output to be like: 1, 0 2, 0 3, 0 4, 4.6875 I've used this line: awk '{c[NR]=$2; s=s+c[NR]; avg= s / NR; var=var+(($2...

Indent and wrap consecutive matching lines with string

regex,bash,awk,sed,markdown
I would like to convert a predictably-formatted file containing code snippets into Markdown. The file looks like this: MY CODE SNIPPETS 2015-05-01 This file contains useful code snippets for every day usage in the Linux command line. SED sed 's/\(.*\)1/\12/g' # Modify anystring1 to anystring2 sed '/^ *#/d; /^ *$/d'...

searching through text file in terminal

linux,unix,awk,grep,less
Hi this might be a basic question for many, but it has however managed to eat a couple of hours of my time. I have large data file as an output from running a script. The file contains around 15 columns and around 100,000 rows. I wish to search through...

awk — adding a new delimiter to the default space delimiter

unix,awk
awk default delimiter space treats any amount of space between two fields as equivalent.. echo "1 2"|awk '{for (i=1;i<=NF;i++) print $i}' #which gives the result (two spaces between 1 and 2) 1 2 How can I add "=" to this existing delimiter? I have tried the following and that has...

Using AWK to parse fields with commas

regex,bash,shell,awk
Edited - TLDR: Using awk to parse fields that include commas. # original config file - confile1 $ cat confile1 list=( app1,"HOSTNAME - port - application name - alert1",99.0,99.0 app2,"HOSTNAME - port - application name - alert1",99.0,99.0 app3,"HOSTNAME - port - service name - alert2",99.0,99.0 web1,"URL - HOSTNAMES(01,02) - http://someurl.com/...

AWK if length statement append

if-statement,awk,append
I am trying to fix a problem of missing data in a csv file. The column, that should be null, is lacking a double field separator to maintain the csv structure. The example below illustrates what I mean: [email protected]@2/[email protected]@[email protected]@WAW [email protected]@2/[email protected]@[email protected]@JKDes [email protected]@2/[email protected]@[email protected] [email protected]@2/[email protected]@[email protected]@GCCon [email protected]@2/[email protected]@[email protected]@SMAs [email protected]@2/[email protected]@[email protected]@SMAs [email protected]@2/[email protected]@[email protected]@DKo As you can see, in...

Replying to Tweets Using t for the Command Line

ruby,twitter,command-line,awk
I'm using t for Ruby to tweet from Terminal. I want to be able to scan my timeline, find the tweet IDs then reply to them. The closest I've got is this command that doesn't work. The issue is that the reply command takes two arguments, an ID and a...

AWK: adding a new field from another file

linux,bash,loops,awk
I have a file that is delimited with a "@". It has recurring data that can be used to split the file in sections. In another file I have data that I would like to add as another column to the first file. The source of the data that is...

Linux awk merge two files

linux,awk,merge
I have below script to combine two files. awk -F"\t" ' {key = $1} !(key in result) {result[key] = $0; next;} { for (i=2; i <= NF; i++) result[key] = result[key] FS $i } END { PROCINFO["sorted_in"] = "@ind_str_asc" # if using GNU awk for (key in result) print result[key]...

How to print text in columns (delimiter = space, ignore specific column with spaces inside)?

bash,shell,awk
I would like to print an output text in 4 columns: SharedCacheMap 0xb89e9720 None \Device\HarddiskVolume1\Windows\System32\WWanAPI.dll ImageSectionObject 0xb89ea5f8 None \Device\HarddiskVolume1\Program Files\McAfee\Host Intrusion Prevention\Resource\0409\McTrayHipRL.dll DataSectionObject 0xb89ea5f8 None \Device\HarddiskVolume1\Program Files\McAfee\Host Intrusion Prevention\Resource\0409\McTrayHipRL.dll I tried: column -s " " -t I don't know how to handle the spaces in the file pathes. Thanks in...

How to use awk regex sort by query string value?

regex,linux,awk
I have a log file with example row: xxx.xxx.xxx.xxx - - [07/Jun/2015:14:18:39 +0000] "GET /file/?t=70 HTTP/1.1" 200 35 "http://1234.com/p/talk-about-owning-it/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome" The 7th column of each row looks like this: /file/?t=70 /file/?t=4785&k=1 /file/?t=120 /file/?t=95&k=0 /file/?t=120 /file/?t=120&k=0 /file/?t=95&k=1 ... Output is arranged according...

Rearranging a csv file

bash,shell,csv,awk,scripting
I have a file with contents similar to the below Boy,Football Boy,Football Boy,Football Boy,Squash Boy,Tennis Boy,Football Girl,Tennis Girl,Squash Girl,Tennis Girl,Tennis Boy,Football How can I use 'awk' or similar to rearrange this to the below: Football Tennis Squash Boy 5 1 1 Girl 0 3 1 I'm not even sure if...

Using blank-line delimited records and colon-separated fields in awk

awk,etl
I'd like to be able to work with a file in awk where records are separated by a blank line and each field consists of a name followed by a colon, some optional whitespace to be ignored/discarded, followed by a value. E.g. Name: Smith, John Age: 42 Name: Jones, Mary...

Sorting unique elements column-wise in a text-file

regex,linux,awk,sed,grep
I have a tab delimited file like chr1 4359314 4361314 Rp1 - chr1 4492735 4494735 Sox17 - chr1 4495330 4498354 Sox17,Sox17,Sox17,Sox17,Sox17,Sox17 -,-,-,-,-,- chr1 4784698 4786739 Mrpl15,Mrpl15,Mrpl15,Mrpl15 -,-,-,- chr1 4806788 4809237 Lypla1,Lypla1,Lypla1,RP24-426M1.3,Lypla1,Lypla1,Lypla1,Lypla1 +,+,+,+,+,+,+,+ chr1 4856814 4859038 Tcea1,Tcea1 +,+ chr1 5017735 5020539 Rgs20,Rgs20,Rgs20 -,-,- chr1 5069018 5071285 Atp6v1h,Rgs20,Rgs20 +,-,- chr1 5082080 5084154...

Suppressing system command called from awk script

windows,awk,system
I am currently running this script in Windows 7. So, I have a program that is meant to color-code output from another command (mkmk) and tally up varying numbers of errors and other notable stats, etc. So right now, it starts as a batch file which Turns off echo Sets...

Check for decimal point and add it at the end if its not there using awk/perl

regex,perl,shell,awk
I have test.dat file with values given below: 20150202,abc,,,,3625.300000,,,,,-5,,,,,,,,,,,,,,,,,,,,,, 20150202,def,,,,32.585,,,,,0,,,,,,,,,,,,,,,,,,,,,, 20150202,xyz,,,,12,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, My expected output is shown below: 20150202,abc,,,,3625.300000,,,,,-5.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,def,,,,32.585,,,,,0.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,xyz,,,,12.,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, ^. added here So if column 6 and 11 doesn't have decimal point in it, then we should add '.' at the end of...

delete lines not matching pattern

shell,awk
I have two files: file1: 123,hj,0 345,788,0 file 2: nmmnm,m kjlfkj jjljk(gjhgjh;id:123;) klkl,nkklmk kljlk kjkn(vnbvmn;nbvbn;id:678;jlk;); hgdsghhj (khj;id:345;) I wanted to delete lines from file2 if the number between id: and ; do not match with column1 in file1. Output: nmmnm,m kjlfkj jjljk(gjhgjh;id:123;) hgdsghhj (khj;id:345;) ...

AWK|BASH, use double FS and ternary operator

bash,awk
Is it possible? I was wondering how to do: Count fields differentiated by comma. Only the obtained first field of the previous step, count words differentiated by space. If there is more than 2 words, print NF, otherwise $0. Input cellular biol immunogenet, rosario escuela estadist, medellin medellin Expected output...

How to insert the missing months (with value NA) in a time series data file using AWK

awk
am new to awk and I have a data file goes from Jan 1958 to Dec 2014, but some data are missing, for example here is a portion of the file where couple months are missing: 2009 6 0.273 2009 7 0.000 2009 10 4.07 2009 11 8.25 and I...

Remove part of a column, if its in a specific Column number. (The column has a variable)

bash,awk,sed
I have a csv with lines like this: Last,First,A00XXXXXX,1492-01-10,2015-06-17,,Sentence Skills 104,,Elementary Algebra 38, Last,First,A00XXXXXX,1492-01-10,2015-06-17,,,,Elementary Algebra 101,College Level Math 56 Last,First,A00XXXXXX,1492-01-10,2015-06-17,Reading Comprehension 102,,,, Last,First,A00XXXXXX,1492-01-10,2015-06-17,,,,Elementary Algebra 118,College Level Math 97 I want to remove the word "Reading Comprehension" but leave the number, but only if its in column 6, if its in...

Change a Script to a For Do Done Loop

linux,bash,for-loop,awk
I have a script that I need to turn into a loop, the script works exactly as I need it to (My Awk-fu is extremely weak, so be nice) when I put in the file name like the example below. #!/bin/bash awk -v FS='(<LastName>|<\/LastName>)' '{print $2}' 17822624.xml >> test.csv awk...

Ignore first few lines and last few lines in a file Linux

linux,awk
I have a file like this and would like to print $0 except the first two and last three lines in linux. Tried awk command but no luck, is there any options I am using the following command - I suppose I am doing something wrong, but not able to...

How to split a CSV file into multiple files based on column value

bash,csv,awk
I have CSV file which could look like this: name1;1;11880 name2;1;260.483 name3;1;3355.82 name4;1;4179.48 name1;2;10740.4 name2;2;1868.69 name3;2;341.375 name4;2;4783.9 there could more or less rows and I need to split it into multiple .dat files each containing rows with the same value of the second column of this file. (Then I will...

Extracting Substring from File Name

string,awk,substring,extract,cut
I have a list of files with the following file name format: [some unknown amount of characters][_d][yyyymmdd][some unknown amount of characters] I want to extract the substring that contains the date (yyyymmdd) which I know will always be proceeded by "_d". So basically I want to extract the first 8...

Search A Variable in Specific Column in BASH with awk

linux,bash,awk
I have LOG_FILE like below: LOG|Server301|2015|05|12|00|58|35|572617|0|COMP|002.01.003.00|DSOHMSN2ViewCallback.cpp|142|CALLBACK:DSOHMSN2ViewCallback::FLIPBypassOn Changed SN=2 old value =2 new value =1 LOG|Client2|2015|05|12|00|58|35|593088|0|COMP|002.01.003.00|AMSN.cpp|12577|CALLBACK:OHMOAM::ohmSN2ViewChange, Old FLIPBypassOnMT LOG|Server302|2015|05|12|00|58|35|593116|0|COMP|002.01.003.00|AMSN.cpp|12590|CALLBACK:OHMOAM::ohmSN4ViewChange, New FLIPBypassOnMT...

Using a command-line utility to perform the following map-updates

shell,command-line,awk,terminal
I'm a complete newbie to using command-line utilities and am wondering how to process information as following: mapping.txt: 80 001 002 81 011 012 013 014 82 021 022 ... input.txt: 81 103823044 80 103823054 81 103823064 ... Desired output.txt: 103823044|011| 103823044|012| 103823044|013| 103823044|014| 103823054|001| 103823054|002| 103823064|011| 103823064|012| 103823064|013| 103823064|014|...

Bash, Using grep, sed or awk to extract section of text and then match

bash,awk,sed,grep
I have a text file and want to extract all interfaces matching "blue" random text random text random text random text random text int 1 random text blue random text random text int 2 random text random text red random text int 3 random text random text random text blue...

Replace a character in a section of a string in Bash

regex,bash,awk,sed
I'm trying to replace - and : with ? only in the middle(second) section delimited by ___ (3 underscore) input: aaa___bb-bb:bbb___cc-cc:ccc d-d___d-ddd:d-d___e-e:e output: aaa___bb?bb?bbb___cc-cc:ccc d-d___d?ddd?d?d___e-e:e I tried below sed command but it only replaces the last occurrence of the -: in the middle section echo "aaa___bb-bb:bbb___cc-cc:ccc d-d___d-ddd:d-d___e-e:e" | sed "s|\(___[^_]*\)[-:]\([^_]*___\)|\1?\2|g"...

How to match and change strings in a column of a semicolon separated file?

regex,awk,sed,gawk
I have a semicolon separated csv-file which looks like this: column1;column2;;123564;128;;IJL;value;;;;;3705;;;;;;;; column1;column2;;26789786413423;;CCE;value value;;;;;;3705;;;;;;;; column1;column2;;4564564;128;;SSE;value;;;;;;;;;;;;; column1;column2;;4645646;128;;JJY;someting X;;;;;;;;;;;;; column1;column2;;123132;128;;ASA;X value;;;;;;;;;;;;; column1;column2;;45643123;128;;TT;9 someting;;;;;;;;;;;;; column1;column2;;456464;128;;KK;VALUE 9 VALUE;;;;;;;;;;;;; column1;column2;;4646;128;;ST;value 6;;;;;;;;;;;;;...

Unique entry set in the first column of all csv files under directory [duplicate]

bash,shell,awk,tcsh
This question already has an answer here: Is there a way to 'uniq' by column? 6 answers I have a list of comma separated files under the directory. There are no headers, and unfortunately they are not even the same length for each row. I want to find the...

Split a column's values into multiple rows awk/unix/python

python,unix,awk
Here is a sample row of input data : header: id,indicator,{(pid,days_remaining)} row: id_558314,1,{(property_66021,7),(property_24444,1),(property_285395,6)} Expected output of data : header: id,indicator,pid,days_remaining row1: id_558314,1,property_66021,7 row2: id_558314,1,property_24444,1 row3: id_558314,1,property_285395,6 Is this something easily doable using awk/unix/python? Thanks in advance. Your time and help is appreciated....

How to remove from second occurrence until the end of the file?

bash,awk,sed
How to remove from a second occurrence of a pattern and delete only all the remaining occurrences in a file? What I have done: awk '/Type ;Rho ;DelSqRho ;Bond Ellipticity ;V ;G ;K ;L ;/{p++}p==1' file Pattern: "Type ;Rho ;DelSqRho ;Bond Ellipticity ;V ;G ;K ;L ;" Input Type ;Rho...

Replace the last six spaces with comma

bash,awk,sed
How would I replace the last six spaces with comma in a text file from each line with bash? I have: $cat myfile foo bar foo 6 1 3 23 1 20 foo bar 6 1 2 18 1 15 foo 5 5 0 15 1 21 What I want...

AWK - Search for a pattern-add it as a variable-search for next line that isn't a variable & print it + variable

regex,linux,awk,sed,gawk
I have a given file: application_1.pp application_2.pp #application_2_version => '1.0.0.1-r1', application_2_version => '1.0.0.2-r3', application_3.pp #application_3_version => '2.0.0.1-r4', application_3_version => '2.0.0.2-r7', application_4.pp application_5.pp #application_5_version => '3.0.0.1-r8', application_5_version => '3.0.0.2-r9', I would like to be able to read this file and search for the string ".pp" When that string is found, it...

Taking multiple header (rows matching condition) and convert into a column

bash,perl,command-line,awk,sed
Hello I have a file that has multiple Headers in it that I need to have turned into column values. The file looks like this: Day1 1,Smith,London 2,Bruce,Seattle 5,Will,Dallas Day2 1,Mike,Frisco 4,James,LA I would like the file to end up looking like this: Day1,1,Smith,London Day1,2,Bruce,Seattle Day1,5,Will,Dallas Day2,1,Mike,Frisco Day2,4,James,LA The file...

awk if statement with simple math

unix,awk,gawk
I'm just trying to do some basic calculations on a CSV file. Data: 31590,Foo,70 28327,Bar,291 25155,Baz,583 24179,Food,694 28670,Spaz,67 22190,bawk,4431 29584,alfred,142 27698,brian,379 24372,peter,22 25064,weinberger,8 Here's my simple awk script: #!/usr/local/bin/gawk -f BEGIN { FPAT="([^,]*)|(\"[^\"]+\")"; OFS=","; OFMT="%.2f"; } NR > 1 END { if ($3>1336) $4=$3*0.03; if ($3<1336) $4=$3*0.05;}1** Wrong output: 31590,Foo,70...

Evaluating condition of if statement in awk using a second file

bash,awk
I am trying to write a bash script that manipulates two files: file1 Region Coords. RsId Position Alleles Disease PValue OddsRatio RegionID 1p13.2 1:113839149..114551845 rs2476601 114377568 G>A Alopecia Areata 8.90E-08 1.34 869 2q13 2:111444884..111809030 rs3789129 111698040 A>C Alopecia Areata 1.50E-08 0.76 871 2q33.2 2:204611195..204817281 rs3096851 204763882 A>C Alopecia Areata 3.58E-08...

using awk for subtraction

awk,gawk
Similar post here: awk if statement with simple math Below is great but now I need to subtract 20 from field $4 if it's less than 20 and if its greater than 20, field five can be set to 0. 31590,Foo,70,3.5 28327,Bar,291,14.55 25155,Baz,583,29.15 24179,Food,694,34.7 28670,Spaz,67,3.35 22190,bawk,4431,132.93 29584,alfred,142,7.1 27698,brian,379,18.95 24372,peter,22,1.1 25064,weinberger,8,.04...

Compare two files and append the values, leave the mismatches as such in the output file

awk,compare,match,mismatch
I'm trying to match two files,file1.txt(50,000 lines), file2.txt(55,000 lines). I want to campare file2 to file 1 extract the values of column 2 and 3 and leave the mismatches as such. Output file must contain all the ids from file2 i.e., it should have 55000 lines. Note: All the ids...

How to remove duplicate comma separate strings using awk

bash,csv,awk
I have a csv file like this: (named test2.csv) lastname,firstname,83494989,1997-05-20,2015-05-07 15:30:43,Sentence Skills 104,Sentence Skills 104,Elementary Algebra 38,Elementary Algebra 38,Sentence Skills 104,Sentence Skills 104,Elementary Algebra 38,Elementary Algebra 38, I want to remove the duplicate entries The closest I have got is the following awk command awk '{a[$0]++} END {for (i in...

Add quotes to first line of CSV

bash,csv,awk
I wish to modify the first row of a CSV (comma-separated file) which contains headings by adding quotes. It's a very large file, I was trying to use AWK for this problem. Before: id,var1,var2 1,0.11,0.44 2,0.22,0.55 3,0.33,0.66 After: "id","var1","var2" 1,0.11,0.44 2,0.22,0.55 3,0.33,0.66 Something like this, which adds quotes for all...

find numbers divisible by 3 in csv file using shell script

bash,shell,unix,awk
I have csv file having content like below : 1|2|3 4|5|6 7|8|9 Now I would like to find the numbers which are divisible by 3 using shell scripting. I would like to use awk command for this. I am learning shell scripting. So could you please help me out to...

how to deletes line from a text file that are taken from another file [duplicate]

shell,awk,sed,grep,sh
This question already has an answer here: Remove duplicates from text file based on second text file 4 answers I have a data.txt file with a lot of lines in it and a lines.txt that contains some lines. I want to delete all lines from data.txt that match any...

Delete the extra space after special character in all the lines of text file

bash,shell,text,awk,sed
Following data represents 2 lines of text file: 1 2340: 2 1930: 1 1 9: 3 4501: 1 45: 1 5620: 2 I want to delete the space after ":". So the output of above text file should be 1 2340:2 1930:1 1 9:3 4501:1 45:1 5620:2 ...

Selecting unique lines based on two columns

unix,awk
I've been trying to figure out a way to select lines that are unique based on the values of two columns. For example, here is a sample/example of my file: chr1 10 12 chr1 10 12 chr1 10 11 chr1 9 12 chr2 15 20 And this is what I...

How to convert following text in required format in unix?

unix,awk
I have a file which has the following format: 1 2 3 4 5 6 I am trying to make it like: 1 2 3 4 5 6 I have tried awk 'BEGIN { ORS= " "} {print}' test.txt It makes 1 2 3 4 5 6. But I need...

Compare strings using awk command

bash,shell,awk
I am at initial stage of learning shell scripting. So please explain me the steps for better understanding. Consider I have two files Content of the two files are as below: File1.txt ABC=10 DEF=20 XYZ=30 File2.txt DEF=15 XYZ=20 I want to write a simple shell script to check both the...

Convert AWK command to sqlite query

sql,awk,sqlite3
I have to parse very big (abt 40Gb) text files(logs) very often. Usually AWK/grep is enough for my needs, but logs are growing and now I'm curious whether sqlite3 will allow me to do the same things more efficiently. I chosen sqlite for my tests as it installed out-of-the-box (SQLite...

Replacing ascii with matching characters using perl / awk or other [on hold]

regex,linux,perl,awk
I have an access log that was written with nginx and lua code. It is url encoded and those some characters are written in the format of \xHexCode (for example, double quotes are written as \x22). I would like to run awk or perl or other fast script to replace...

Replace the alphanumeric, numeric values with empty space [closed]

linux,shell,awk
File looks like this: 1234567890|asdfg|werty|asfgtr21|12345|zxcv=sdf|zsde=1234 2345678901|we1|sd1=12df|werty|dfghj|sdf=fgh|hjkl And the required output is: 1234567890|asdfg|werty|||zxcv=sdf| 2345678901|||werty|dfghj|sdf=fgh|hjkl ...

Using sed, awk or grep to split a string with numbers [duplicate]

regex,bash,shell,awk,sed
This question already has an answer here: Learning Regular Expressions [closed] 1 answer I have this file: mmD_154Lbb_e_dxk_83233.orc 154L_bbe_Bddxk_3259.txt 14Lbe_3233.orc m2_154Lbbe_dxk_67233.op mZZ_1A4Lbbe_dxk_32823.op mmD_154Lbbe_dxk_99333.orc mmD_oS154be_dxk_12338.txt I'm trying to use sed or awk to split the numbers and I don't have solution: I need out put: 83233 2597 3233 67233 32823...

Detecting corrupt characters in UTF-8 encoded text file

regex,encoding,awk,utf-8,scripting
I have a text file that was edited with the wrong character encoding and thus has some mojibake and corrupt characters in some of the strings when I open it using UTF-8. What scripting language would be the most efficient at detecting these corrupt characters? Perl is not an option....

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

linux,bash,awk,sed,sh
I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

Store String var in awk command

bash,variables,awk
I have txt file like this: Dupont Charles Martin Paul Dupuis Jean I want, for each line, to make a login corresponding to first 2 caracters of each names. For instance : Dupont Charles ==> duch awk '{print tolower(substr($1,1,1)substr($2,1,1))}' liste.txt works perfectly. but i want to store each login in...

How to find average and maximum in an interval using Shell [closed]

linux,bash,shell,unix,awk
I would like to extract sum, mean and average in each 6 numbers interval from a column. I found many discussions related to this problem, but all those are for whole column. e.g. To compute sum of a column: awk '{sum+=$1} END { print sum}' To calculate Average: awk '{sum+=$1}...

awk not capturing first line / separator

bash,awk
I don't understand the following behaviour: This is a text file: example.txt 12345 4321 hello hello this is a test blobb 14324 2131 another test , incoming ! blubb 52341 1231 last test now shutting down bla ... It consists of x rows of text, 4 tab-separated columns each. I...

convert first column from hex to decimal using awk

linux,shell,awk,sed,hex
I have a log file which contains logs time stamp in Hexadecimal in column one and then log text in rest of the columns.For example: B60C0056: aaa1 bb1 ccc1 ddd1 eee1 fff B60C0056: aaa2 bb2 ccc2 ddd2 eee2 fff B60C0057: aaa3 bb3 ccc3 ddd3 eee3 fff B60C0058: aaa4 bb4 ccc4...

Finding columns with only white space in a text file and replace them with a unique separator

regex,r,bash,awk,sed
I have a file like this: aaa b b ccc 345 ddd fgt f u 3456 e r der der 5 674 As you can see the only way that we can separate the columns is by finding columns that have only one or more spaces. How can we identify...

remove line from file if more than one pattern appears in different line

shell,awk,sed,file-processing
I have a file with a patter like this: 1 1 1 2 0 1 0.5 1 2 2 2 0 2 0.5 2 1 1 1 0 1 0.25 2 1 2 2 0 2 0.5 2 3 3 3 0 3 0.25 I want to remove a line...

Linux - Read a record till to the end with awk

linux,awk
lets say i have this text from the logfile: Jun 10 11:09:07 mylinux daemon.notice openvpn[3710]: TCPv4_CLIENT link remote: 1.22.333.444:1111 But i don't need the part between "mylinux" and the next colon: Thats the part i try to remove: daemon.notice openvpn[3710] I "solved" it with awk, but thats not a good...

bash: read each line from a file and use as a regular expression to match and print column awk

regex,bash,awk
I would like to use each line of a file, samples.txt, as a regular expression and print the entire column that matches this from input.txt. samples.txt aa bb cc input.txt s aa v dd jj bb ww cc 1 1 1 1 2 3 3 8 3 5 4 5...

awk select columns from input list

parsing,csv,awk
I'd like an awk script to select the columns from a file based on a list of columns in another file. For example: $cat cols 3 2 6 4 $cat text a b c d e f g h i j k l m n $awk_script cols text c b...

awk, if else statment, assigning conditions to arrays

awk
If I have a file where in the first column is a set of words: sad dog hand paper computer rabbit cow and I want to get the word length of the column, check if it's even or odd, and if even assign it to an array and if odd,...

Match repetitive data from file with second column from second file

arrays,bash,awk,sed
Hello I have a file with repetitive data as such: ENGLAND ENGLAND ENGLAND JAPAN JAPAN JAPAN JAPAN AMERICA AMERICA AMERICA And a second file with unique data that has two columns(separated by "=" ), with the first column being considered a key: ENGLAND=LONDON JAPAN=TOKYO AMERICA=WASHINGTON DC AUSTRALIA=SYDNEY IRELAND=DUBLIN I am...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters

bash,awk,sed
How can I replace [a-z],[a-z] with [a-z], [a-z] and keeping the letters? Input suny stony brook, stony brook,usa. Output suny stony brook, stony brook, usa. What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." sed 's/[a-z],[a-z]/, /g' <<< "suny stony brook, stony brook,usa." ...

Access a bash array in awk loop

arrays,bash,awk
I have a bash array like myarray = (1 2 3 4 5 ... n) Also I am reading a file with an input of only one line for example: 1 2 3 4 5 ... n I am reading it line by line into an array and printing it...

Bash modify CSV to change a field

linux,bash,awk
I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

AWK write to new column base on if else of other column

linux,bash,shell,awk,sed
I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Find the maximum values in 2nd column for each distinct values in 1st column using Linux

linux,bash,unix,awk
I have two columns as follows ifile.dat 1 10 3 34 1 4 3 32 5 3 2 2 4 20 3 13 4 50 1 40 2 20 What I look for is to find the maximum values in 2nd column for each 1,2,3,4,5 in 1st column. ofile.dat 1...

exclude hash sign with awk regexp

regex,awk
I have a file with the following: #sometext=0 sometext=1 #sometext=5 sometext=34 and can't get sometext=1 sometext=34 I have tried: for (i=1; i<=4; i++){ if ($i ~ /[^#]sometext=/){ print $i } } but this is not working...

Insert variable in specific place using sed [closed]

variables,awk,sed
Please help me i'm trying to insert variable in a file with sed but has not yet successfully(( I have variable $time and i want insert it in index.html file in specific string(line number 53): the time: <span id="$time"></span> Please help me to solve it! Thank you!...

awk ternay operator, count fs with ,

awk
How to make this command line: awk -F "," '{NF>0?$NF:$0}' to print the last field of a line if NF>0, otherwise print the whole line? Working data bogota dept math, bogota ...

split file into columns with awk

bash,file,awk,multiple-columns
I have a file which looks like this: 1. result = 1.2.3.4 (1.2.3.4) info: [Affected] 2. result = www.addr.com (2.3.4.5) info: [not Affected] And now I want to split it in three columns, example: 1.2.3.4 1.2.3.4 Affected www.addr.de 2.3.4.5 not Affected I am using awk for that: cat filename.txt |...

if a string exist (including a variable), flip it using awk or sed

bash,awk,sed
I have a csv file like this: ,College Level Math 55,Elementary Algebra 112 ,Elementary Algebra 79, ,College Level Math 102,Elementary Algebra 54 ,,College Level Math 54 I need an awk or sed command that does the following if College Level Math *,Elementary Alegrbra * exist flip it so it looks...

how to break one column into multiple columns in Linux [duplicate]

linux,bash,shell,unix,awk
This question already has an answer here: split a column in multiple columns 3 answers I want to break a column into different rows. e.g., test.dat 1 4 3 4 6 2 4 6 8 What I want is to break it into a 3 x 3 matrix out.dat...

Bash Scripting - AWK with Delimiter?

bash,unix,awk
I've got a header in a text file that looks like this: First-Name:Last-Name:City:Home-Phone:Cell-Phone What I need to do is display the Last-Name and the Cell-Phone number of each entry. I'm new to bash scripting, and I'm not really sure what I can use to do this. I was told awk...

Awk Script to process data from a trace file

table,awk,rows,trace
I have a table (.tr file) with different rows (events). **Event** **Time** **PacketLength** PacketId sent 1 100 1 dropped 2 100 1 sent 3 100 2 sent 4.5 100 3 dropped 5 100 2 sent 6 100 4 sent 7 100 5 sent 8 100 6 sent 10 100 7...

Grep specific part of string from another file

awk,grep
I want to grep the first three digits of numbers in 1.txt from the first three digits after zeros in 2.txt. cat 1.txt 23456 12345 6789 cat 2.txt 20000023485 xxx888 20000012356 xxx888 20000067234 xxx234 Expected output 20000023485 xxx888 20000012356 xxx888 ...

awk | Extract Log on the basis of Time

awk
I have server log in the following format and I want awk which can extract logs between 2 dates. Log Format: 00:00:00,002 INFO [LOG.XXX] XXX 01:11:00,001 INFO [LOG.XXX] XXX 02:00:01,002 INFO [LOG.XXX] XXX SOME JUNK 02:02:00,002 INFO [LOG.XXX] XXX 03:11:00,001 INFO [LOG.XXX] XXX SOME JUNK 03:00:00,002 INFO [LOG.XXX] XXX 04:00:00,001...

how to remove two lines above after we find a match in a file in BASH?

bash,awk,sed,grep
I have a file that contain data like: 123 456 789 I want to delete the above two lines (123 and 456) when I find a match 789. Is it possible to do it with sed or awk? Please Help...

How to remove characters before and including an underscore?

linux,string,bash,unix,awk
In bash scripting what's an efficient way to do the following please? var="fooo_barrrr" What is the best way to remove all characters before and including the '_' so that var becomes "barrrr" please?...

awk to print all colums formatted

awk,pattern-matching
I want to format below un-formatted inputs. I would like to print all the fields if the second column begins with "2". Actual file contains many columns. Is there any easiest way to print all columns instead of typing print $2,$3,$4,$5,...$n? Input.csv --------------------------- |Data statistics|Number of| |-------------------------| |Records passed |...

how to convert date using awk linux

date,awk,converter
I have txt file (test.txt) contains the following data: 00:06:55.351;C39EE9AF;2015-01-05 09:52:20;0;5;295;test; 00:06:51.368;C39EE78B;2015-01-05 09:52:17;0;5;295;test2; I need to view date as below using awk, like this: 2015-01-05 test 2015-01-05 test2 I have tried the following: cat test.txt | awk -v FS=";" '{print $3,$7}' ...

Finding the average of a column excluding certain rows using AWK

linux,bash,awk,scripting
I want to calculate the average of the 5th column (last column) excluding the rows with the value "9999". Would appreciate your feedback. 77.300 16 1 3.6112914285714268 9.4 77.300 16 2 -0.001737142857145102 20.0 77.300 16 3 5.1570742857142857 8.9 77.300 17 0 3.6112914285714268 8.9 77.300 17 1 2.9484342857142849 11.7 77.300 17...

loop add a comma after nth comma using awk

bash,awk
I feel like this should be a fairly straight forward question, but I cant seem to get it to work. I have a csv file and I need to add comma after the nth comma in each row. I believe I have to use gsub to get it to loop....