unix,awk , awk — adding a new delimiter to the default space delimiter

awk — adding a new delimiter to the default space delimiter


Tag: unix,awk

awk default delimiter space treats any amount of space between two fields as equivalent..

echo "1  2"|awk '{for (i=1;i<=NF;i++) print $i}'
#which gives the result (two spaces between 1 and 2)

How can I add "=" to this existing delimiter? I have tried the following and that has started to consider "single" space character as a delimiter and spoiled the above result.

 echo "1  2"|awk -F"[ |=]" '{for (i=1;i<=NF;i++) print $i}'
    #which gives the result


How can I give any amount of space as a delimiter here? Thanks in advance.


You can specify a regular expression as the delimiter:

 echo "1  2"|awk -F"[ |=]+" '{for (i=1;i<=NF;i++) print $i}'

It also means

 echo "1  2 3==5"|awk -F"[ |=]+" '{for (i=1;i<=NF;i++) print $i}'

would print



Suppressing system command called from awk script

I am currently running this script in Windows 7. So, I have a program that is meant to color-code output from another command (mkmk) and tally up varying numbers of errors and other notable stats, etc. So right now, it starts as a batch file which Turns off echo Sets...

Ignore first few lines and last few lines in a file Linux

I have a file like this and would like to print $0 except the first two and last three lines in linux. Tried awk command but no luck, is there any options I am using the following command - I suppose I am doing something wrong, but not able to...

Taking multiple header (rows matching condition) and convert into a column

Hello I have a file that has multiple Headers in it that I need to have turned into column values. The file looks like this: Day1 1,Smith,London 2,Bruce,Seattle 5,Will,Dallas Day2 1,Mike,Frisco 4,James,LA I would like the file to end up looking like this: Day1,1,Smith,London Day1,2,Bruce,Seattle Day1,5,Will,Dallas Day2,1,Mike,Frisco Day2,4,James,LA The file...

Check for decimal point and add it at the end if its not there using awk/perl

I have test.dat file with values given below: 20150202,abc,,,,3625.300000,,,,,-5,,,,,,,,,,,,,,,,,,,,,, 20150202,def,,,,32.585,,,,,0,,,,,,,,,,,,,,,,,,,,,, 20150202,xyz,,,,12,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, My expected output is shown below: 20150202,abc,,,,3625.300000,,,,,-5.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,def,,,,32.585,,,,,0.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,xyz,,,,12.,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, ^. added here So if column 6 and 11 doesn't have decimal point in it, then we should add '.' at the end of...

Bash modify CSV to change a field

I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

Using blank-line delimited records and colon-separated fields in awk

I'd like to be able to work with a file in awk where records are separated by a blank line and each field consists of a name followed by a colon, some optional whitespace to be ignored/discarded, followed by a value. E.g. Name: Smith, John Age: 42 Name: Jones, Mary...

Extracting Substring from File Name

I have a list of files with the following file name format: [some unknown amount of characters][_d][yyyymmdd][some unknown amount of characters] I want to extract the substring that contains the date (yyyymmdd) which I know will always be proceeded by "_d". So basically I want to extract the first 8...

Access a bash array in awk loop

I have a bash array like myarray = (1 2 3 4 5 ... n) Also I am reading a file with an input of only one line for example: 1 2 3 4 5 ... n I am reading it line by line into an array and printing it...

How to extract single-/multiline regex-matching items from an unpredictably formatted file and put each one in a single line into output file?

I have a very huge file which looks like this: <a>text</a>text blah <b>data1</b>abc<b>data2</b> <b>data3</b>blahblah <c>text</c> <d>text</d> <x>blahblah<b>data4 data5 data6</b> <b>data7 </x> That is, its formatting is unpredictable. I need to extract each <b>...</b> item (it might contain multiline text!) and put every one of them in a single separate line....

How to remove characters before and including an underscore?

In bash scripting what's an efficient way to do the following please? var="fooo_barrrr" What is the best way to remove all characters before and including the '_' so that var becomes "barrrr" please?...

AWK|BASH, use double FS and ternary operator

Is it possible? I was wondering how to do: Count fields differentiated by comma. Only the obtained first field of the previous step, count words differentiated by space. If there is more than 2 words, print NF, otherwise $0. Input cellular biol immunogenet, rosario escuela estadist, medellin medellin Expected output...

BASH - conditional sum of columns and rows in csv file

i have CSV file with some database benchmark results here is the example: Date;dbms;type;description;W;D;S;results;time;id Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;570;265;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;420;215;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;500;365;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;530;255;50 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;870;265;99 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;620;215;99...

Finding columns with only white space in a text file and replace them with a unique separator

I have a file like this: aaa b b ccc 345 ddd fgt f u 3456 e r der der 5 674 As you can see the only way that we can separate the columns is by finding columns that have only one or more spaces. How can we identify...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

Extracting columns within a range AWK

I've been trying to extract lines where a number in one columns falls within a particular range. Lets say I have a file that looks as so: chrom prediction chrom1 0.75 chrom2 0.5 chrom4 0.76 If I wanted to print lines where the prediction value was in the range from...

How to open a new terminal from my working terminal with same directory in Linux?

Is there any command to open a new terminal from my working terminal with same directory while working with ssh in Linux? e.g., I am working in the following terminal. My pwd is /home/work/Kayan01/test_run I want to open a new terminal directly from my working terminal, so that the pwd...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters

How can I replace [a-z],[a-z] with [a-z], [a-z] and keeping the letters? Input suny stony brook, stony brook,usa. Output suny stony brook, stony brook, usa. What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." sed 's/[a-z],[a-z]/, /g' <<< "suny stony brook, stony brook,usa." ...

Calling find more than once on the same folder tree

I'm running a find command multiple times on the same group of files. The results of my find commands are usually disjoint sets, AKA I'm running find -mmin +35; find -mmin -25, and doing different things to the results. It seems sort of silly to search through the entire file...

Convert AWK command to sqlite query

I have to parse very big (abt 40Gb) text files(logs) very often. Usually AWK/grep is enough for my needs, but logs are growing and now I'm curious whether sqlite3 will allow me to do the same things more efficiently. I chosen sqlite for my tests as it installed out-of-the-box (SQLite...

Redirect output from file to stdout

I have a program that can output its results only to files, with -o option. This time I need to output it to console, i.e. stdout. Here is my first try: myprog -o /dev/stdout input_file But it says: /dev/ not writable I've found this question that's similar to mine, but...

How to match and change strings in a column of a semicolon separated file?

I have a semicolon separated csv-file which looks like this: column1;column2;;123564;128;;IJL;value;;;;;3705;;;;;;;; column1;column2;;26789786413423;;CCE;value value;;;;;;3705;;;;;;;; column1;column2;;4564564;128;;SSE;value;;;;;;;;;;;;; column1;column2;;4645646;128;;JJY;someting X;;;;;;;;;;;;; column1;column2;;123132;128;;ASA;X value;;;;;;;;;;;;; column1;column2;;45643123;128;;TT;9 someting;;;;;;;;;;;;; column1;column2;;456464;128;;KK;VALUE 9 VALUE;;;;;;;;;;;;; column1;column2;;4646;128;;ST;value 6;;;;;;;;;;;;;...

Repeating the format specifiers in awk

I am trying to format the output of the AWK's printf() function. More precisely, I am trying to print a matrix with very long rows and I would like to wrap them and continue on the next line. What I am trying to do is best illustrated using Fortran. Consider...

Using AWK to parse fields with commas

Edited - TLDR: Using awk to parse fields that include commas. # original config file - confile1 $ cat confile1 list=( app1,"HOSTNAME - port - application name - alert1",99.0,99.0 app2,"HOSTNAME - port - application name - alert1",99.0,99.0 app3,"HOSTNAME - port - service name - alert2",99.0,99.0 web1,"URL - HOSTNAMES(01,02) - http://someurl.com/...

awk ternay operator, count fs with ,

How to make this command line: awk -F "," '{NF>0?$NF:$0}' to print the last field of a line if NF>0, otherwise print the whole line? Working data bogota dept math, bogota ...

if a string exist (including a variable), flip it using awk or sed

I have a csv file like this: ,College Level Math 55,Elementary Algebra 112 ,Elementary Algebra 79, ,College Level Math 102,Elementary Algebra 54 ,,College Level Math 54 I need an awk or sed command that does the following if College Level Math *,Elementary Alegrbra * exist flip it so it looks...

How to replace newlines/linebreaks with a single space, but only if they are inside of start/end regexes?

I have an extremely big (many GBs in size) file that looks like [x data1 data2 data3 data4 y] [a data5 data 6 data7 data 8 b> [x data y] ...and so on How to replace newlines (which might be surrounded by spaces) with a single space, but only if...

Use Unix Executable File to Run Shell Script and MPKG File

I have 2 shell scripts and 2 mpkg installer, I am trying to use an unix excitable file to run them all. here is the script I have, but it always has error message "No such file or directory" ? #!/bin/sh # Find the absolute script current path path=$( cd...

Split a column's values into multiple rows awk/unix/python

Here is a sample row of input data : header: id,indicator,{(pid,days_remaining)} row: id_558314,1,{(property_66021,7),(property_24444,1),(property_285395,6)} Expected output of data : header: id,indicator,pid,days_remaining row1: id_558314,1,property_66021,7 row2: id_558314,1,property_24444,1 row3: id_558314,1,property_285395,6 Is this something easily doable using awk/unix/python? Thanks in advance. Your time and help is appreciated....

Finding the average of a column excluding certain rows using AWK

I want to calculate the average of the 5th column (last column) excluding the rows with the value "9999". Would appreciate your feedback. 77.300 16 1 3.6112914285714268 9.4 77.300 16 2 -0.001737142857145102 20.0 77.300 16 3 5.1570742857142857 8.9 77.300 17 0 3.6112914285714268 8.9 77.300 17 1 2.9484342857142849 11.7 77.300 17...

Change a Script to a For Do Done Loop

I have a script that I need to turn into a loop, the script works exactly as I need it to (My Awk-fu is extremely weak, so be nice) when I put in the file name like the example below. #!/bin/bash awk -v FS='(<LastName>|<\/LastName>)' '{print $2}' 17822624.xml >> test.csv awk...

storing 'du' result in a variable [duplicate]

This question already has an answer here: How to return the output of program in a variable? 4 answers I am using Putty with bash-4.2. Therein, I am outputting file size with: du -m myfile.csv which returns: 1.25 myfile.csv How do i store this line in a variable so...

How to fix btrfs root inode errors

Running btrfsck, or more officially, btrfs check --repair gives this output stating that there are root inode errors. The repair command does not fix the issue and reruns will display the same output. The system is fully mountable and operational, but I cannot perform advanced operations on the partition (resizing)....

Why can I view some Unix executable files in Mac OS X and not others?

I am on a Macbook Pro on Mac OS X 10.10 (Yosemite). When I go to /usr/bin, git is there as a unix executable file. When I open it up in Sublime Text, all I get is unreadable machine code. However, when I open up a different Unix executable file—in...

Multiple line search in a file using java or unix command

I want to find the following pattern in a file like subclass "Pool1" 11:22:33:44:55:66 { dynamic; } the above pattern I have to find in a file. How can I find either using java or Unix command?...

Capture tee's argument inside piped Perl execution

How to capture piped command's argument ? I use : perl my_script.pl -some_args | tee arg_filename How to get arg_filename 's value inside my_script.pl ? CONTEXT I need to send this filename in a mail which my_script.pl sends at the end. I need to use tee because we dump huge...

Mounting GEOM_ELI Encrypted ZFS Pool as root

I have a 3 Disk RAIDz1 Pool, encrypted with AES128 in GEOM_ELI, that I have been using in FreeNAS since version 8. There have been many zpool upgrades, and over all I am very happy with ZFS. Lately however I have been growing frustrated with FreeNAS. Largely many bugs that...

how to deletes line from a text file that are taken from another file [duplicate]

This question already has an answer here: Remove duplicates from text file based on second text file 4 answers I have a data.txt file with a lot of lines in it and a lines.txt that contains some lines. I want to delete all lines from data.txt that match any...

Identifying when a file is changed- Bash

So in my bash shell script, I have it running through a for loop. Inside the for loop, I use find "$myarray[i]" >> $tmp to look for a certain directory each time through the loop. Sometimes, it finds the variable in myarray[i] and sometimes it doesn't. When it does find...

find numbers divisible by 3 in csv file using shell script

I have csv file having content like below : 1|2|3 4|5|6 7|8|9 Now I would like to find the numbers which are divisible by 3 using shell scripting. I would like to use awk command for this. I am learning shell scripting. So could you please help me out to...

pass enter key from Java to Shell script

I am trying a Java program to run multiple commands in unix environment. I would need to pass 'ENTER' after each command. Is there some way to pass enter in the InputStream. JSch jsch=new JSch(); Session session=jsch.getSession("MYUSERNAME", "SERVER", 22); session.setPassword("MYPASSWORD"); Properties config = new Properties(); config.put("StrictHostKeyChecking", "no"); session.setConfig(config); session.connect(); Channel...

Print the last 1,2,3..Nth or first 1,2,3…Nth matching block pattern using awk or sed

pattern1 a b pattern2 cd pattern1 re pattern2 gh pattern1 ef pattern2 qw e I can show all matching pattern by sed -n '/pattern1/,/pattern2/p' Choose the second matching pattern or any Nth by awk -vM=2 '(x+=/pattern1/)==M&&x+=/pattern2/' file pattern1 re pattern2 Print only last matching pattern by awk 'x+=/pattern1|pattern2/{!y++&&B="";B=B?B"\n"$0:$0;x==2&&y=x=0}END{print B}' file...

AWK count number of times a term appear with respect to other columns

Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

loop add a comma after nth comma using awk

I feel like this should be a fairly straight forward question, but I cant seem to get it to work. I have a csv file and I need to add comma after the nth comma in each row. I believe I have to use gsub to get it to loop....

AWK write to new column base on if else of other column

I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

How to find average and maximum in an interval using Shell [closed]

I would like to extract sum, mean and average in each 6 numbers interval from a column. I found many discussions related to this problem, but all those are for whole column. e.g. To compute sum of a column: awk '{sum+=$1} END { print sum}' To calculate Average: awk '{sum+=$1}...