bash,file,awk,multiple-columns , split file into columns with awk


split file into columns with awk

Question:

Tag: bash,file,awk,multiple-columns

I have a file which looks like this:

1. result = 1.2.3.4 (1.2.3.4)
   info: [Affected]

2. result = www.addr.com (2.3.4.5)
   info: [not Affected]

And now I want to split it in three columns, example:

1.2.3.4       1.2.3.4   Affected
www.addr.de   2.3.4.5   not Affected

I am using awk for that: cat filename.txt | awk -F "[=()'']" '{print $2 $3 $4}'

but I still not get three columns in a row. How can I fix it? the second question: is there a better alternative than awk?


Answer:

You can unset the record separator to read in each block separately, like this:

$ cat file
1. result = 1.2.3.4 (1.2.3.4)
   info: [Affected]

2. result = www.addr.com (2.3.4.5)
   info: [not Affected]
$ awk -F'[]=():[:space:][]+' -v RS= '{print $3, $4, $6 (NF==8?" " $7:"")}' file
1.2.3.4 1.2.3.4 Affected
www.addr.com 2.3.4.5 not Affected

The ternary at the end handles the two different numbers of fields (7 or 8, depending on "Affected" or "not Affected"). If there are 8 fields, then the seventh one is printed after a space, otherwise, nothing is printed.

To achieve a more neatly formatted output, you can use printf instead of print:

$ awk -F'[]=():[:space:][]+' -v RS= '{printf "%-12s%10s   %s%s%s", $3, $4, $6, (NF==8?" " $7:""), ORS}' file
1.2.3.4        1.2.3.4   Affected
www.addr.com   2.3.4.5   not Affected

The format specifiers dictate the width of each field. A - causes the content to be left-aligned. ORS is the Output Record Separator, which is a newline on your platform by default.

In terms of aligning the columns, it depends on whether you're looking for something human- or machine-readable. If you're looking to import this data into a spreadsheet, perhaps you could separate each column using a tab character \t (for example), which could be done by adding -v OFS='\t' to the first version of my answer.


Related:


How to upload file in PHP and store information in SQLi database?


php,file,mysqli,upload
So I'm trying to do this file upload, and store information such as the size of the file, name, and URL of it in a database, while having the file just upload to a folder on my computer (for testing purposes). It's uploading no problem, the only issue I'm having...

Extra backslash when storing grep in a value


linux,bash
In a bash script I have: Check="grep -e '"'\(-S mount\)'"' /etc/audit/audit.rules" set -x When you run it it shows it as: CHECK='grep -e '\''\(-S mount\)'\'' /etc/audit/audit.rules' Now it works exactly what I want but I want to understand it. Why is there 2 extra \'s?...

print filenames into scripts in bash


bash,printf,echo
How do you print each name of a file from a directory into a string and make make new scripts? to print each file name for i in `ls new_manifest*`; do echo $i; done but when I try and print the rest of the string with $i like this is...

BASH - conditional sum of columns and rows in csv file


linux,bash,csv,awk
i have CSV file with some database benchmark results here is the example: Date;dbms;type;description;W;D;S;results;time;id Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;570;265;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;420;215;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;500;365;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;530;255;50 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;870;265;99 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;620;215;99...

IllegalStateException: Iterator already obtained [duplicate]


java,file,loops,path
This question already has an answer here: java.lang.IllegalStateException: Iterator already obtained 1 answer so I wrote a little Java program to test a little stack language I made vie various test file, but for some reason it won't work. Here is the code: import org.apache.commons.io.FilenameUtils; import java.io.IOException; import java.nio.file.*;...

Why does `sort file > file` result in an empty file? [duplicate]


bash
This question already has an answer here: bash redirect input from file back into same file 7 answers When you try to sort a file in-place with sort afile > afile you silently end up with afile being an empty file. Why is that? I'd expect either an error...

Delphi - Use a string variable's name in assignfile()


file,delphi,variables,assign
Is it possible to use a variable in the assignfile command? Eg. f : Textfile ; sFile : string ; {contains 'MyFile.txt' as content} ... cFileDir = 'C:\Users\User\Desktop\Data Engine\Data\Country' ; ... Assignfile(f, cFileDir + '\' + sFile) ; ... I appreciate your help very much. if it's unclear I'll edit...

shell script cut from variables


bash,shell,shellcode
The file is like this aaa&123 bbb&234 ccc&345 aaa&456 aaa$567 bbb&678 I want to output:(contain "aaa" and text after &) 123 456 I want to do in in shell script, Follow code be consider #!/bin/bash raw=$(grep 'aaa' 1.txt) var=$(cut -f2 -d"&" "$raw") echo $var It give me a error like...

bash interactive script pass input


bash,command-line-arguments
I'm running the following interactive Jar. java -jar script.jar argument-line-here \n Now I'm creating a bash script which runs the jar file. How do I pass the argument line and the conformation "\n" to this interactive script? these are input lines for the script. This question has some answers, expect...

Assign and use of a variable in the same subshell


bash,scope,subshell
I was doing something very simple like: v=5 echo "$v" and expected it to print 5. However, it does not. The value that was just set is not available for the next command. I recently learnt that "In most shells, each command of a pipeline is executed in a separate...

linux - running a process and tailing a file simultaneously


bash,shell,tail
I want a run a long task on a remote machine (with python fabric using ssh). It logs to a file on the remote machine. What I want to do is to run that script and tail (actively display) the log file content until the script execution ends. The problem...

Why can I view some Unix executable files in Mac OS X and not others?


git,bash,shell,unix,binary
I am on a Macbook Pro on Mac OS X 10.10 (Yosemite). When I go to /usr/bin, git is there as a unix executable file. When I open it up in Sublime Text, all I get is unreadable machine code. However, when I open up a different Unix executable fileā€”in...

Aggregate failure codes in bash


bash
I've got a script which has multiple stages, and at each stage it's possible to fail, but the script can carry on running. Concretely, I generate some json, and check if the diff is correct. The diff could be wrong, but it doesn't stop the next stage of json being...

using sed to replace a line with back slashes in a shell script


regex,bash,shell,ssh,sed
I am trying to replace the bottom one of these 2 lines with sed in a file. <rule>out_prefix=orderid ^1\\d\+ updatemtnotif/</rule>\n\ <rule>out_prefix=orderid ^2\\d\+ updatemtnotif/</rule>\n\ And the following command seems to do that when executed as a command at the bash prompt sed -i [email protected]_prefix=orderid ^2\\\\d\\+ [email protected]_prefix=orderid ^2\\\\d\\+ [email protected]' /opt/temp/rules.txt however, when...

How to change svn:externals from bash file non-interactive


bash,svn,svn-externals
I have multi externals need to be set within a file externals.txt and I attempt to change the svn:externals from a bash: svn pe svn:externals svn://hostname/branchname -F extenals.txt But the command throws out an error: svn: E205007: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no...

While loop in bash using variable from txt file


linux,bash,rhel
I am new to bash and writing a script to read variables that is stored on each line of a text file (there are thousands of these variables). So I tried to write a script that would read the lines and automatically output the solution to the screen and save...

Group instances based on NA values in r


r,file,csv,instance,na
I am reading a csv file and unfortunately my dataframe has many missing values. A small snip is as following: df <- data.frame(Size= c(800, 850, 1100, 1200, 1000), Value= c(900, NA, 1300, 1100, NA), Location= c(NA, 'midcity', 'uptown', NA, 'Lakeview'), Num1 = c(2, NA, 3, 2, NA), Num2 = c(2,3,3,1,2),...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters


bash,awk,sed
How can I replace [a-z],[a-z] with [a-z], [a-z] and keeping the letters? Input suny stony brook, stony brook,usa. Output suny stony brook, stony brook, usa. What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." sed 's/[a-z],[a-z]/, /g' <<< "suny stony brook, stony brook,usa." ...

ValueError: dictionary update sequence element #0 has length 1; 2 is required while reading from file


python,file,dictionary
I'm trying to read a dictionary off a file and then make the string into a dictionary. I have this, with open("../resources/enemyStats.txt", "r") as stats: for line in stats: if self.kind in line: line = line.replace(self.kind + " ", "") line = dict(line) return line and the line in the...

AWK|BASH, use double FS and ternary operator


bash,awk
Is it possible? I was wondering how to do: Count fields differentiated by comma. Only the obtained first field of the previous step, count words differentiated by space. If there is more than 2 words, print NF, otherwise $0. Input cellular biol immunogenet, rosario escuela estadist, medellin medellin Expected output...

Bash alias function with predefined argument


bash
I have an alias gl which is an wrapper for git log. Basically like this function gitLog(){ if [ $# -eq 0 ] then git log else git log -n $1 } alias gl=gitLog I want to add an alias which just calls gitLog with an argument like this alias...

Getting HTTP 302 when downloading file in Java using Apache Commons


java,file,url,apache-commons,fileutils
I am using the following method to download a file off the internet: try { URL url = new URL("http://search.maven.org/remotecontent?filepath=com/cedarsoftware/json-io/4.0.0/json-io-4.0.0.jar"); FileUtils.copyURLToFile(url, new File(jsonerFolder.getParent() + "\\mods\\json-io-4.0.0.jar")); } catch (Exception e1) { logger.error("Downloading json-io failed with an exception"); e1.printStackTrace(); } But the downloaded file is not a jar, rather, it is an...

Bash modify CSV to change a field


linux,bash,awk
I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

Rust: Lifetime of String from file [duplicate]


file,io,rust
This question already has an answer here: Return local String as slice 1 answer I'm trying to read in some external GLSL code into Rust. The reading works properly, but I run into a lifetime issue in the final expression (in the Ok(_) branch) error: s does not live...

How to get rid of .ignore file in Git?


git,file,bitbucket,ignore
I have create an ignore file , the list included .jar files . I then merged my branch to a development branch and even though I had resolved all my conflicts , I still had a lot of errors. Most of them had something to do with support.v4 (the import...

shell script for counting replacements


bash,replace,count
Ten files located in a directory. Each file content has "apple" word. write a bash shell script to replace "apple" with "banana" in all ten files and Print the number of replacements for each file. Have tried in this way but dont know how to get number of replacement. can...

How do I check whether a file or file directory exist in bash?


bash,if-statement
I currently have this bash script (which is located in my home directory, i.e., /home/fusion809/ and I am running it as root as it's necessary for the icon copying lines): cd /home/fusion809/Pictures/Icon* declare -a A={Arch,Debian,Fedora,Mageia,Manjaro,OpenSUSE} declare -a B={Adwaita,Faenza,gnome,Humanity} for i in $A; do for j in $B; do if test...

Changing file name to the user's name PHP


php,file,upload
Instead of the file name being what the users want to name it, I would like the file name to be the user's username. As you can see in the code below, I got the user's username from the session. Now, I would just like to apply it to $_FILES,...

Macports switch PHP CLI version


php,bash,drupal,terminal,macports
I'm trying to switch my Terminal PHP version to 5.4 because I ran into some issues with Drush while updating my Drupal core. http://drupal.stackexchange.com/questions/112090/drush-command-errors The reason for these issues is my Terminal PHP version is different then my localhost. php -v in Terminal returns PHP 5.5.13 (cli) but my localhost...

how to immediately login through ssh?


bash,ssh
Here is my problem. I need to run a command ./deploy.sh -u 1540 This will fetch version 1540 of deploy.sh on SVN When I do, the script access SVN and ask for a password. I'm using ssh. It will first ask me a password since it guesses my SVN login...

AWK write to new column base on if else of other column


linux,bash,shell,awk,sed
I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Capitalize all files in a directory using Bash


osx,bash,rename
I have a directory called "Theme_1" and I want to capitalize all the filenames starting with "theme". The first letter of every file name is lowercase and I want it to be upcase. I've tried this but all the letters in the filename are upcase, which is not what I...

Matching string inside file and returning result


regex,string,bash,shell,grep
I've got a few peculiar issues with trying to search for a string inside of a .db file. The way I tried was by using grep, which does apparently find the string(s), although this is the output: $ grep "ext" *.db Binary file enormous.db matches There are a couple problems...

How do I silence the HEAD of a curl request while using the silent flag?


bash,shell,curl,command-line,pipe
When I run the curl command and direct the data to a file, I get back the content of the site as expected. $ curl "www.site.com" > file.txt $ head file.txt Top of site ... However, this command shows a progress bar, which I do not want: % Total %...

how to modify an array value with given index?


arrays,linux,bash
I want to modify an array cell, which I can do when I know the cell as a number. However here my cell position is given by $i. pomme[`${i}`]="" I tried without the `` and it doesn't work either? How am I suppose to do it?...

Identifying when a file is changed- Bash


bash,shell,unix
So in my bash shell script, I have it running through a for loop. Inside the for loop, I use find "$myarray[i]" >> $tmp to look for a certain directory each time through the loop. Sometimes, it finds the variable in myarray[i] and sometimes it doesn't. When it does find...

Calling find more than once on the same folder tree


linux,bash,shell,unix,find
I'm running a find command multiple times on the same group of files. The results of my find commands are usually disjoint sets, AKA I'm running find -mmin +35; find -mmin -25, and doing different things to the results. It seems sort of silly to search through the entire file...

Bash script using sed acts differently when passing variable


bash,sed
I have a script that I am writing that checks a value and then based on the value modifies it. I am trying to understand why it works this one way and not the other. Based on the google and stackoverflow searches I did, nothing really fits what I am...

Saying there are 0 arguments when I have 2? Trying to open a CSV file to write into


ruby,file,csv,dir
I'm trying to read from a CSV file and codify people into groups using an equation. I append the name of their group they fall into to the end of the array that their row creates. Then I write it to a new file so I don't overwrite the original...

How to extract first letters of dashed separated words in a bash variable?


linux,string,bash,shell,variables
I would like to extract the first letter of dashed separated words value of my bash variable, like this: MY_TEXT=this-is-my-custom-text I would like to create a second variable like this: MY_INITIALS=timct...

Bash script that removes C source comments


bash
I need to write a bash script which copies all .c files from a directory $2 to another directory $1, but without comments. I only have to remove comments that begin with //, might have tabs/spaces before the comment, but not letters. Also, I need to do it with only...

How to test if a command is a shell reserved word?


bash,shell
I am writing a bash script and I would like to verify if a string is a shell reserved word (like if, for, alias, etc...). How can I can do this?...

Storing columns on disk and reading rows


c++,file,matrix,io
So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the...

Strings vs binary for storing variables inside the file format


c++,file,hdf5,dataformat
We aim at using HDF5 for our data format. HDF5 has been selected because it is a hierarchical filesystem-like cross-platform data format and it supports large amounts of data. The file will contain arrays and some parameters. The question is about how to store the parameters (which are not made...

Check if file exists using Apache Commons VFS2


java,file,sftp,apache-commons-vfs
I would like to ask if there is a way how to check if file already exists in the folder using only Apache Commons. I have method which uploads into the SFTP folder but it overwrites current files anytime the method is running. The method is set to run every...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk


linux,bash,awk,sed,sh
I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

Finding the average of a column excluding certain rows using AWK


linux,bash,awk,scripting
I want to calculate the average of the 5th column (last column) excluding the rows with the value "9999". Would appreciate your feedback. 77.300 16 1 3.6112914285714268 9.4 77.300 16 2 -0.001737142857145102 20.0 77.300 16 3 5.1570742857142857 8.9 77.300 17 0 3.6112914285714268 8.9 77.300 17 1 2.9484342857142849 11.7 77.300 17...

Shell script to loop over files with same names but different extensions


linux,bash,shell
For example say I have: filename1.ext1 filename1.ext2 filename2.ext1 filename2.ext2 I need to write a shell script to feed these files into a program like so: program filename1.ext1 filename1.ext2 program filename2.ext1 filename2.ext2 Additionally the .ext1 files must be entered first and the .ext2 files second. Any help would be appreciated....

Permission denied in find: why do we need 2>&1?


bash,error-handling,find,io-redirection
I was recently searching for a way to filter out 'Permission denied' errors while searching for a file using the find command, and I found this link: How can I exclude all "permission denied" messages from "find"? Here's an the answer that worked for me from the link: find ....

Python: can't access newly defined environment variables


python,bash,environment-variables
I can't access my env var: import subprocess, os print os.environ.get('PATH') # Works well print os.environ.get('BONSAI') # doesn't work But the env var is well added in my /home/me/.bashrc: BONSAI=/home/me/Utils/bonsai_v3.2 export BONSAI And I can access this env var from a new terminal....