bash,file,awk,multiple-columns , split file into columns with awk

split file into columns with awk


Tag: bash,file,awk,multiple-columns

I have a file which looks like this:

1. result = (
   info: [Affected]

2. result = (
   info: [not Affected]

And now I want to split it in three columns, example:   Affected   not Affected

I am using awk for that: cat filename.txt | awk -F "[=()'']" '{print $2 $3 $4}'

but I still not get three columns in a row. How can I fix it? the second question: is there a better alternative than awk?


You can unset the record separator to read in each block separately, like this:

$ cat file
1. result = (
   info: [Affected]

2. result = (
   info: [not Affected]
$ awk -F'[]=():[:space:][]+' -v RS= '{print $3, $4, $6 (NF==8?" " $7:"")}' file Affected not Affected

The ternary at the end handles the two different numbers of fields (7 or 8, depending on "Affected" or "not Affected"). If there are 8 fields, then the seventh one is printed after a space, otherwise, nothing is printed.

To achieve a more neatly formatted output, you can use printf instead of print:

$ awk -F'[]=():[:space:][]+' -v RS= '{printf "%-12s%10s   %s%s%s", $3, $4, $6, (NF==8?" " $7:""), ORS}' file   Affected   not Affected

The format specifiers dictate the width of each field. A - causes the content to be left-aligned. ORS is the Output Record Separator, which is a newline on your platform by default.

In terms of aligning the columns, it depends on whether you're looking for something human- or machine-readable. If you're looking to import this data into a spreadsheet, perhaps you could separate each column using a tab character \t (for example), which could be done by adding -v OFS='\t' to the first version of my answer.


shell script cut from variables

The file is like this aaa&123 bbb&234 ccc&345 aaa&456 aaa$567 bbb&678 I want to output:(contain "aaa" and text after &) 123 456 I want to do in in shell script, Follow code be consider #!/bin/bash raw=$(grep 'aaa' 1.txt) var=$(cut -f2 -d"&" "$raw") echo $var It give me a error like...

How to upload file in PHP and store information in SQLi database?

So I'm trying to do this file upload, and store information such as the size of the file, name, and URL of it in a database, while having the file just upload to a folder on my computer (for testing purposes). It's uploading no problem, the only issue I'm having...

Replace [a-z],[a-z] with [a-z], [a-z] and keep the letters

How can I replace [a-z],[a-z] with [a-z], [a-z] and keeping the letters? Input suny stony brook, stony brook,usa. Output suny stony brook, stony brook, usa. What I have tried sed 's/[a-z],[a-z]/[a-z], [a-z]/g' <<< "suny stony brook, stony brook,usa." sed 's/[a-z],[a-z]/, /g' <<< "suny stony brook, stony brook,usa." ...

bash interactive script pass input

I'm running the following interactive Jar. java -jar script.jar argument-line-here \n Now I'm creating a bash script which runs the jar file. How do I pass the argument line and the conformation "\n" to this interactive script? these are input lines for the script. This question has some answers, expect...

Strings vs binary for storing variables inside the file format

We aim at using HDF5 for our data format. HDF5 has been selected because it is a hierarchical filesystem-like cross-platform data format and it supports large amounts of data. The file will contain arrays and some parameters. The question is about how to store the parameters (which are not made...

Bash script using sed acts differently when passing variable

I have a script that I am writing that checks a value and then based on the value modifies it. I am trying to understand why it works this one way and not the other. Based on the google and stackoverflow searches I did, nothing really fits what I am...

Check if file exists using Apache Commons VFS2

I would like to ask if there is a way how to check if file already exists in the folder using only Apache Commons. I have method which uploads into the SFTP folder but it overwrites current files anytime the method is running. The method is set to run every...

Capitalize all files in a directory using Bash

I have a directory called "Theme_1" and I want to capitalize all the filenames starting with "theme". The first letter of every file name is lowercase and I want it to be upcase. I've tried this but all the letters in the filename are upcase, which is not what I...

Rust: Lifetime of String from file [duplicate]

This question already has an answer here: Return local String as slice 1 answer I'm trying to read in some external GLSL code into Rust. The reading works properly, but I run into a lifetime issue in the final expression (in the Ok(_) branch) error: s does not live...

Delphi - Use a string variable's name in assignfile()

Is it possible to use a variable in the assignfile command? Eg. f : Textfile ; sFile : string ; {contains 'MyFile.txt' as content} ... cFileDir = 'C:\Users\User\Desktop\Data Engine\Data\Country' ; ... Assignfile(f, cFileDir + '\' + sFile) ; ... I appreciate your help very much. if it's unclear I'll edit...

shell script for counting replacements

Ten files located in a directory. Each file content has "apple" word. write a bash shell script to replace "apple" with "banana" in all ten files and Print the number of replacements for each file. Have tried in this way but dont know how to get number of replacement. can...

using sed to replace a line with back slashes in a shell script

I am trying to replace the bottom one of these 2 lines with sed in a file. <rule>out_prefix=orderid ^1\\d\+ updatemtnotif/</rule>\n\ <rule>out_prefix=orderid ^2\\d\+ updatemtnotif/</rule>\n\ And the following command seems to do that when executed as a command at the bash prompt sed -i '[email protected]_prefix=orderid ^2\\\\d\\+ updatemtnotif/@out_prefix=orderid ^2\\\\d\\+ updatemtnotif_fr/@g' /opt/temp/rules.txt however, when...

Group instances based on NA values in r

I am reading a csv file and unfortunately my dataframe has many missing values. A small snip is as following: df <- data.frame(Size= c(800, 850, 1100, 1200, 1000), Value= c(900, NA, 1300, 1100, NA), Location= c(NA, 'midcity', 'uptown', NA, 'Lakeview'), Num1 = c(2, NA, 3, 2, NA), Num2 = c(2,3,3,1,2),...

Bash alias function with predefined argument

I have an alias gl which is an wrapper for git log. Basically like this function gitLog(){ if [ $# -eq 0 ] then git log else git log -n $1 } alias gl=gitLog I want to add an alias which just calls gitLog with an argument like this alias...

AWK write to new column base on if else of other column

I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

linux - running a process and tailing a file simultaneously

I want a run a long task on a remote machine (with python fabric using ssh). It logs to a file on the remote machine. What I want to do is to run that script and tail (actively display) the log file content until the script execution ends. The problem...

How to extract first letters of dashed separated words in a bash variable?

I would like to extract the first letter of dashed separated words value of my bash variable, like this: MY_TEXT=this-is-my-custom-text I would like to create a second variable like this: MY_INITIALS=timct...

Calling find more than once on the same folder tree

I'm running a find command multiple times on the same group of files. The results of my find commands are usually disjoint sets, AKA I'm running find -mmin +35; find -mmin -25, and doing different things to the results. It seems sort of silly to search through the entire file...

Matching string inside file and returning result

I've got a few peculiar issues with trying to search for a string inside of a .db file. The way I tried was by using grep, which does apparently find the string(s), although this is the output: $ grep "ext" *.db Binary file enormous.db matches There are a couple problems...

Why can I view some Unix executable files in Mac OS X and not others?

I am on a Macbook Pro on Mac OS X 10.10 (Yosemite). When I go to /usr/bin, git is there as a unix executable file. When I open it up in Sublime Text, all I get is unreadable machine code. However, when I open up a different Unix executable fileā€”in...

Getting HTTP 302 when downloading file in Java using Apache Commons

I am using the following method to download a file off the internet: try { URL url = new URL(""); FileUtils.copyURLToFile(url, new File(jsonerFolder.getParent() + "\\mods\\json-io-4.0.0.jar")); } catch (Exception e1) { logger.error("Downloading json-io failed with an exception"); e1.printStackTrace(); } But the downloaded file is not a jar, rather, it is an...

IllegalStateException: Iterator already obtained [duplicate]

This question already has an answer here: java.lang.IllegalStateException: Iterator already obtained 1 answer so I wrote a little Java program to test a little stack language I made vie various test file, but for some reason it won't work. Here is the code: import; import; import java.nio.file.*;...

How do I check whether a file or file directory exist in bash?

I currently have this bash script (which is located in my home directory, i.e., /home/fusion809/ and I am running it as root as it's necessary for the icon copying lines): cd /home/fusion809/Pictures/Icon* declare -a A={Arch,Debian,Fedora,Mageia,Manjaro,OpenSUSE} declare -a B={Adwaita,Faenza,gnome,Humanity} for i in $A; do for j in $B; do if test...

Bash modify CSV to change a field

I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

Shell script to loop over files with same names but different extensions

For example say I have: filename1.ext1 filename1.ext2 filename2.ext1 filename2.ext2 I need to write a shell script to feed these files into a program like so: program filename1.ext1 filename1.ext2 program filename2.ext1 filename2.ext2 Additionally the .ext1 files must be entered first and the .ext2 files second. Any help would be appreciated....

BASH - conditional sum of columns and rows in csv file

i have CSV file with some database benchmark results here is the example: Date;dbms;type;description;W;D;S;results;time;id Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;570;265;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;420;215;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;500;365;50 Mon Jun 15 14:22:20 CEST 2015;sqlite;on-disk;text;2;1;1;530;255;50 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;870;265;99 Mon Jun 15 14:22:20 CEST 2015;hsql;on-disk;text;2;1;1;620;215;99...

Saying there are 0 arguments when I have 2? Trying to open a CSV file to write into

I'm trying to read from a CSV file and codify people into groups using an equation. I append the name of their group they fall into to the end of the array that their row creates. Then I write it to a new file so I don't overwrite the original...

How to get rid of .ignore file in Git?

I have create an ignore file , the list included .jar files . I then merged my branch to a development branch and even though I had resolved all my conflicts , I still had a lot of errors. Most of them had something to do with support.v4 (the import...

how to modify an array value with given index?

I want to modify an array cell, which I can do when I know the cell as a number. However here my cell position is given by $i. pomme[`${i}`]="" I tried without the `` and it doesn't work either? How am I suppose to do it?...

Storing columns on disk and reading rows

So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the...

How to append entry the end of a multi-line entry using any of stream editors like sed or awk

I am trying to use a script to append the host name at the end of a multi-line entry of a specific Host_Alias field in sudoers file. The current sudoers file has an entry similar to : Host_Alias srv_linuxestate= \ host10,host12,host13,host1,host50,\ host16,host1,host2,host11,host15,host21,\ host3,host14 My required output would be something like...

Permission denied in find: why do we need 2>&1?

I was recently searching for a way to filter out 'Permission denied' errors while searching for a file using the find command, and I found this link: How can I exclude all "permission denied" messages from "find"? Here's an the answer that worked for me from the link: find ....

ValueError: dictionary update sequence element #0 has length 1; 2 is required while reading from file

I'm trying to read a dictionary off a file and then make the string into a dictionary. I have this, with open("../resources/enemyStats.txt", "r") as stats: for line in stats: if self.kind in line: line = line.replace(self.kind + " ", "") line = dict(line) return line and the line in the...

Aggregate failure codes in bash

I've got a script which has multiple stages, and at each stage it's possible to fail, but the script can carry on running. Concretely, I generate some json, and check if the diff is correct. The diff could be wrong, but it doesn't stop the next stage of json being...

Assign and use of a variable in the same subshell

I was doing something very simple like: v=5 echo "$v" and expected it to print 5. However, it does not. The value that was just set is not available for the next command. I recently learnt that "In most shells, each command of a pipeline is executed in a separate...

How to test if a command is a shell reserved word?

I am writing a bash script and I would like to verify if a string is a shell reserved word (like if, for, alias, etc...). How can I can do this?...

Macports switch PHP CLI version

I'm trying to switch my Terminal PHP version to 5.4 because I ran into some issues with Drush while updating my Drupal core. The reason for these issues is my Terminal PHP version is different then my localhost. php -v in Terminal returns PHP 5.5.13 (cli) but my localhost...

While loop in bash using variable from txt file

I am new to bash and writing a script to read variables that is stored on each line of a text file (there are thousands of these variables). So I tried to write a script that would read the lines and automatically output the solution to the screen and save...

AWK|BASH, use double FS and ternary operator

Is it possible? I was wondering how to do: Count fields differentiated by comma. Only the obtained first field of the previous step, count words differentiated by space. If there is more than 2 words, print NF, otherwise $0. Input cellular biol immunogenet, rosario escuela estadist, medellin medellin Expected output...

Bash script that removes C source comments

I need to write a bash script which copies all .c files from a directory $2 to another directory $1, but without comments. I only have to remove comments that begin with //, might have tabs/spaces before the comment, but not letters. Also, I need to do it with only...

Identifying when a file is changed- Bash

So in my bash shell script, I have it running through a for loop. Inside the for loop, I use find "$myarray[i]" >> $tmp to look for a certain directory each time through the loop. Sometimes, it finds the variable in myarray[i] and sometimes it doesn't. When it does find...

Why does `sort file > file` result in an empty file? [duplicate]

This question already has an answer here: bash redirect input from file back into same file 7 answers When you try to sort a file in-place with sort afile > afile you silently end up with afile being an empty file. Why is that? I'd expect either an error...

How to change svn:externals from bash file non-interactive

I have multi externals need to be set within a file externals.txt and I attempt to change the svn:externals from a bash: svn pe svn:externals svn://hostname/branchname -F extenals.txt But the command throws out an error: svn: E205007: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no...

how to immediately login through ssh?

Here is my problem. I need to run a command ./ -u 1540 This will fetch version 1540 of on SVN When I do, the script access SVN and ask for a password. I'm using ssh. It will first ask me a password since it guesses my SVN login...

print filenames into scripts in bash

How do you print each name of a file from a directory into a string and make make new scripts? to print each file name for i in `ls new_manifest*`; do echo $i; done but when I try and print the rest of the string with $i like this is...

Finding the average of a column excluding certain rows using AWK

I want to calculate the average of the 5th column (last column) excluding the rows with the value "9999". Would appreciate your feedback. 77.300 16 1 3.6112914285714268 9.4 77.300 16 2 -0.001737142857145102 20.0 77.300 16 3 5.1570742857142857 8.9 77.300 17 0 3.6112914285714268 8.9 77.300 17 1 2.9484342857142849 11.7 77.300 17...

Changing file name to the user's name PHP

Instead of the file name being what the users want to name it, I would like the file name to be the user's username. As you can see in the code below, I got the user's username from the session. Now, I would just like to apply it to $_FILES,...

How do I silence the HEAD of a curl request while using the silent flag?

When I run the curl command and direct the data to a file, I get back the content of the site as expected. $ curl "" > file.txt $ head file.txt Top of site ... However, this command shows a progress bar, which I do not want: % Total %...

Extra backslash when storing grep in a value

In a bash script I have: Check="grep -e '"'\(-S mount\)'"' /etc/audit/audit.rules" set -x When you run it it shows it as: CHECK='grep -e '\''\(-S mount\)'\'' /etc/audit/audit.rules' Now it works exactly what I want but I want to understand it. Why is there 2 extra \'s?...

Python: can't access newly defined environment variables

I can't access my env var: import subprocess, os print os.environ.get('PATH') # Works well print os.environ.get('BONSAI') # doesn't work But the env var is well added in my /home/me/.bashrc: BONSAI=/home/me/Utils/bonsai_v3.2 export BONSAI And I can access this env var from a new terminal....