unix,grep,cut , How to change horizontal input to vertical output

How to change horizontal input to vertical output


Tag: unix,grep,cut

I am using UNIX Korn shell. I am trying to create a program in Unix that will search a text file which is shown below:

Last Name:First Name:City:State:Class:Semester Enrolled:Year First Enrolled

Beckman:John:Ft. Polk:LA:Freshman:Spring:2014

Then take that line and cut it out and display it vertically. So if I search Gilman. It would produce:


However included in this I should also be able to produce the following layout:

Last Name: Gilman
First Name: Randy
City: Manhattan
State: KS
Class: Junior
Semester Enrolled: Spring
Year First Enrolled: 2010

Now I have figured out most of it which I will display in my code below:

                         A - Add Student Information 
                         D - Delete Student Information
                         M - Modify Student Information
                         I - Inquiry on a Student
                         X - Exit


    echo -en '\n'
    echo -en '\n'
    echo "                      Pleasa choose one of the following:                      "

    #take input from operation

    read choice

    case $choice in
        a|A) ;; 
        d|D) ;;
        m|M) ;;
            #Create Inguiry Message
            echo "                                 Record Inquiry                                 "
            echo -en '\n'
            echo -en '\n'
            echo "What is the last name of the person"
            #Gather search parameter
            read last_name
            grep -i "$last_name" $1 | cut -f 1-7 -d ':' ;;

        x|X) echo "The program is ending" ; exit 0;;
        *) echo -en '\n' ;echo "Not an acceptable entry." ;echo "Please press enter to try again"; read JUNK | tee -a $2 ; continue

I am just really having trouble on redirecting input to the proper output. There is much more to the program but this is the relevant portion for this problem. Any help would be appreciated.


Ok the final answer is shown below:

while IFS=: read c1 c2 c3 c4 c5 c6 c7 rest ; do
                case "$c1" in
                    "$last_name" )
                        echo -e "Last Name: $c1\nFirst Name: $c2\nCity: $c3\nState: $c4\nClass: $c5\nSemester Enrolled: $c6\nYear First Enrolled: $c7\n\n"
            done < $1

This performed the intended outcome correctly thank you everyone for your help. I still have some other questions but I will make a new topic for that so it doesn't get to jumbled together.


Replace your line

 grep -i "$last_name" $1 | cut -f 1-7 -d ':' ;;


  awk -F: -vnameMatch="$last_name" \
      printf("LastName:%s\nFirstName:%s\nCity:%s\nState:%s\nClass:%s\nSemester Enrolled:%s\nYear First Enrolled:%s\n\n", \
             $1, $2, $3, $4, $5, $6, $7)
    }' $1 

It's pretty much the same idea in ksh.

while IFS=: read c1 c2 c3 c4 c5 c6 c7 rest ; do
  case "$c1" in 
    "$last_name" )
        printf "LastName:%s\nFirstName:%s\nCity:%s\nState:%s\nClass:%s\nSemester Enrolled:%s\nYear First Enrolled:%s\n\n",
             "$c1" "$c2" "$c3" "$c4" "$c5" "$c6" "$c7"
  done < $1

I think I've got all the syntax right, but don't have the energy tonight to test :-/ ... If you can use this, and there are problems, post a comment and I'll clean it up.



#Case statement for conducting an inquiry of the file
            #Create Inguiry Message
            echo "                                 Record Inquiry                                 "
            echo -en '\n'
            echo -en '\n'
            echo "What is the last name of the person:"
            #Gather search parameter
            read last_name
            while IFS=: read c1 c2 c3 c4 c5 c6 c7 rest ; do
                case "$c1" in
                    "$last_name" )
                        printf "Last Name:%s\nFirst Name:%s\nCity:%s\nState:%s\nClass:%s\nSemester Enrolled:%s\nYear First Enrolled:%s\n\n", "$c1", "$c2", "$c3", "$c4", "$c5", "$c6", "$c7"
            done < $2
        #Case statement for ending the program          
        x|X) echo "The program is ending" ; exit 0;;

The error message is:

./asg7s: line 26: syntax error at line 93: `)' unexpected

Line 93 is

x|X) echo "The program is ending" ; exit 0;;

Kind of weird because I didn't mess with that part of the program so I know it has to be something in the portion I changed.


Adding something extra so I can save RLG edit without other's "approval"


How to fix btrfs root inode errors

Running btrfsck, or more officially, btrfs check --repair gives this output stating that there are root inode errors. The repair command does not fix the issue and reruns will display the same output. The system is fully mountable and operational, but I cannot perform advanced operations on the partition (resizing)....

Delete some lines from text using Linux command

I know how to match text using regex patterns but not how to manipulate them. I have used grep to match and extract lines from a text file, but I want to remove those lines from the text. How can I achieve this without having to write a python or...

How to open a new terminal from my working terminal with same directory in Linux?

Is there any command to open a new terminal from my working terminal with same directory while working with ssh in Linux? e.g., I am working in the following terminal. My pwd is /home/work/Kayan01/test_run I want to open a new terminal directly from my working terminal, so that the pwd...

How to do the expansion of variable in shell script? [duplicate]

This question already has an answer here: Bash expand variable in a variable 4 answers I hava a column in database which contains the following data: sql_proc|test_sql.sql|/home/Desktop/myfile.txt|$IMP_FILES/myFolder|convert I have fetched this into a variable and I have used the cut command with "|" as delimiter and saved each field...

How to remove characters before and including an underscore?

In bash scripting what's an efficient way to do the following please? var="fooo_barrrr" What is the best way to remove all characters before and including the '_' so that var becomes "barrrr" please?...

How to copy matches from an extremely large file if it contains no newlines?

The problem is I cannot avoid working with extremely big files which contain no newlines in them: <a>text1</a>...gigabytes of data here, all in one single line...[a text to extract b> What should I do if I want to copy matches from this file (putting every match in a separate line,...

ImageMagick - Making 2 GIFs into side by side GIFs using IM convert

I have 2 GIFs that are the same length. I want to put the GIFs beside each other to have 1 GIF with both playing at the same time. I have tried to use the convert tool with: convert +append 1.gif1 2.gif output.gif However, this seems to blend all the...

Reading rsync source from file results in improper parsing of file names with white space

I wrote a simple script that searches through a specific directory defined by the variable "SCOPE" producing a list of directories that were modified within the past 24 hours printing them to a temp file. The first line of the file is deleted (to exclude the root level of the...

gzip and pipe to output (performance consideration)

q1) Can i check if I do a gzip -c file | encrypt (some parameters) a) does gzip print out the output line by line and pipe it to the encrypt function or b) gzip will be perform 1st, then the output will be pipe all at once to the...

SFTP Processbuilder

I have to invoke unix commands from Java and I have to use ProcessBuilder. I want to login through SFTP using key authentication and I want to SFTP the file from local directory to remote location. ProcessBuilder pb = new ProcessBuilder("sftp","-oIdentityFile=privateKey","-b","commands.txt","[email protected]"); Content in commands.txt is put localFileDirectory remoteDirectory exit Is...

pass enter key from Java to Shell script

I am trying a Java program to run multiple commands in unix environment. I would need to pass 'ENTER' after each command. Is there some way to pass enter in the InputStream. JSch jsch=new JSch(); Session session=jsch.getSession("MYUSERNAME", "SERVER", 22); session.setPassword("MYPASSWORD"); Properties config = new Properties(); config.put("StrictHostKeyChecking", "no"); session.setConfig(config); session.connect(); Channel...

Bash, Using grep, sed or awk to extract section of text and then match

I have a text file and want to extract all interfaces matching "blue" random text random text random text random text random text int 1 random text blue random text random text int 2 random text random text red random text int 3 random text random text random text blue...

Capture tee's argument inside piped Perl execution

How to capture piped command's argument ? I use : perl my_script.pl -some_args | tee arg_filename How to get arg_filename 's value inside my_script.pl ? CONTEXT I need to send this filename in a mail which my_script.pl sends at the end. I need to use tee because we dump huge...

How to find average and maximum in an interval using Shell [closed]

I would like to extract sum, mean and average in each 6 numbers interval from a column. I found many discussions related to this problem, but all those are for whole column. e.g. To compute sum of a column: awk '{sum+=$1} END { print sum}' To calculate Average: awk '{sum+=$1}...

Doubts with for loop in Unix

Sorry if its sounds dumb but I'm currently studying for an exam to apply for a job (administrative one) and I learnt about java, php, python but not much about unix scripts. I can see it's very similar to php in structure and I have many questions about it ....

Parse IP and Download-Total from mikrotik

I wanna extract IP and download-total from mikrotik command /queue simple print stat Here's some example : 0 name="101" target= rate=0bps/0bps total-rate=0bps packet-rate=0/0 total-packet-rate=0 queued-bytes=0/0 total-queued-bytes=0 queued-packets=0/0 total-queued-packets=0 bytes=17574842/389197663 total-bytes=0 packets=191226/308561 total-packets=0 dropped=9/5899 total-dropped=0 1 name="102" target= rate=0bps/0bps total-rate=0bps packet-rate=0/0 total-packet-rate=0 queued-bytes=0/0 total-queued-bytes=0...

What is wrong with these formats for Unix Shell Scripting Comparing Strings [duplicate]

This question already has an answer here: bash output command not found 4 answers I've been reading through StackExchanges posts on scripting and none of the things that I have tried are working as I would expect. Also this is my 1st shell script. # Script File Begin echo...

Join Statement omitting entries

Using: Unix 2.6.18-194.el5 I am having an issue where this join statement is omitting values/indexes from the match. I found out the values are between 11-90 (out of about 3.5 Million entries) and I have tried to look for foreign characters but I may be overlooking something (Tried cat -v...

Matching string inside file and returning result

I've got a few peculiar issues with trying to search for a string inside of a .db file. The way I tried was by using grep, which does apparently find the string(s), although this is the output: $ grep "ext" *.db Binary file enormous.db matches There are a couple problems...

find numbers divisible by 3 in csv file using shell script

I have csv file having content like below : 1|2|3 4|5|6 7|8|9 Now I would like to find the numbers which are divisible by 3 using shell scripting. I would like to use awk command for this. I am learning shell scripting. So could you please help me out to...

Split a column's values into multiple rows awk/unix/python

Here is a sample row of input data : header: id,indicator,{(pid,days_remaining)} row: id_558314,1,{(property_66021,7),(property_24444,1),(property_285395,6)} Expected output of data : header: id,indicator,pid,days_remaining row1: id_558314,1,property_66021,7 row2: id_558314,1,property_24444,1 row3: id_558314,1,property_285395,6 Is this something easily doable using awk/unix/python? Thanks in advance. Your time and help is appreciated....

Improving a sort program's efficiency; displaying column sort % completion in terminal

I have a big pipe-delimited input file approx 6 million lines as below: 24|BBG000SJFVB0|EQ0000000009296012|OI SA-ADR|OIBR/C|US|ADR|Equity 16|BBG002PHVB83|EQ0000000022353186|BLOOM SELECT INCOME FUND|BLB-U|CT|Closed-End Fund|Equity -50|BBG000V0TN75|EQ0000000010271114|MECHEL-PREF SPON ADR|MTL/P|US|ADR|Equity 20|BBG002S0ZR60|EQ0000000022739316|DIVIDEND 15 SPLIT CORP II-RT|DF-R|CT|Closed-End Fund|Equity -20|BBG001R3LGM8|EQ0000000017879513|ING FLOATING RATE SENIOR LOA|ISL/U|CT|Closed-End Fund|Equity 0|BBG006M6SXL2|EQ0000000006846232|AA...

Identifying when a file is changed- Bash

So in my bash shell script, I have it running through a for loop. Inside the for loop, I use find "$myarray[i]" >> $tmp to look for a certain directory each time through the loop. Sometimes, it finds the variable in myarray[i] and sometimes it doesn't. When it does find...

Mounting GEOM_ELI Encrypted ZFS Pool as root

I have a 3 Disk RAIDz1 Pool, encrypted with AES128 in GEOM_ELI, that I have been using in FreeNAS since version 8. There have been many zpool upgrades, and over all I am very happy with ZFS. Lately however I have been growing frustrated with FreeNAS. Largely many bugs that...

C++ Compile error on NetBSD '::system' has not been declared

I have some C++ code that compiles nicely on Linux but so far I'm having trouble getting it to compile correctly on NetBSD. These are my includes: #include <fstream> #include <iostream> #include <cstdlib> #include <unistd.h> #include <iomanip> #include <boost/regex.hpp> I compile with this command: g++ -v -O2 -fno-strict-aliasing -I /usr/pkg/include...

How to replace newlines/linebreaks with a single space, but only if they are inside of start/end regexes?

I have an extremely big (many GBs in size) file that looks like [x data1 data2 data3 data4 y] [a data5 data 6 data7 data 8 b> [x data y] ...and so on How to replace newlines (which might be surrounded by spaces) with a single space, but only if...

grep first n rows, return file name only

I can do the following to search for what I need and return the file name: grep -l "mysearchstring" ./*.xml However the files I am searching are huge so this takes forever. The string I am searching will appear in the first 200 rows so how can I search only...

how to deletes line from a text file that are taken from another file [duplicate]

This question already has an answer here: Remove duplicates from text file based on second text file 4 answers I have a data.txt file with a lot of lines in it and a lines.txt that contains some lines. I want to delete all lines from data.txt that match any...

Calling find more than once on the same folder tree

I'm running a find command multiple times on the same group of files. The results of my find commands are usually disjoint sets, AKA I'm running find -mmin +35; find -mmin -25, and doing different things to the results. It seems sort of silly to search through the entire file...

how to check a directory exist and can write file

I need to check directory test_file exist, if yes then need to check directory test_file is having write permission so that files will be writen in the same. this functionality i needed in shell script(ksh). What would be return code from sqlldr for single table load and multi table...

How to pass url to wget

If I have a document with many links and I want to download especially one picture with the name: www.website.de/picture/example_2015-06-15.jpeg. How can I write a command that downloads me automatically exactly this one I extracted out of my document. My idea would be this but I'll get a failure message...

Why can I view some Unix executable files in Mac OS X and not others?

I am on a Macbook Pro on Mac OS X 10.10 (Yosemite). When I go to /usr/bin, git is there as a unix executable file. When I open it up in Sublime Text, all I get is unreadable machine code. However, when I open up a different Unix executable fileā€”in...

Redirect output from file to stdout

I have a program that can output its results only to files, with -o option. This time I need to output it to console, i.e. stdout. Here is my first try: myprog -o /dev/stdout input_file But it says: /dev/ not writable I've found this question that's similar to mine, but...

Extracting columns within a range AWK

I've been trying to extract lines where a number in one columns falls within a particular range. Lets say I have a file that looks as so: chrom prediction chrom1 0.75 chrom2 0.5 chrom4 0.76 If I wanted to print lines where the prediction value was in the range from...

Why is Unix/Terminal faster than R?

I'm new to Unix, however, I have recently realized that very simple Unix commands can do very simple things to large data set very very quickly. My question is why are these Unix commands so fast relative to R? Let's begin by assuming that the data is big, but not...

Using pure Python over grep?

I am not familiar with grep as I've always been on a Windows system so when someone suggested I add these lines to my code, I'm a little confused... grep = 'grep -E \'@import.*{}\' * -l'.format(name) proc = Popen(grep, shell=True, cwd=info['path'], stdout=PIPE, stderr=PIPE) From my understanding, this is trying to...

How to extract single-/multiline regex-matching items from an unpredictably formatted file and put each one in a single line into output file?

I have a very huge file which looks like this: <a>text</a>text blah <b>data1</b>abc<b>data2</b> <b>data3</b>blahblah <c>text</c> <d>text</d> <x>blahblah<b>data4 data5 data6</b> <b>data7 </x> That is, its formatting is unpredictable. I need to extract each <b>...</b> item (it might contain multiline text!) and put every one of them in a single separate line....

Converting values from for loop to json format

Below is a snippet of a for loop where I sort txt file names. I am then trying to save the results in a json format file. However it results in an invalid json format due to the last , inserted in obj. How could i convert to json format...

storing 'du' result in a variable [duplicate]

This question already has an answer here: How to return the output of program in a variable? 4 answers I am using Putty with bash-4.2. Therein, I am outputting file size with: du -m myfile.csv which returns: 1.25 myfile.csv How do i store this line in a variable so...

grep in R does not find rows when terms are ORed together

I am new to R. I am working on a very large dataset with 1M+ rows and 1000+ columns that needs serious data cleaning. I have managed to clean the data for most part, however that has left me with some messy repetitive code. I am trying to use grep...

Random password generate in shell script with one special character

I have following code </dev/urandom tr -dc '[email protected]#$%&_+=' | head -c 16 which is randomly generating password perfectly. I want two changes: It should only contain 1 special symbols that i have listed there It should choose random length of the password I tried with length = $(($RANDOM%8+9)) then putting...

Enumerating executable files in C

I am trying to enumerate only files with the execute (+x) bit set. My code seems to list all files. It also seems to enumerate directory and the above directory which I do not want. Example: .. should_not_be_executable.sh . Is there a way to filter '..' and '.' without strstr()?...

Remove specific (complex) line from MANY files (sed?)

Server was hacked, every php file on the server now starts with malicious code: <?php if(!isset($GLOBALS["\x61\156\x75\156\x61"])) { $ua=strtolower($_SERVER["\x48\124\x54\120\x5f\125\x53\105\x52\137\x41\107\x45\116\x54"]); if ((! strstr($ua,"\x6d\163\x69\145")) and (! strstr($ua,"\x72\166\x3a\61\x31"))) $GLOBALS["\x61\156\x75\156\x61"]=1; } ?><?php $zdsnpbzghe =...

How do I grep GZ files to extract PNG files?

Ok, so I have a tone of .GZ files in a folder, and I'm looking to recursively look through each one of them and extract all the PNG files into another destination folder. How would I do that? EDIT: I've been using this command from the terminal to find a...

Use Unix Executable File to Run Shell Script and MPKG File

I have 2 shell scripts and 2 mpkg installer, I am trying to use an unix excitable file to run them all. here is the script I have, but it always has error message "No such file or directory" ? #!/bin/sh # Find the absolute script current path path=$( cd...

Multiple line search in a file using java or unix command

I want to find the following pattern in a file like subclass "Pool1" 11:22:33:44:55:66 { dynamic; } the above pattern I have to find in a file. How can I find either using java or Unix command?...

Selecting unique lines based on two columns

I've been trying to figure out a way to select lines that are unique based on the values of two columns. For example, here is a sample/example of my file: chr1 10 12 chr1 10 12 chr1 10 11 chr1 9 12 chr2 15 20 And this is what I...

Unable to connect to mysql after automating mysql-server installation in bash script

Generates a random string: passdb="$(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-2};)" Feeds passdb to debconf i.e to set random passwords for mysql users: debconf-set-selections <<< 'mysql-server mysql-server/root_password password $passdb' debconf-set-selections <<< 'mysql-server mysql-server/root_password_again password $passdb' apt-get -y install mysql-server It get's installed successfully but I'm getting an error after...

grep string between two other strings as delimiters

I have to do a report on how many times a certain CSS class appears in the content of our pages (over 10k pages). The trouble is, the header and footer contains that class, so a grep returns every single page. So, how do I grep for content? EDIT: I...