shell,command-line,awk,terminal , Using a command-line utility to perform the following map-updates


Using a command-line utility to perform the following map-updates

Question:

Tag: shell,command-line,awk,terminal

I'm a complete newbie to using command-line utilities and am wondering how to process information as following:

mapping.txt:

80  001 002
81  011 012 013 014
82  021 022
...

input.txt:

81  103823044
80  103823054
81  103823064
...

Desired output.txt:

103823044|011|
103823044|012|
103823044|013|
103823044|014|
103823054|001|
103823054|002|
103823064|011|
103823064|012|
103823064|013|
103823064|014|

I've done simple mapping wherein the column numbers are fixed but I'm unsure of how to map a dynamic number of columns to the desired output


Answer:

If order is not important, join and awk can do the job easily.

$ join <(sort input.txt) <(sort mapping.txt) | awk -v OFS="|" '{for (i=3;i<NF;i++) print $2, $i OFS}'
103823054|001|
103823044|011|
103823044|012|
103823044|013|
103823064|011|
103823064|012|
103823064|013|

Related:


Group of data in shell language


linux,bash,shell
I wonder if the data in the group aren’t numbers ; the script works? When x="1 2 3" I can do a for loop: for i in $x I wonder if this is correct : read x read y xm=`cat $x` ym=`cat $y` zm="$xm $ym" for i in $zm do...

Using the Rails console to delete odd-numbered records


ruby-on-rails-4,command-line,console
I'd like to delete records with odd numbered ID's from Hero table using the Rails console. Would someone be so kind as to suggest a way for me to do this? Kind Regards, Ryan...

Redirect output from file to stdout


bash,shell,unix,stdout
I have a program that can output its results only to files, with -o option. This time I need to output it to console, i.e. stdout. Here is my first try: myprog -o /dev/stdout input_file But it says: /dev/ not writable I've found this question that's similar to mine, but...

Allowing an executable to be terminated w/ ctrl+c in a shell script w/o exiting


bash,shell
Example: #!/bin/bash command_a # starting a executable command_b # should be executed after after exiting A A is exited via ctrl+C. I didn't really know how to search for this....

How to extract single-/multiline regex-matching items from an unpredictably formatted file and put each one in a single line into output file?


linux,shell,unix,replace,grep
I have a very huge file which looks like this: <a>text</a>text blah <b>data1</b>abc<b>data2</b> <b>data3</b>blahblah <c>text</c> <d>text</d> <x>blahblah<b>data4 data5 data6</b> <b>data7 </x> That is, its formatting is unpredictable. I need to extract each <b>...</b> item (it might contain multiline text!) and put every one of them in a single separate line....

How to set the classpath in Windows Command Line correctly


java,windows,command-line,classnotfoundexception
I have made many attempts to enter a command to run a JAR file correctly from the terminal, and I am 100% sure that all the JAR files and such are in the given paths, but I keep getting a ClassDefNotFoundException. Does it have anything to do with the way...

How to store command line arguments for future use?


bash,shell
I have a big bash script (version 3.2.51(1)-release) and I need to store command line arguments for use in the script. I checked many threads here and people suggested to use "[email protected]" to access all arguments. If I run this script (test.sh) - #!/bin/bash echo "[email protected]"; $> ./test.sh 1 2...

AWK write to new column base on if else of other column


linux,bash,shell,awk,sed
I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

Sqoop Export with Missing Data


sql,postgresql,shell,hadoop,sqoop
I am trying to use Sqoop to export data from HDFS into Postgresql. However, I receive an error partially through the export that it can't parse the input. I manually went into the file I was exporting and saw that this row had two columns missing. I have tried a...

Using the shell provided with NodeJS


javascript,node.js,shell,require,node-modules
Once you've installed NodeJS, you'll have a executable in your computer named NodeJS which is a shell. I was wondering what can I do with that... here you're able to run JS code as, for example, in the browser's console, great. Now, is it possible to require modules in that...

Multiple line search in a file using java or unix command


java,shell,unix,command
I want to find the following pattern in a file like subclass "Pool1" 11:22:33:44:55:66 { dynamic; } the above pattern I have to find in a file. How can I find either using java or Unix command?...

Calling find more than once on the same folder tree


linux,bash,shell,unix,find
I'm running a find command multiple times on the same group of files. The results of my find commands are usually disjoint sets, AKA I'm running find -mmin +35; find -mmin -25, and doing different things to the results. It seems sort of silly to search through the entire file...

Using AWK to parse fields with commas


regex,bash,shell,awk
Edited - TLDR: Using awk to parse fields that include commas. # original config file - confile1 $ cat confile1 list=( app1,"HOSTNAME - port - application name - alert1",99.0,99.0 app2,"HOSTNAME - port - application name - alert1",99.0,99.0 app3,"HOSTNAME - port - service name - alert2",99.0,99.0 web1,"URL - HOSTNAMES(01,02) - http://someurl.com/...

Why can I view some Unix executable files in Mac OS X and not others?


git,bash,shell,unix,binary
I am on a Macbook Pro on Mac OS X 10.10 (Yosemite). When I go to /usr/bin, git is there as a unix executable file. When I open it up in Sublime Text, all I get is unreadable machine code. However, when I open up a different Unix executable file—in...

Delete some lines from text using Linux command


linux,shell,sed,grep,pattern-matching
I know how to match text using regex patterns but not how to manipulate them. I have used grep to match and extract lines from a text file, but I want to remove those lines from the text. How can I achieve this without having to write a python or...

How to test if a command is a shell reserved word?


bash,shell
I am writing a bash script and I would like to verify if a string is a shell reserved word (like if, for, alias, etc...). How can I can do this?...

shell script cut from variables


bash,shell,shellcode
The file is like this aaa&123 bbb&234 ccc&345 aaa&456 aaa$567 bbb&678 I want to output:(contain "aaa" and text after &) 123 456 I want to do in in shell script, Follow code be consider #!/bin/bash raw=$(grep 'aaa' 1.txt) var=$(cut -f2 -d"&" "$raw") echo $var It give me a error like...

shell script to shutdown/restart Linux system


linux,shell
Is there any suitable shell script for shutting down or restarting a Linux machine? I have tried a shell script for shutdown, but when I enter sudo shutdown it will ask for the password. How we can enter the password using the script?

Missing one condition to display a string


shell
I am working on a small script that simply takes a parameter and read the output of ls -l and displays the user and name of the file that start with that parameter. For exemple : $> ls -l | ./script.sh "ok" John ok_file Mark ok_test Here is what the...

AWK count number of times a term appear with respect to other columns


linux,shell,command-line,awk,sed
Given a CSV file: id, fruit, binary 1, apple, 1 2, orange, 0 3, pear, 1 4, apple, 0 5, peach, 0 6, apple, 1 How can i calculate for each unique values in fruit, the number of times the binary value =1 / number of occurences of that fruit...

Why does `su` not work in Jenkins?


shell,jenkins,su
su doesn't seem to work as intended when run in Jenkins build job script. su username1234 whoami returns jenkins My initial guess is that each line of the Jenkins shell script is a new shell. But doing repeated echo $$ on separate lines shows that's not the case. ...

Export database from a remote server into a specific folder on my computer


php,mysql,database,shell,command-line-interface
I'm having trouble with a database export that is too big for phpmyadmin to handle. I'm curious what the syntax would be for exporting a mysql database from a remote server and getting the export file into a specific folder on my own computer, all using the command line. Say...

Use Unix Executable File to Run Shell Script and MPKG File


osx,shell,unix
I have 2 shell scripts and 2 mpkg installer, I am trying to use an unix excitable file to run them all. here is the script I have, but it always has error message "No such file or directory" ? #!/bin/sh # Find the absolute script current path path=$( cd...

Shell script to loop over files with same names but different extensions


linux,bash,shell
For example say I have: filename1.ext1 filename1.ext2 filename2.ext1 filename2.ext2 I need to write a shell script to feed these files into a program like so: program filename1.ext1 filename1.ext2 program filename2.ext1 filename2.ext2 Additionally the .ext1 files must be entered first and the .ext2 files second. Any help would be appreciated....

linux - running a process and tailing a file simultaneously


bash,shell,tail
I want a run a long task on a remote machine (with python fabric using ssh). It logs to a file on the remote machine. What I want to do is to run that script and tail (actively display) the log file content until the script execution ends. The problem...

How to uninstall a program using C#? [duplicate]


c#,visual-studio,visual-studio-2012,command-line,windows-installer
This question already has an answer here: How to uninstall MSI using its Product Code in c# 4 answers I'm trying to uninstall a program using C# via Visual Studio and possibly CMD. I made several attempts but could not getting anything to go. Attempt #1: RegistryKey localMachine =...

running testng.xml via command line- error Cannot find class in classpath: com.companyname.SSProject.LaunchSSProject


command-line,testng
I am using following command on command line: C:\Users\MyScriptDirectory>java -classpath ".;Lib\selenium-2.45.0\libs\testng-6.8.5.jar" org.testng.TestNG testng.xml Getting error message: [TestNG] [Error] Cannot find class in classpath: com.companyname.SSProject.LaunchSSProject I have made following entry in testng.xml: <class name="com.companyname.SSProject.LaunchSSProject" /> "com.companyname.SSProject" is my package having LaunchSSProject.java file inside it When I am running the same testng.xml...

Check for decimal point and add it at the end if its not there using awk/perl


regex,perl,shell,awk
I have test.dat file with values given below: 20150202,abc,,,,3625.300000,,,,,-5,,,,,,,,,,,,,,,,,,,,,, 20150202,def,,,,32.585,,,,,0,,,,,,,,,,,,,,,,,,,,,, 20150202,xyz,,,,12,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, My expected output is shown below: 20150202,abc,,,,3625.300000,,,,,-5.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,def,,,,32.585,,,,,0.,,,,,,,,,,,,,,,,,,,,,, ^. added here 20150202,xyz,,,,12.,,,,,0.004167,,,,,,,,,,,,,,,,,,,,,, ^. added here So if column 6 and 11 doesn't have decimal point in it, then we should add '.' at the end of...

Identifying when a file is changed- Bash


bash,shell,unix
So in my bash shell script, I have it running through a for loop. Inside the for loop, I use find "$myarray[i]" >> $tmp to look for a certain directory each time through the loop. Sometimes, it finds the variable in myarray[i] and sometimes it doesn't. When it does find...

How to replace newlines/linebreaks with a single space, but only if they are inside of start/end regexes?


regex,linux,shell,unix,replace
I have an extremely big (many GBs in size) file that looks like [x data1 data2 data3 data4 y] [a data5 data 6 data7 data 8 b> [x data y] ...and so on How to replace newlines (which might be surrounded by spaces) with a single space, but only if...

using sed to replace a line with back slashes in a shell script


regex,bash,shell,ssh,sed
I am trying to replace the bottom one of these 2 lines with sed in a file. <rule>out_prefix=orderid ^1\\d\+ updatemtnotif/</rule>\n\ <rule>out_prefix=orderid ^2\\d\+ updatemtnotif/</rule>\n\ And the following command seems to do that when executed as a command at the bash prompt sed -i '[email protected]_prefix=orderid ^2\\\\d\\+ updatemtnotif/@out_prefix=orderid ^2\\\\d\\+ updatemtnotif_fr/@g' /opt/temp/rules.txt however, when...

how to spy on linux binaries for testing of shell scripts


linux,bash,shell,testing,spy
sorry if this is a duplicate, i had no idea what to search for... my use case is more complex, but can be narrowed down to the following problem: i want to run a bash script, which invokes all sorts of binaries, for example: grep. i want to assert that...

Linux-wget command


linux,shell,wget
I need a quick help on customizing my wget command in a shell script: The wget command looks something like this: wget http://infamvn:8081/nexus/content/groups/LDM_REPO_LIN64/com/infa/com.infa. products.ldm.ingestion.server.scala/10.0.0.135.527-SNAPSHOT/com.infa.products.ldm.ingestion.server.scala-10.0.0.135.527-20150622.210643-1-sources.jar Here I'd like to add the 10.0.0.135.527 in a variable, so I created a script something like this: n = 10.0.0.135.527 wget...

If exist and errorlevels in a batch (.bat) file


windows,batch-file,command-line
I'm trying to run a delete command on the result of an SQL command but only if it returns an error code of 0. Below is my code: SqlCmd command... REM if SqlCmd command is successful run the below if exist statement if errorlevel 0 ( REM if the file...

Mac OSX - Allow for user input in shell script via GUI or Prompt


osx,bash,shell
first time post so please let me know how I could improve.. I created a shell script which requires a person to input their name and then generates a report. The script works as needed when chmod'd into an executable script and run from terminal. But, I would like to...

command line Start.Info.Argument error


c#,command-line,wix,heat
I would like run this command with C# : "%WIX%/bin/heat.exe" dir "C:\Documents and Settings\APP" -gg -sfrag -cg Appli -out wixappli.wxs I have this code (Program.cs) : System.Diagnostics.Process process1 = new System.Diagnostics.Process(); process1.StartInfo = new System.Diagnostics.ProcessStartInfo("cmd.exe"); process1.StartInfo.Arguments = String.Format("/k \"%WIX%/bin/heat.exe\" dir \"{0}\" -gg -sfrag -cg Appli -out wixappli.wxs ", @"C:\Documents and...

How do I silence the HEAD of a curl request while using the silent flag?


bash,shell,curl,command-line,pipe
When I run the curl command and direct the data to a file, I get back the content of the site as expected. $ curl "www.site.com" > file.txt $ head file.txt Top of site ... However, this command shows a progress bar, which I do not want: % Total %...

oh-my-zsh error after upgrade: ~/.oh-my-zsh/lib/misc.zsh:3: parse error near `then'


shell,zsh,zshrc,oh-my-zsh
today I upgrade my oh-my-zsh and then start this error: ~/.oh-my-zsh/lib/misc.zsh:3: parse error near `then' Inside this file there is this shell loop... ## Load smart urls if available for d in $fpath; do if [[ -e "$url/d-quote-magic"]]; then autoload -U url-quote-magic zle -N self-insert url-quote-magic fi done I think...

How to release binaries on Github for different plattforms as seperate downloads?


git,github,command-line,software-distribution
I found this at github https://github.com/blog/1547-release-your-software. Their page provide a way to attach binaries to a special release. My build script compiles binaries for Windows MacOS and Linux. How to supply seperate downloads for different platforms there? AS far I can see these releases are simple git tags. Is it...

Shell script file, checks if text file is empty or not


shell,text-files
I want to check if .todo.txt is empty or not. If is is empty, I want to add the echo in the file, and if it already has content, I want to do something else. I tried: hasData() { echo "Hello" } isEmpty() { echo -e "$n\t$comment\t$dueD/$dueM/$dueY " >> .todo.txt...

how to deletes line from a text file that are taken from another file [duplicate]


shell,awk,sed,grep,sh
This question already has an answer here: Remove duplicates from text file based on second text file 4 answers I have a data.txt file with a lot of lines in it and a lines.txt that contains some lines. I want to delete all lines from data.txt that match any...

find numbers divisible by 3 in csv file using shell script


bash,shell,unix,awk
I have csv file having content like below : 1|2|3 4|5|6 7|8|9 Now I would like to find the numbers which are divisible by 3 using shell scripting. I would like to use awk command for this. I am learning shell scripting. So could you please help me out to...

Reading tab (\t) separated text from file in shell


bash,shell,text-files
my text file looks like this 1 get a 06-05-2000 2 get b 06-05-2001 3 get c 06-05-2002 4 get d 06-05-1442 5 get e 06-05-1998 6 get f 06-05-1909 I want to read what is after the first \t (tab) and store that in varA and what is after...

How do I align my output?


shell
I'm writing a script that takes a parameter and reads the output of ls -l. It then displays the user and the name of the file STARTING with that parameter. For example : $> ls -l | ./script.sh "o" John ok_test RandomUser o_file My script works just fine, but I...

Matching string inside file and returning result


regex,string,bash,shell,grep
I've got a few peculiar issues with trying to search for a string inside of a .db file. The way I tried was by using grep, which does apparently find the string(s), although this is the output: $ grep "ext" *.db Binary file enormous.db matches There are a couple problems...

Increment Serial Number using EXIF


windows,powershell,command-line,exif,exiftool
I am using ExifTool to change the camera body serial number to be a unique serial number for each image in a group of images numbering several hundred. The camera body serial number is being used as a second place, in addition to where the serial number for the image...

Set Java path in command line for only one directory


java,command-line,command-prompt
I am running a program that utilizes Scala 2.10 for work and is not compatible with Java 8, only Java 7. In a Windows 7 command line, how can I set the java path to use Java 7 ONLY for that directory?

How to extract first letters of dashed separated words in a bash variable?


linux,string,bash,shell,variables
I would like to extract the first letter of dashed separated words value of my bash variable, like this: MY_TEXT=this-is-my-custom-text I would like to create a second variable like this: MY_INITIALS=timct...