regex,linux,awk , How to use awk regex sort by query string value?


How to use awk regex sort by query string value?

Question:

Tag: regex,linux,awk

I have a log file with example row:

xxx.xxx.xxx.xxx - - [07/Jun/2015:14:18:39 +0000] "GET /file/?t=70 HTTP/1.1" 200 35 "http://1234.com/p/talk-about-owning-it/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome"

The 7th column of each row looks like this:

/file/?t=70
/file/?t=4785&k=1
/file/?t=120
/file/?t=95&k=0
/file/?t=120
/file/?t=120&k=0
/file/?t=95&k=1
...

Output is arranged according to the number of lines in decreasing order containing unique values of t.

Desired OUTPUT:

120  -  3
95   -  2
4785 -  1
70   -  1
...

I am using awk but it is not giving desired output:

awk -F'[=&]' '{print $2}' /var/log/nginx/t.access.log | sort | uniq -c | sort -rn

It outputs all columns after 7th which is not required. What am I doing wrong? Any suggestions please.


Answer:

Using your 1 sample input line:

$ awk '{split($7,a,/[=&]/); print a[2]}' file | sort | uniq -c | sort -rn
      1 70

or if the rest of your input lines follow EXACTLY the format of that one line:

$ awk -F'[=& ]' '{print $8}' file | sort | uniq -c | sort -rn
      1 70

or entirely in awk:

$ cat tst.awk
{
    split($7,a,/[=&]/)
    sum[a[2]]++
}
END {
    PROCINFO["sorted_in"] = "@val_num_desc"
    for (val in sum) {
        print val "\t- " sum[val]
    }
}
$ awk -f tst.awk file
70      - 1

or:

$ cat tst.awk
BEGIN { FS="[=& ]" }
{ sum[$8]++ }
END {
    PROCINFO["sorted_in"] = "@val_num_desc"
    for (val in sum) {
        print val "\t- " sum[val]
    }
}
$
$ awk -f tst.awk file
70      - 1

The above uses GNU awk 4.* for PROCINFO["sorted_in"] to sort the output. If you don't have that, remove that line and pipe to sort -rn with appropriate args. You do not need the intermediate | sort | uniq -c either way.


Related:


Regex that allow void fractional part of number


c#,regex
@"[+-]?\d+(\.\d+)?" -this is a regex I have wrote for numbers it allows [+-] minus before the number digits before and digits after the point the question is how to change this to allow "not finished" values so that input of "5." - is fine too ?...

Regex to remove `.` from a sub-string enclosed in square brackets


c#,.net,regex,string,replace
I have this regex in C#: \[.+?\] This regex extracts the sub-strings enclosed between square brackets. But before doing that I want to remove . inside these sub-strings. For example, the string hello,[how are yo.u?]There are [300.2] billion stars in [Milkyw.?ay]. should become hello,[how are you?]There are [3002] billion stars...

Swing regular expression for phone number validation


java,regex
I want to validate phone number field in swing, so I am writing code to allow user to enter only digits, comma, spaces. For this I am using regular expression, when user enter characters or other than the pattern text field will consume. My code is not working. Can anyone...

Please can someone help me understand the exec method for regular expressions?


javascript,regex
The best place I have found for the exec method is Eloquent Javascript Chapter 9: "Regular expressions also have an exec (execute) method that will return null if no match was found and return an object with information about the match otherwise. An object returned from exec has an index...

Using an ad-hoc libc with a tool which is an argument of another tool


linux,shared-libraries
I need to use a particular libc to run a tool (cp). The problem is that this tool has to be used as argument of another tool (for example timeout) and I don't want to use the modified libc with this one. I tried to do: timeout 10 LD_LIBRARY_PATH=/path/to/mod/libc/ cp...

How do I isolate the text between 2 delimiters on the left and 7 delimiters on the right in Python?


python,regex,string,split
I have a string: string = ""7807161604","Sat Jan 16 00:00:57 +0000 2010","Global focus begins tonight. Pretty interested to hear more about it.","Madison Alabama","al","17428434","81","51","Sun Nov 16 21:46:24 +0000 2008","243" I only want the text: "Global focus begins tonight. Pretty interested to hear more about it."" which is between the 2nd and...

Python regular expression, matching the last word


python,regex,list
I've the following problem. I'm looking to find all words in a string that typically looks like so HelloWorldToYou Notice, each word is capitalized as a start followed by the next word and so on. I'm looking to create a list of words from it. So the final expected output...

How to create the javascript regular expression for number with some special symbols


javascript,regex
what can be the java-script regular expression which gives the numbers with some symbols For example following condition must be pass. Number can start with $ Can have the . or , : symbols between and % sign at the send. Passing valus: $233 48.23% 278 22.33 45:23 10,000 Number...

NASM: copying a pointer from a register to a buffer in .data


linux,assembly,nasm,x86-64
I am new to asm. I am trying to copy a pointer from a register to a .data variable using NASM, on linux 64-bit. Concider this program: section .data ptr: dq 0 section .text global _start _start: mov [ptr], rsp mov rax, 60 mov rdi, 0 syscall Here I try...

javascript replace dot (not period) character


javascript,regex,replace
I have some text being rendered that has an inline list with the bullet points notated by the "•" character, all on one line of text. It's ugly, but I don't want to change the source data. So I thought I'd just do a quick javascript replace to insert breaks...

How can I resolve the “Could not fix timestamps in …” “…Error: The requested feature is not implemented.”


linux,build,f#
I have been trying to build a project in F# on Linux that I have located here on github. It's a basic kata project that I am working on as a demo. However on Linux (specifically Ubuntu 14.04 LTS Desktop) I haven't been able to get it to build yet...

Identify that a string could be a datetime object


python,regex,algorithm,python-2.7,datetime
If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

Ignore first few lines and last few lines in a file Linux


linux,awk
I have a file like this and would like to print $0 except the first two and last three lines in linux. Tried awk command but no luck, is there any options I am using the following command - I suppose I am doing something wrong, but not able to...

How to match words in 2 list against another string of words without sub-string matching in Python?


python,regex,string,loops,twitter
I have 2 lists with keywords in them: slangNames = [Vikes, Demmies, D, MS Contin] riskNames = [enough, pop, final, stress, trade] i also have a dictionary called overallDict, that contains tweets. The key value pairs are {ID: Tweet text) For eg: {1:"Vikes is not enough for me", 2:"Demmies is...

Python match whole file name, not just extension


python,regex,nsregularexpression
Objective: Use Python regex to return a list of filenames have certain file extensions. List of files: X8SIA9.ROM X8SIA0.rom X8SIA0.928 Regex to find files that end in *.ROM or 3x numbers: bios_file = re.findall(r'.*\.(rom|[0-9]{3})+', name, re.I) Issue: bios_file is returning ['ROM'], ['rom'], [928]. bios_file should be returning the entire filename...

Get number from string


regex
I am trying to get the enclosed number between two slashes in a URL using regex. The code regex I have is not working, I am fairly new to regex and don't really understand it. The regex: http:\/\/?www\.?example\.com\/g\/(^\d$)\/\w The URL: http://www.example.com/g/1337/Game-Title Trying to get the "1337", which is the PlaceId....

REGEX python find previous string


python,regex,string
I'm trying to find if the last word of the string is followed by a space or a special char, and if yes return the string without this space/special char For example : "do you love dogs ?" ==> return "do you love dogs" "i love my dog " (space...

Get all prices with $ from string into an array in Javascript


javascript,regex,currency
var string = 'Our Prices are $355.00 and $550, down form $999.00'; How can I get those 3 prices into an array?...

ret_from_syscall source code and when it is called


linux,linux-kernel,kernel,linux-device-driver,system-calls
In the below call trace we see function called ret_from_syscall. Which function is this ? When it will called during system call ? Where is the corresponding code for this ? May 7 16:40:34.322086 warn TCU-0 kernel: [cf83ddc0] [00009751] 0x9751 (unreliable) May 7 16:40:34.322086 warn TCU-0 kernel: [cf83ddd0] [c00469ac] do_syslog+0x198/0x424...

Regex with whitespaces and preceding zeros


regex,sas
I want to match the string 11 with a regular Expression in SAS. The 11 can be preceded by zero or more 0 and/or by white spaces. Any other character is not allowed. Likewise, if anything there should only be white spaces following the 11. Examples: Match: 0000011 11 11<space><space>...

Force linux to use php as php55


php,linux,fedora
Helllo, I have a little problem here. I have PHP 5.3 installed and it's accessible via php command, I also have php 5.5 accessible as php55 command. Now I need to force linux to use php55 when I write php command. Simple way: How I switch the versions of php?...

AWK write to new column base on if else of other column


linux,bash,shell,awk,sed
I have a CSV file with columns A,B,C,D. Column D contains values on a scale of 0 to 1. I want to use AWK to write to a new column E base in values in column D. For example: if value in column D <0.7, value in column E =...

How to write RegEx for inserting line break for line length more than 30 characters?


regex
I am using a text editor which lets use regular expression to find / replace text. I have a large text file. I want to insert new line in each lines which are more than 30 characters. I want the line to break after 30th character (doesnt matter if a...

Regular Expression for whole world


regex,c#-4.0,vb6
First of all, I use C# 4.0 to parse the code of a VB6 application. I have some old VB6 code and about 500+ copies of it. And I use a regular expression to grab all kinds of global variables from the code. The code is described as "Yuck" and...

Java - Enforce TextField Format - UX - 00:00:00;00


java,regex,user-interface
Please see below image of the UI of my current application Currently, I am enforcing Data Format of the text fields in the back end. If the RegEx does match (00:00:00;00), it will throw an error. My question is from a UX perspective, is it possible to make the input...

How many characters are visible like a space, but are not space characters?


php,regex
If I want to discover the hexadecimal equivalent of a space in PHP I can play with bin2hex: php > echo var_dump(bin2hex(" ")); string(2) "20" I can also obtain space character from "20" php > echo var_dump(hex2bin("20")); string(1) " " But there exist Unicode versions of a "visible" space: php...

Finding embeded xpaths in a String


java,regex
I have a string where I have the user should be able to specify xpaths that will be evaluated at runtime. I was thinking about having a the following way to specify it. String = "Hi my name is (/message/user) how can i help you with (/message/message) "; How can...

match line break except line begin with spcific word or blank line


regex,notepad++
If I have text that the line breaks is broken: Chapter 1 Lorem ipsum dolor sit amet, consectetur adipisci ng elit, sed do eiusmod tempor incididunt ut la bore et dolore magna aliqua. Ut enim ad minim ve niam, quis nostrud exercitation ullamco labo ris nisi ut aliquip ex ea...

Regex pass dynamic values with boundry


c#,regex,string,boundary
I'm trying to pass a dynamic value at runtime with a boundary \b to a Regex function. My code is: static void Main(string[] args) { string sent = "Accelerometer, gyro, proximity, compass, barometer, gesture, heart rate"; string match = "gyro"; string bound = @"\b"; if (Regex.IsMatch(sent, @"\bgyro", RegexOptions.IgnoreCase)) { Console.WriteLine("match...

While loop in bash using variable from txt file


linux,bash,rhel
I am new to bash and writing a script to read variables that is stored on each line of a text file (there are thousands of these variables). So I tried to write a script that would read the lines and automatically output the solution to the screen and save...

How to Match a string with the format: “20959WC-01” in php?


php,regex
i want to restrict a user to enter a value which is similar to the value "20959WC-01", means it must contains 5 integers followed by two character, a '-' and two integers, can anyone please give me a solution to sort out this problem. Thanks in advance :) ...

regex - Match filename with or without extension


regex,logstash-grok
Need a regex pattern to match all of the following: hello hello. hello.cc I tried \b\w+\.?\w+?\b, but this doesn't match "hello." (the second string mentioned above)....

BeautifulSoup: Parsing bad Wordpress HTML


python,html,regex,wordpress,beautifulsoup
So I need to scrape some a site using Python but the problem is that the markup is random, unstructured, and proving hard to work with. For example <p style='font-size: 24px;'> <strong>Title A</strong> </p> <p> <strong> First Subtitle of Title A </strong> "Text for first subtitle" </p> Then it will...

Bash modify CSV to change a field


linux,bash,awk
I have a very big CSV file (aprox. 10.000 rows and 400 columns) and I need to modify certain columns (like 15, 156, 220) to change format from 20140321132233 to 2014-03-21 13:22:33. All fields that I need to modify are datetime. I saw some examples using awk but for math...

What are correct permissions for Linux Apache2 PHP 5.3 log file?


php,linux,apache,logging,permissions
I discovered the reason why I was not getting entries into my php_errors.log file related to permissions. Right now, I have set it to 666 (rw-rw-rw-) but surely this is a security weakness? Thus, my question. php.ini file: error_log /var/log/httpd/php_errors.log log_errors On # ls -ld /var/log /var/log/httpd /var/log/httpd/php_errors.log drwxr-xr-x 6...

Regex not working in HTML5 pattern


regex,html5
So I have this regex intended to let pass all text but those that contain as initial chars the "34" sequence: ^(?!34)(?=([\w]+)) The regex is working fine for me in https://regex101.com/r/iN1yN3/2 , check the tests to see the intended behavior. Any Idea why it isn't working in my form? <form>...

Extracting strings from HTML with Python wont work with regex or BeautifulSoup


python,regex,parsing,beautifulsoup,python-requests
Im using Python 2.7, BeautifulSoup4, regex, and requests on windows 7. I've scraped some code from a website and I am having problems parsing and extracting the bits I want and storing them in a dictionary. What I'm after is text that is presented as follows in the code: @CAD_DTA\">I...

PHP Regular Expressions Counting starting consonants in a string


php,regex
I need to find out how many starting consonants a word has. The number is used later in the program. The code below does work, I am wondering if it is possible to do this with a regular expression. $mystring ="SomeStringExample"; $mystring2 =("bcdfghjklmnpqrstvwxyzABCDFGHJKLMNPQRSTWVXYZ"); $var = strspn($mystring, $mystring2); Using a regular...

What does it indicate if /proc/PID/maps shows zero for all addresses?


linux,linux-kernel
I'm debugging a problem with a Linux DNS server. Curiously, when I look at /proc/PID/maps for the DNS server process, this is what I get: 00000000-00000000 r-xp 00000000 00:0e 2344 /usr/sbin/unbound 00000000-00000000 rw-p 00000000 00:0e 2344 /usr/sbin/unbound 00000000-00000000 ---p 00000000 00:00 0 00000000-00000000 rw-p 00000000 00:00 0 [heap] 00000000-00000000 rw-p...

MySQL substring match using regular expression; substring contain 'man' not 'woman'


mysql,regex
I have an issue while I fetch data from database using regular expression. While I search for 'man' in tags it returns tags contains 'woman' too; because its substring. SELECT '#hellowomanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 correct, it contains 'woman' SELECT '#helloowmanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 incorrect, it can...

Reg ex matching a word


regex
I need to match only first two files, out of four files listed below: ABD_DEF_GHIJ_20150611 ABD_DEF_GHIJ ABD_DEF_GHIJ_FX_20150611 ABD_DEF_GHIJ_FX I am using reg ex - ABD_DEF_GHIJ(_\d{8}|\b) and it's working fine. I would like to know if my solution is ok or there is any better alternate solution....

jQuery / Regex: How to compare string against several substrings


jquery,regex,string,substring,substr
I have a special validation need where I need to check if a string contains ANY out of several substrings like the following list: 12 23 34 45 56 67 78 89 90 01 I am new to jQuery and the only thing I could think of here is the...

Match a pattern preceded by a specific pattern without using a lookbehind


regex,eclipse,lookahead
Is there a way to match a B only if preceded by an A? The A can be at any position behind the B, with any amount of characters between. Examples: A_B (Matches `B`) C_B (No match) I've tried: (?=A)[^B]*B But it matches all the characters preceeding B as well....

Warning: preg_match_all(): Unknown modifier '\' [duplicate]


php,regex,warnings
This question already has an answer here: PHP - preg_match and “Unknown modifier” error 3 answers I know there are tons of duplicate question like this. But I am stuck here. Can anyone help me out? Here's the code: <?php function count_images($string){ preg_match_all('/[^/\s]+/\S+\.(jpg|png|gif)/', $string, $results); return count($results[1]); } ?>...

Syncing Vagrant VMs across different physical servers


linux,vagrant,backup,virtual-machine,sync
I'm using Vagrant to deploy my VMs and my current setup looks like this: server1 = VM1, VM2, VM3 ( main production server ) server2 = VM1, VM2, VM3 ( backup server ) My questions is, can I somehow sync the VMs across the different physical servers in case one...

Validate part of mail suffix


c#,regex
I'm validating mail addresses that could look something like this: [email protected] but also [email protected] I would like to check whether if xx is present or not. The first thing that came to mind was String.Contains, but it would obviously match any occurrences of xx in the string. Is regexp the...

Store regex pattern as a string in PHP when regex pattern contains both single and double quotes


php,regex
UPDATE: It turned out that what broke my reg expressions after escaping the quotes was the / deliminator terminating the expression early. I'm attempting to store a regex pattern as a string like so: $link_pattern = "/<a\s+href=(?:"([^"]+)"|'([^']+)').*?>(.*?)</a>/"; However I'm unable to properly encapsulate it since the regex pattern contains both...

Does there exist an algorithm for iterating through all strings that conform to a particular regex?


c#,regex,algorithm
I'm making a script to try and hack into an account whose login password is at least 8 characters long and includes at least 1 number, 1 special character and 1 capital letter. I will use brute force. Is there a compact, elegant and efficient way to iterate through every...

Django MySQLClient pip compile failure on Linux


python,linux,django,gcc,pip
Django documentation as of v1.8 recommends using mysqlclient connector for the framework. I'm attempting to pip install the package on Ubuntu 14.04 with Python 3.4 and running into a GCC error that I'm unable to find reference to. I'm not an expert on compiling software, so was hoping somebody can...

Delete some lines from text using Linux command


linux,shell,sed,grep,pattern-matching
I know how to match text using regex patterns but not how to manipulate them. I have used grep to match and extract lines from a text file, but I want to remove those lines from the text. How can I achieve this without having to write a python or...