regex,encoding,awk,utf-8,scripting , Detecting corrupt characters in UTF-8 encoded text file


Detecting corrupt characters in UTF-8 encoded text file

Question:

Tag: regex,encoding,awk,utf-8,scripting

I have a text file that was edited with the wrong character encoding and thus has some mojibake and corrupt characters in some of the strings when I open it using UTF-8. What scripting language would be the most efficient at detecting these corrupt characters? Perl is not an option. I am basically trying to find a way to scan through a text file using a script and output the line numbers and possibly offset where a corrupted character is found. How do I go about this? I was thinking about using AWk, but I don't know what regular expression to use in searching for the corrupted characters. If I could be pointed in the right direction, that would be great.

More Comprehensive Input:

I want the script to tell me the line number that has the corrupted characters which would be the fifth line in the above example. Also, there are different languages in the text file. I have English Chinese, French, Spanish, Russian, Portuguese, Turkish, French_Euro, German, Dutch, Flemish, Korean, Portuguese_Moz. And I have a few special characters also like # and ! and ***

I used this if statement to get the above output:

if($1 ~ /[^\x00-\x7F]/){
print NR ":" , $0 > "output.txt";
count++;
}

Answer:

This finds all chars outside of the ASCII range:

$ awk '/[^\x00-\x7F]/{ print NR ":", $0 }' file
1: Interruptor EC não está em DESLOCAR
4: 辅助驾驶室门关闭
5: Porte cab. aux. fermée
7: Дверь аппаратной камеры закрыта
13: 高压ä¿æŠ¤æ‰‹æŸ„å‘下
14: Barrière descendue
16: Огранич. Планка ВВК опущ.
19: Barra de separação descida
22: DP未å¯åŠ¨
23: Puiss. rép. non activée
25: !!! ВнешнÑÑ Ð¼Ð¾Ñ‰Ð½Ð¾ÑÑ‚ÑŒ не включена
26: Potência Dist Não Ativada
28: Potência dist não activada
31: 机车未移动
33: Motor no se está moviendo
34: Локомотив неподвижен
35: Auto Não se Movendo
37: A não se move
40: 机车状况å…许自动åœæœº
41: Conditions auto\npermettent arrêt auto
43: УÑтановки локомотива\nПредуÑматривают Ð     °Ð²Ñ‚оматичеÑкую оÑтановку
44: Condições da moto\nPermitem Auto Parada

Is that good enough? If not please edit your question to show more comprehensive sample input including cases for which the above does not work.


Related:


Identify that a string could be a datetime object


python,regex,algorithm,python-2.7,datetime
If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

Regular expression to get url in string swift


ios,regex,swift
I try to get urls in text. So befor I used such expression let re = NSRegularExpression(pattern: "https?:\\/.*", options: nil, error: nil)! But I had a problem, when user input urls with Capitalized symbols like it doesn't match it ( I tried let re = NSRegularExpression(pattern: "(h|H)(t|T)(t|T)(p|P)s?:\\/.*", options: nil, error:...

Store regex pattern as a string in PHP when regex pattern contains both single and double quotes


php,regex
UPDATE: It turned out that what broke my reg expressions after escaping the quotes was the / deliminator terminating the expression early. I'm attempting to store a regex pattern as a string like so: $link_pattern = "/<a\s+href=(?:"([^"]+)"|'([^']+)').*?>(.*?)</a>/"; However I'm unable to properly encapsulate it since the regex pattern contains both...

JavaScript Regex: Escape the string “c++”?


javascript,regex,escaping
I'm working on some regex for a input text field and have had some trouble escaping special characters such as "+" or "?". I've used these two questions for doing so and they work for a string such as c+ but if I enter c++ I get following error in...

Python regular expression, matching the last word


python,regex,list
I've the following problem. I'm looking to find all words in a string that typically looks like so HelloWorldToYou Notice, each word is capitalized as a start followed by the next word and so on. I'm looking to create a list of words from it. So the final expected output...

Finding embeded xpaths in a String


java,regex
I have a string where I have the user should be able to specify xpaths that will be evaluated at runtime. I was thinking about having a the following way to specify it. String = "Hi my name is (/message/user) how can i help you with (/message/message) "; How can...

javascript replace dot (not period) character


javascript,regex,replace
I have some text being rendered that has an inline list with the bullet points notated by the "•" character, all on one line of text. It's ugly, but I don't want to change the source data. So I thought I'd just do a quick javascript replace to insert breaks...

Regex pass dynamic values with boundry


c#,regex,string,boundary
I'm trying to pass a dynamic value at runtime with a boundary \b to a Regex function. My code is: static void Main(string[] args) { string sent = "Accelerometer, gyro, proximity, compass, barometer, gesture, heart rate"; string match = "gyro"; string bound = @"\b"; if (Regex.IsMatch(sent, @"\bgyro", RegexOptions.IgnoreCase)) { Console.WriteLine("match...

match line break except line begin with spcific word or blank line


regex,notepad++
If I have text that the line breaks is broken: Chapter 1 Lorem ipsum dolor sit amet, consectetur adipisci ng elit, sed do eiusmod tempor incididunt ut la bore et dolore magna aliqua. Ut enim ad minim ve niam, quis nostrud exercitation ullamco labo ris nisi ut aliquip ex ea...

Python match whole file name, not just extension


python,regex,nsregularexpression
Objective: Use Python regex to return a list of filenames have certain file extensions. List of files: X8SIA9.ROM X8SIA0.rom X8SIA0.928 Regex to find files that end in *.ROM or 3x numbers: bios_file = re.findall(r'.*\.(rom|[0-9]{3})+', name, re.I) Issue: bios_file is returning ['ROM'], ['rom'], [928]. bios_file should be returning the entire filename...

Pharo punctuation marks [duplicate]


regex,pharo,punctuation
This question already has an answer here: Learning Regular Expressions [closed] 1 answer Is there any way to capture all the places in a String, where there are punctuation marks like '.,:;!?' Basically, am looking for a regex that captures all the above....

How do I isolate the text between 2 delimiters on the left and 7 delimiters on the right in Python?


python,regex,string,split
I have a string: string = ""7807161604","Sat Jan 16 00:00:57 +0000 2010","Global focus begins tonight. Pretty interested to hear more about it.","Madison Alabama","al","17428434","81","51","Sun Nov 16 21:46:24 +0000 2008","243" I only want the text: "Global focus begins tonight. Pretty interested to hear more about it."" which is between the 2nd and...

Regex not working in HTML5 pattern


regex,html5
So I have this regex intended to let pass all text but those that contain as initial chars the "34" sequence: ^(?!34)(?=([\w]+)) The regex is working fine for me in https://regex101.com/r/iN1yN3/2 , check the tests to see the intended behavior. Any Idea why it isn't working in my form? <form>...

Regular Expression for whole world


regex,c#-4.0,vb6
First of all, I use C# 4.0 to parse the code of a VB6 application. I have some old VB6 code and about 500+ copies of it. And I use a regular expression to grab all kinds of global variables from the code. The code is described as "Yuck" and...

Remove plus sign from url using .htaccess


php,regex,apache,.htaccess,mod-rewrite
My current url is: http://localhost/mywbsite/search/one+piece/149 I want to change link to like this http://localhost/mywbsite/search/one-piece/149 I have already .htaccess file with this code Options -MultiViews RewriteEngine On RewriteBase /indianrealitybytes/ RewriteCond %{THE_REQUEST} /search_advance\.php\?keywords=([^&]+)&f=([^\s&]+) [NC] RewriteRule ^ search/%1/%2? [R=301,L] RewriteRule ^search/([^/]+)/([^/]+)/?$ search_advance.php?keywords=$1&f=$2 [QSA,L,NC] I don't know where to change to remove plus(+) sign...

Validate part of mail suffix


c#,regex
I'm validating mail addresses that could look something like this: [email protected] but also [email protected] I would like to check whether if xx is present or not. The first thing that came to mind was String.Contains, but it would obviously match any occurrences of xx in the string. Is regexp the...

How to Match a string with the format: “20959WC-01” in php?


php,regex
i want to restrict a user to enter a value which is similar to the value "20959WC-01", means it must contains 5 integers followed by two character, a '-' and two integers, can anyone please give me a solution to sort out this problem. Thanks in advance :) ...

BeautifulSoup: Parsing bad Wordpress HTML


python,html,regex,wordpress,beautifulsoup
So I need to scrape some a site using Python but the problem is that the markup is random, unstructured, and proving hard to work with. For example <p style='font-size: 24px;'> <strong>Title A</strong> </p> <p> <strong> First Subtitle of Title A </strong> "Text for first subtitle" </p> Then it will...

Regex to remove `.` from a sub-string enclosed in square brackets


c#,.net,regex,string,replace
I have this regex in C#: \[.+?\] This regex extracts the sub-strings enclosed between square brackets. But before doing that I want to remove . inside these sub-strings. For example, the string hello,[how are yo.u?]There are [300.2] billion stars in [Milkyw.?ay]. should become hello,[how are you?]There are [3002] billion stars...

regex - Match filename with or without extension


regex,logstash-grok
Need a regex pattern to match all of the following: hello hello. hello.cc I tried \b\w+\.?\w+?\b, but this doesn't match "hello." (the second string mentioned above)....

Split by a comma that is not inside parentheses, skipping anything inside them


java,regex,string,split
I know it might be another topic about regexes, but despite I searched it, I couldn't get the clear answer. So here is my problem- I have a string like this: {1,2,{3,{4},5},{5,6}} I'm removing the most outside parentheses (they are there from input, and I don't need them), so now...

How to use regex to search in a string


python,regex
I am now using regex to handle some string, now I have a special string that is not easy to use regex to handle. below are some sample string. TP-LINK is a sample, it can be other router vendor with similar format. TP-LINK Wireless Range Extender WA810RE TP-LINK Wireless WA850RE...

I need to make sure that only certain characters are in a list?


python,regex,string,list,python-2.7
I have this to get input and put it in a list: def start(): move_order=[raw_input("Enter your moves: ").split()] And I only want the characters A, D, S, C, H (it's for a game >_>) to be allowed. I've tried using the regular expressions stuff: if re.match('[ADSCH]+', [move_order]) is False: print...

Matching string inside file and returning result


regex,string,bash,shell,grep
I've got a few peculiar issues with trying to search for a string inside of a .db file. The way I tried was by using grep, which does apparently find the string(s), although this is the output: $ grep "ext" *.db Binary file enormous.db matches There are a couple problems...

RegExp to check if file is image


javascript,regex
Is there a way to know if the file extension is an image? i got this. image/png Already try with var imageReg = /\.(gif|jpg|jpeg|tiff|png)$/i; string = "image/png" imageReg.test(string) But this return false;...

How to create the javascript regular expression for number with some special symbols


javascript,regex
what can be the java-script regular expression which gives the numbers with some symbols For example following condition must be pass. Number can start with $ Can have the . or , : symbols between and % sign at the send. Passing valus: $233 48.23% 278 22.33 45:23 10,000 Number...

Extracting strings from HTML with Python wont work with regex or BeautifulSoup


python,regex,parsing,beautifulsoup,python-requests
Im using Python 2.7, BeautifulSoup4, regex, and requests on windows 7. I've scraped some code from a website and I am having problems parsing and extracting the bits I want and storing them in a dictionary. What I'm after is text that is presented as follows in the code: @CAD_DTA\">I...

Regex with whitespaces and preceding zeros


regex,sas
I want to match the string 11 with a regular Expression in SAS. The 11 can be preceded by zero or more 0 and/or by white spaces. Any other character is not allowed. Likewise, if anything there should only be white spaces following the 11. Examples: Match: 0000011 11 11<space><space>...

Swing regular expression for phone number validation


java,regex
I want to validate phone number field in swing, so I am writing code to allow user to enter only digits, comma, spaces. For this I am using regular expression, when user enter characters or other than the pattern text field will consume. My code is not working. Can anyone...

Negate a specific group in regular expressions


regex,vb6
How do you get a string not containing a specific group? (?:[0-9-+*/()x]|abs|pow|ln|pi|e|a?(sin|cos|tan)h?)+ Above string is the expression for mathematical expressions. How do you get the string that is not a mathematical expression? Example:WIDTH+LENGTH*abs(2) Supposed Output: WIDTH LENGTH...

Regex that allow void fractional part of number


c#,regex
@"[+-]?\d+(\.\d+)?" -this is a regex I have wrote for numbers it allows [+-] minus before the number digits before and digits after the point the question is how to change this to allow "not finished" values so that input of "5." - is fine too ?...

regular expression in sublime text 2 to match text


regex,sublimetext2
I have a string that looks like this: lonfksa.newsvine.com and I have tons of file that looks like this: http://ricambi.ru/avtomobilnie-novosti/lexus-gotovit-k-debiutu-obnovlenniy-rx http://www.kiwibox.com/hoytboar/blog/entry/121424391/modis-tshirt-tips-untuk-womens-clothing/ http://www.euro-rockradio.com/archives/category/interview http://lonfksa.newsvine.com/_news/2014/04/18/23538711-vampir-romantis-clothing...

Reg ex matching a word


regex
I need to match only first two files, out of four files listed below: ABD_DEF_GHIJ_20150611 ABD_DEF_GHIJ ABD_DEF_GHIJ_FX_20150611 ABD_DEF_GHIJ_FX I am using reg ex - ABD_DEF_GHIJ(_\d{8}|\b) and it's working fine. I would like to know if my solution is ok or there is any better alternate solution....

Patterns (preferably java) find floor space in sq ft


java,regex,pattern-matching
I have the following data that needs parsing. The pattern could be (1) approx 1,000 sq ft (2)c. 500sqft (3) 2,100 sq ft This is my code to find a digit but I need the above... in java Pattern regex = Pattern.compile("\\d[\\d,\\.]+"); Matcher finder = regex.matcher(price); if(finder.find()){ try { String...

How to write RegEx for inserting line break for line length more than 30 characters?


regex
I am using a text editor which lets use regular expression to find / replace text. I have a large text file. I want to insert new line in each lines which are more than 30 characters. I want the line to break after 30th character (doesnt matter if a...

Please can someone help me understand the exec method for regular expressions?


javascript,regex
The best place I have found for the exec method is Eloquent Javascript Chapter 9: "Regular expressions also have an exec (execute) method that will return null if no match was found and return an object with information about the match otherwise. An object returned from exec has an index...

MySQL substring match using regular expression; substring contain 'man' not 'woman'


mysql,regex
I have an issue while I fetch data from database using regular expression. While I search for 'man' in tags it returns tags contains 'woman' too; because its substring. SELECT '#hellowomanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 correct, it contains 'woman' SELECT '#helloowmanclothing' REGEXP '^(.)*[^wo]man(.)*$'; # returns 0 incorrect, it can...

jQuery / Regex: How to compare string against several substrings


jquery,regex,string,substring,substr
I have a special validation need where I need to check if a string contains ANY out of several substrings like the following list: 12 23 34 45 56 67 78 89 90 01 I am new to jQuery and the only thing I could think of here is the...

Warning: preg_match_all(): Unknown modifier '\' [duplicate]


php,regex,warnings
This question already has an answer here: PHP - preg_match and “Unknown modifier” error 3 answers I know there are tons of duplicate question like this. But I am stuck here. Can anyone help me out? Here's the code: <?php function count_images($string){ preg_match_all('/[^/\s]+/\S+\.(jpg|png|gif)/', $string, $results); return count($results[1]); } ?>...

like and regexp_like


sql,regex,oracle,oracle11g,regexp-like
For like we have %. for ex: if we give ad% it ll fetch all the records which starts with ad but i should use regexp_like. for regexp_like what can be used so that it acts as % for like. i cant use ^ad because from UI we ll give...

Does there exist an algorithm for iterating through all strings that conform to a particular regex?


c#,regex,algorithm
I'm making a script to try and hack into an account whose login password is at least 8 characters long and includes at least 1 number, 1 special character and 1 capital letter. I will use brute force. Is there a compact, elegant and efficient way to iterate through every...

Get all prices with $ from string into an array in Javascript


javascript,regex,currency
var string = 'Our Prices are $355.00 and $550, down form $999.00'; How can I get those 3 prices into an array?...

Match a pattern preceded by a specific pattern without using a lookbehind


regex,eclipse,lookahead
Is there a way to match a B only if preceded by an A? The A can be at any position behind the B, with any amount of characters between. Examples: A_B (Matches `B`) C_B (No match) I've tried: (?=A)[^B]*B But it matches all the characters preceeding B as well....

How to match words in 2 list against another string of words without sub-string matching in Python?


python,regex,string,loops,twitter
I have 2 lists with keywords in them: slangNames = [Vikes, Demmies, D, MS Contin] riskNames = [enough, pop, final, stress, trade] i also have a dictionary called overallDict, that contains tweets. The key value pairs are {ID: Tweet text) For eg: {1:"Vikes is not enough for me", 2:"Demmies is...

How many characters are visible like a space, but are not space characters?


php,regex
If I want to discover the hexadecimal equivalent of a space in PHP I can play with bin2hex: php > echo var_dump(bin2hex(" ")); string(2) "20" I can also obtain space character from "20" php > echo var_dump(hex2bin("20")); string(1) " " But there exist Unicode versions of a "visible" space: php...

PHP Regular Expressions Counting starting consonants in a string


php,regex
I need to find out how many starting consonants a word has. The number is used later in the program. The code below does work, I am wondering if it is possible to do this with a regular expression. $mystring ="SomeStringExample"; $mystring2 =("bcdfghjklmnpqrstvwxyzABCDFGHJKLMNPQRSTWVXYZ"); $var = strspn($mystring, $mystring2); Using a regular...

Get number from string


regex
I am trying to get the enclosed number between two slashes in a URL using regex. The code regex I have is not working, I am fairly new to regex and don't really understand it. The regex: http:\/\/?www\.?example\.com\/g\/(^\d$)\/\w The URL: http://www.example.com/g/1337/Game-Title Trying to get the "1337", which is the PlaceId....

Java - Enforce TextField Format - UX - 00:00:00;00


java,regex,user-interface
Please see below image of the UI of my current application Currently, I am enforcing Data Format of the text fields in the back end. If the RegEx does match (00:00:00;00), it will throw an error. My question is from a UX perspective, is it possible to make the input...

REGEX python find previous string


python,regex,string
I'm trying to find if the last word of the string is followed by a space or a special char, and if yes return the string without this space/special char For example : "do you love dogs ?" ==> return "do you love dogs" "i love my dog " (space...

Ruby gsub group parameters do not work when preceded by escaped slashes


ruby,regex
I am trying to perform a trivial substitution, that in any other language I have come across, work as per the documentation. However, my substitution fails for some reason. The documentation examples list: "hello".gsub(/[aeiou]/, '*') #=> "h*ll*" "hello".gsub(/([aeiou])/, '<\1>') #=> "h<e>ll<o>" "hello".gsub(/./) {|s| s.ord.to_s + ' '} #=> "104 101...