6 ms·

Ftfy and Unidecode[1] are two of the first libraries I reach for when munging around with a new dataset. My third favorite is the regex filter [^ -~]. If you kn

by cosmie 9y ago

Ftfy and Unidecode[1] are two of the first libraries I reach for when munging around with a new dataset. My third favorite is the regex filter [^ -~]. If you know you’re working with ASCII data, that single regex against any untrusted data is resolves soooo many potential headaches. I manage and source data for a lead generation/marketing firm, and dealing with free text data from all over the place is a nightmare. Even when working with billion dollar firms, data purchases take the form of CSV files with limited or no documentation on structure or formatting or encoding, sales-oriented data dictionaries, and FTP drops. I have a preprocessing script in python that strips out lines that can’t be parsed as utf8, stage it into a Redshift cluster, then hit it with a UDF in Redshift called kitchen_sink that encapsulates a lot of my text cleanup and validation heuristics like the above.

jonathan_n 9y ago
This. Ftfy has saved my as many many times. Thanks for the regex. brilliant.
- cosmie 9y ago
  I don't remember where I came across that regex, but it's saved me from so many headaches I quite literally get giddy any time I can insert it into a processing stream. A developer upstream naively split on /n and leave some errant /r characters everywhere? Fixed. Embedded backspaces or nulls or tabs or any sort of control character? Gone. Non-ASCII data in a field you know should be ASCII only? Ha, not today good sir! Until you've had to deal with the hell that is raw, free form data from who-knows-where, you cannot even fathom how satisfying it is to be able to deploy that regex (when appropriate) and know beyond a doubt that you can't have any more gremlins hiding in that particular field/data that'll hit you later.
  - thiagocsf 9y ago
    It’s nice not having to deal with languages other than English. In Portuguese, which I’ve worked with, you develop other tricks, like replacing à, á, â or ã with a. But, in order to do this, you still need to find out the encoding used before you can create the “ASCII” equivalent. Fun trivia: coco means coconut; cocô means poo. So, by replacing ô with o, you’re guaranteed a chuckle at some stage.
    - Piskvorrr 9y ago
      True. But if that specific case is dealing in, let's say, URLs, then all of the content should be ASCII - either as a direct character representation, or encoded, again to ASCII. Never did I see the parent mention "this is sufficient for humans", or just "...for English" - even that would be a naïve assumption (see? ;)).
      - rspeer 9y ago
        If your definition of "URLs" includes "IRIs" (a term nobody really uses but which encompasses the idea that you can use Unicode in URLs), then this isn't a good assumption to make. I would rather pass around a link to https://ja.wikipedia.org/wiki/メインページ https://ja.wikipedia.org/wiki/メインページ than to https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8 https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3... , and lots of software will support both. And if you want all URLs to be ASCII, you'll need to convert the first into the second, not just delete the Japanese characters.
        zlynx 9y ago
        If the URL is "on the wire" it had better be 7-bit ASCII. Actually even more restrictive than that. Because that's the spec. https://tools.ietf.org/html/rfc3986 https://tools.ietf.org/html/rfc3986 In user interaction with a browser or wherever else it seems that anything goes.
        rspeer 9y ago
        That's the spec but this is also the spec: https://www.ietf.org/rfc/rfc3987.txt https://www.ietf.org/rfc/rfc3987.txt It's true that protocols such as HTTP only use ASCII URIs on the wire. If you are implementing an HTTP client yourself, you will need to implement percent-encoding. Which is different from saying "well, I know URLs should be ASCII on the wire, so I can safely delete all the non-ASCII characters from a URL." That's not true.
        9y ago
- kuro68k 9y ago
  I was hoping it was a replacement for Unicode... Unicode is so broken that we need something new.
aw3c2 9y ago
What does that regex do?
- cosmie 9y ago
  Matches anything that's not between the space and tilde in the ASCII code range[1], which is the entire range of printable ASCII characters. It's similar to [a-Z] you see a lot, but expanded to include the space character, numbers, and punctuation. The regex [ -~] lets you match those characters, whereas [^ -~] negates that and matches anything that's not a printable ASCII character (useful for regex replace functions). If you look at the table at the top of [1], you'll notice all of the characters at the beginning of the ASCII range which are non-printing and therefore invisible. Plus, at the end for some insane reason, the DELETE character. If there's no valid reason for any of these characters to exist in your dataset, nor for higher code point (UTF) characters to exist, then [^ -~] will match them and let you strip them out all in one go. [1] http://www.asciitable.com/ http://www.asciitable.com/
  - xstartup 9y ago
    Do you've any idea whether it's possible to search invisible characters using google?
    - cosmie 9y ago
      I'm not sure, but Google Search in general strips punctuation that isn't specifically designated as a search operator[1], and likely strips control/non-printing characters in a similar manner. Testing it[2] with a few URL-encoded control and invisible characters[3], they seem to be ignored. But they do make it into the query parameter, at least until you edit it for a subsequent search, at which point they get converted into spaces/+ symbols in the query. [1] https://support.google.com/websearch/answer/2466433?hl=en https://support.google.com/websearch/answer/2466433?hl=en [2] https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" [3] You can see the full string being searched here, it's 7 characters long: https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F
LittlePeter 9y ago
Note that when using GNU grep (version 2.25) you have to use -P flag for Perl compatible regular expressions. E.g.: echo "lörem ipsum" | grep -P "[^ -~]" --color and not echo "lörem ipsum" | grep "[^ -~]" --color

by cosmie 9y ago

jonathan_n 9y ago

This. Ftfy has saved my as many many times. Thanks for the regex. brilliant.

cosmie 9y ago
I don't remember where I came across that regex, but it's saved me from so many headaches I quite literally get giddy any time I can insert it into a processing stream. A developer upstream naively split on /n and leave some errant /r characters everywhere? Fixed. Embedded backspaces or nulls or tabs or any sort of control character? Gone. Non-ASCII data in a field you know should be ASCII only? Ha, not today good sir! Until you've had to deal with the hell that is raw, free form data from who-knows-where, you cannot even fathom how satisfying it is to be able to deploy that regex (when appropriate) and know beyond a doubt that you can't have any more gremlins hiding in that particular field/data that'll hit you later.
- thiagocsf 9y ago
  It’s nice not having to deal with languages other than English. In Portuguese, which I’ve worked with, you develop other tricks, like replacing à, á, â or ã with a. But, in order to do this, you still need to find out the encoding used before you can create the “ASCII” equivalent. Fun trivia: coco means coconut; cocô means poo. So, by replacing ô with o, you’re guaranteed a chuckle at some stage.
  - Piskvorrr 9y ago
    True. But if that specific case is dealing in, let's say, URLs, then all of the content should be ASCII - either as a direct character representation, or encoded, again to ASCII. Never did I see the parent mention "this is sufficient for humans", or just "...for English" - even that would be a naïve assumption (see? ;)).
    - rspeer 9y ago
      If your definition of "URLs" includes "IRIs" (a term nobody really uses but which encompasses the idea that you can use Unicode in URLs), then this isn't a good assumption to make. I would rather pass around a link to https://ja.wikipedia.org/wiki/メインページ https://ja.wikipedia.org/wiki/メインページ than to https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8 https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3... , and lots of software will support both. And if you want all URLs to be ASCII, you'll need to convert the first into the second, not just delete the Japanese characters.
      - zlynx 9y ago
        If the URL is "on the wire" it had better be 7-bit ASCII. Actually even more restrictive than that. Because that's the spec. https://tools.ietf.org/html/rfc3986 https://tools.ietf.org/html/rfc3986 In user interaction with a browser or wherever else it seems that anything goes.
        rspeer 9y ago
        That's the spec but this is also the spec: https://www.ietf.org/rfc/rfc3987.txt https://www.ietf.org/rfc/rfc3987.txt It's true that protocols such as HTTP only use ASCII URIs on the wire. If you are implementing an HTTP client yourself, you will need to implement percent-encoding. Which is different from saying "well, I know URLs should be ASCII on the wire, so I can safely delete all the non-ASCII characters from a URL." That's not true.
        9y ago
kuro68k 9y ago
I was hoping it was a replacement for Unicode... Unicode is so broken that we need something new.

aw3c2 9y ago

What does that regex do?

cosmie 9y ago
Matches anything that's not between the space and tilde in the ASCII code range[1], which is the entire range of printable ASCII characters. It's similar to [a-Z] you see a lot, but expanded to include the space character, numbers, and punctuation. The regex [ -~] lets you match those characters, whereas [^ -~] negates that and matches anything that's not a printable ASCII character (useful for regex replace functions). If you look at the table at the top of [1], you'll notice all of the characters at the beginning of the ASCII range which are non-printing and therefore invisible. Plus, at the end for some insane reason, the DELETE character. If there's no valid reason for any of these characters to exist in your dataset, nor for higher code point (UTF) characters to exist, then [^ -~] will match them and let you strip them out all in one go. [1] http://www.asciitable.com/ http://www.asciitable.com/
- xstartup 9y ago
  Do you've any idea whether it's possible to search invisible characters using google?
  - cosmie 9y ago
    I'm not sure, but Google Search in general strips punctuation that isn't specifically designated as a search operator[1], and likely strips control/non-printing characters in a similar manner. Testing it[2] with a few URL-encoded control and invisible characters[3], they seem to be ignored. But they do make it into the query parameter, at least until you edit it for a subsequent search, at which point they get converted into spaces/+ symbols in the query. [1] https://support.google.com/websearch/answer/2466433?hl=en https://support.google.com/websearch/answer/2466433?hl=en [2] https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" [3] You can see the full string being searched here, it's 7 characters long: https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F

LittlePeter 9y ago

Note that when using GNU grep (version 2.25) you have to use -P flag for Perl compatible regular expressions. E.g.: echo "lörem ipsum" | grep -P "[^ -~]" --color and not echo "lörem ipsum" | grep "[^ -~]" --color