5 ms·

What does that regex do?

by aw3c2 9y ago

cosmie 9y ago
Matches anything that's not between the space and tilde in the ASCII code range[1], which is the entire range of printable ASCII characters. It's similar to [a-Z] you see a lot, but expanded to include the space character, numbers, and punctuation. The regex [ -~] lets you match those characters, whereas [^ -~] negates that and matches anything that's not a printable ASCII character (useful for regex replace functions). If you look at the table at the top of [1], you'll notice all of the characters at the beginning of the ASCII range which are non-printing and therefore invisible. Plus, at the end for some insane reason, the DELETE character. If there's no valid reason for any of these characters to exist in your dataset, nor for higher code point (UTF) characters to exist, then [^ -~] will match them and let you strip them out all in one go. [1] http://www.asciitable.com/ http://www.asciitable.com/
- xstartup 9y ago
  Do you've any idea whether it's possible to search invisible characters using google?
  - cosmie 9y ago
    I'm not sure, but Google Search in general strips punctuation that isn't specifically designated as a search operator[1], and likely strips control/non-printing characters in a similar manner. Testing it[2] with a few URL-encoded control and invisible characters[3], they seem to be ignored. But they do make it into the query parameter, at least until you edit it for a subsequent search, at which point they get converted into spaces/+ symbols in the query. [1] https://support.google.com/websearch/answer/2466433?hl=en https://support.google.com/websearch/answer/2466433?hl=en [2] https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" [3] You can see the full string being searched here, it's 7 characters long: https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F
- dfox 9y ago
  The reason for DEL being 0x7f is that it is originally intended as marker for "this character was deleted and should be skipped over" and not as command to delete previous/next character. And you can change any ASCII character into 0x7f by ORing it with with 0x7f, that is, punching all the holes in punched tape (or punchcard, but ASCII usually was not used for them and punching all holes on punchcard is not advisable for mechanical reasons).
  - mehrdadn 9y ago
    Love these historical notes that make everything make sense. Thanks for posting!
- aw3c2 9y ago
  Thank you very much for this detailed reply! Made my day :)
- sundarurfriend 9y ago
  > The regex [ -~] lets you match those characters, whereas [^ -~] negates that and matches anything that's not a printable ASCII character Oh dear, I think I finally understand that mysterious phenomenon in some websites where, text I write in my native language gets saved and displayed as empty text, but writing in English works. I've come across this a few times over the years, in random forgotten corners of the Internet. It could well be that this regex got written down somewhere as good security practice, and some developers out there copied it into the code without thinking about whether the ASCII restriction is applicable.
  - cosmie 9y ago
    It's probably a more mundane issue related to either older versions of HTML (which did not support UTF8[1]), or issues with an older database that stored things as latin1 or ASCII by default, or issues with an older programming language that doesn't seamlessly support non-latin1 or non-ascii characters without deliberately doing so (like Python 2). That said, I do agree it's not something that should be used without thought. I mentioned that my usage generally involves knowledge that the data set has no valid reason to contain higher code points, and usually involves usage of Unidecode[2] to convert higher code points into ASCII equivalents (which in many cases strips out contextual knowledge, but is sometimes an acceptable trade off for stability and predictability of the data sent downstream). [1] https://www.w3schools.com/html/html_charset.asp https://www.w3schools.com/html/html_charset.asp [2] https://pypi.python.org/pypi/Unidecode https://pypi.python.org/pypi/Unidecode
- deleted 9y ago
  [deleted]

What does that regex do?

by aw3c2 9y ago

What does that regex do?

Matches anything that's not between the space and tilde in the ASCII code range[1], which is the entire range of printable ASCII characters. It's similar to [a-Z] you see a lot, but expanded to include the space character, numbers, and punctuation. The regex [ -~] lets you match those characters, whereas [^ -~] negates that and matches anything that's not a printable ASCII character (useful for regex replace functions). If you look at the table at the top of [1], you'll notice all of the characters at the beginning of the ASCII range which are non-printing and therefore invisible. Plus, at the end for some insane reason, the DELETE character. If there's no valid reason for any of these characters to exist in your dataset, nor for higher code point (UTF) characters to exist, then [^ -~] will match them and let you strip them out all in one go. [1] http://www.asciitable.com/ http://www.asciitable.com/

xstartup 9y ago
Do you've any idea whether it's possible to search invisible characters using google?
- cosmie 9y ago
  I'm not sure, but Google Search in general strips punctuation that isn't specifically designated as a search operator[1], and likely strips control/non-printing characters in a similar manner. Testing it[2] with a few URL-encoded control and invisible characters[3], they seem to be ignored. But they do make it into the query parameter, at least until you edit it for a subsequent search, at which point they get converted into spaces/+ symbols in the query. [1] https://support.google.com/websearch/answer/2466433?hl=en https://support.google.com/websearch/answer/2466433?hl=en [2] https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F" [3] You can see the full string being searched here, it's 7 characters long: https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F
dfox 9y ago
The reason for DEL being 0x7f is that it is originally intended as marker for "this character was deleted and should be skipped over" and not as command to delete previous/next character. And you can change any ASCII character into 0x7f by ORing it with with 0x7f, that is, punching all the holes in punched tape (or punchcard, but ASCII usually was not used for them and punching all holes on punchcard is not advisable for mechanical reasons).
- mehrdadn 9y ago
  Love these historical notes that make everything make sense. Thanks for posting!
aw3c2 9y ago
Thank you very much for this detailed reply! Made my day :)
sundarurfriend 9y ago
> The regex [ -~] lets you match those characters, whereas [^ -~] negates that and matches anything that's not a printable ASCII character Oh dear, I think I finally understand that mysterious phenomenon in some websites where, text I write in my native language gets saved and displayed as empty text, but writing in English works. I've come across this a few times over the years, in random forgotten corners of the Internet. It could well be that this regex got written down somewhere as good security practice, and some developers out there copied it into the code without thinking about whether the ASCII restriction is applicable.
- cosmie 9y ago
  It's probably a more mundane issue related to either older versions of HTML (which did not support UTF8[1]), or issues with an older database that stored things as latin1 or ASCII by default, or issues with an older programming language that doesn't seamlessly support non-latin1 or non-ascii characters without deliberately doing so (like Python 2). That said, I do agree it's not something that should be used without thought. I mentioned that my usage generally involves knowledge that the data set has no valid reason to contain higher code points, and usually involves usage of Unidecode[2] to convert higher code points into ASCII equivalents (which in many cases strips out contextual knowledge, but is sometimes an acceptable trade off for stability and predictability of the data sent downstream). [1] https://www.w3schools.com/html/html_charset.asp https://www.w3schools.com/html/html_charset.asp [2] https://pypi.python.org/pypi/Unidecode https://pypi.python.org/pypi/Unidecode
deleted 9y ago
[deleted]