7 ms·

I don't remember where I came across that regex, but it's saved me from so many headaches I quite literally get giddy any time I can insert it into a processing

by cosmie 9y ago

I don't remember where I came across that regex, but it's saved me from so many headaches I quite literally get giddy any time I can insert it into a processing stream. A developer upstream naively split on /n and leave some errant /r characters everywhere? Fixed. Embedded backspaces or nulls or tabs or any sort of control character? Gone. Non-ASCII data in a field you know should be ASCII only? Ha, not today good sir! Until you've had to deal with the hell that is raw, free form data from who-knows-where, you cannot even fathom how satisfying it is to be able to deploy that regex (when appropriate) and know beyond a doubt that you can't have any more gremlins hiding in that particular field/data that'll hit you later.

thiagocsf 9y ago
It’s nice not having to deal with languages other than English. In Portuguese, which I’ve worked with, you develop other tricks, like replacing à, á, â or ã with a. But, in order to do this, you still need to find out the encoding used before you can create the “ASCII” equivalent. Fun trivia: coco means coconut; cocô means poo. So, by replacing ô with o, you’re guaranteed a chuckle at some stage.
- Piskvorrr 9y ago
  True. But if that specific case is dealing in, let's say, URLs, then all of the content should be ASCII - either as a direct character representation, or encoded, again to ASCII. Never did I see the parent mention "this is sufficient for humans", or just "...for English" - even that would be a naïve assumption (see? ;)).
  - rspeer 9y ago
    If your definition of "URLs" includes "IRIs" (a term nobody really uses but which encompasses the idea that you can use Unicode in URLs), then this isn't a good assumption to make. I would rather pass around a link to https://ja.wikipedia.org/wiki/メインページ https://ja.wikipedia.org/wiki/メインページ than to https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8 https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3... , and lots of software will support both. And if you want all URLs to be ASCII, you'll need to convert the first into the second, not just delete the Japanese characters.
    - zlynx 9y ago
      If the URL is "on the wire" it had better be 7-bit ASCII. Actually even more restrictive than that. Because that's the spec. https://tools.ietf.org/html/rfc3986 https://tools.ietf.org/html/rfc3986 In user interaction with a browser or wherever else it seems that anything goes.
      - rspeer 9y ago
        That's the spec but this is also the spec: https://www.ietf.org/rfc/rfc3987.txt https://www.ietf.org/rfc/rfc3987.txt It's true that protocols such as HTTP only use ASCII URIs on the wire. If you are implementing an HTTP client yourself, you will need to implement percent-encoding. Which is different from saying "well, I know URLs should be ASCII on the wire, so I can safely delete all the non-ASCII characters from a URL." That's not true.
        Piskvorrr 9y ago
        As soon as you add or remove to the string, it no longer points to the same resource; I considered this too obvious to mention. OTOH, for _validation_, this is useful: "you have a ZWJ character in an URL, that's unlikely". And yes, I understand that there are protocols that allow you to pass around the full Unicode or aunt Matilda or whatever - I should have been more specific.
    - Piskvorrr 9y ago
      The first version is much more readable and less hacky; alas, unless you are positive that all the software you're interfacing with can handle Unicode (never have I ever been in such a glorious situation), fallback to URL-encoding it is.

by cosmie 9y ago

thiagocsf 9y ago

It’s nice not having to deal with languages other than English. In Portuguese, which I’ve worked with, you develop other tricks, like replacing à, á, â or ã with a. But, in order to do this, you still need to find out the encoding used before you can create the “ASCII” equivalent. Fun trivia: coco means coconut; cocô means poo. So, by replacing ô with o, you’re guaranteed a chuckle at some stage.

Piskvorrr 9y ago
True. But if that specific case is dealing in, let's say, URLs, then all of the content should be ASCII - either as a direct character representation, or encoded, again to ASCII. Never did I see the parent mention "this is sufficient for humans", or just "...for English" - even that would be a naïve assumption (see? ;)).
- rspeer 9y ago
  If your definition of "URLs" includes "IRIs" (a term nobody really uses but which encompasses the idea that you can use Unicode in URLs), then this isn't a good assumption to make. I would rather pass around a link to https://ja.wikipedia.org/wiki/メインページ https://ja.wikipedia.org/wiki/メインページ than to https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8 https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3... , and lots of software will support both. And if you want all URLs to be ASCII, you'll need to convert the first into the second, not just delete the Japanese characters.
  - zlynx 9y ago
    If the URL is "on the wire" it had better be 7-bit ASCII. Actually even more restrictive than that. Because that's the spec. https://tools.ietf.org/html/rfc3986 https://tools.ietf.org/html/rfc3986 In user interaction with a browser or wherever else it seems that anything goes.
    - rspeer 9y ago
      That's the spec but this is also the spec: https://www.ietf.org/rfc/rfc3987.txt https://www.ietf.org/rfc/rfc3987.txt It's true that protocols such as HTTP only use ASCII URIs on the wire. If you are implementing an HTTP client yourself, you will need to implement percent-encoding. Which is different from saying "well, I know URLs should be ASCII on the wire, so I can safely delete all the non-ASCII characters from a URL." That's not true.
      - Piskvorrr 9y ago
        As soon as you add or remove to the string, it no longer points to the same resource; I considered this too obvious to mention. OTOH, for _validation_, this is useful: "you have a ZWJ character in an URL, that's unlikely". And yes, I understand that there are protocols that allow you to pass around the full Unicode or aunt Matilda or whatever - I should have been more specific.
  - Piskvorrr 9y ago
    The first version is much more readable and less hacky; alas, unless you are positive that all the software you're interfacing with can handle Unicode (never have I ever been in such a glorious situation), fallback to URL-encoding it is.