5 ms·

Leaving the behavior undefined in almost all commonly used locales (i.e. anything involving UTF-8) doesn't seem to be a particularly helpful standard. It's unr

by kijin 8y ago

Leaving the behavior undefined in almost all commonly used locales (i.e. anything involving UTF-8) doesn't seem to be a particularly helpful standard. It's unreasonable to expect someone who writes regexp to anticipate which locales a user will execute his program in. It's just as unreasonable to tell him to stick to an outdated locale. How about we ignore locales altogether and just use code points? Code point order is the only ordering that every locale can agree on. [a-z] should match any character whose code point is between U+0061 and U+007A regardless of the locale.

pgeorgi 8y ago
your proposal changes behavior for EBCDIC
- kijin 8y ago
  According to the linked article, recent versions of gawk have already shown the middle finger to EBCDIC, by taking the liberty to interpret "undefined" as "we'll just go back to ASCII". It would be nice to expand that to the full UTF-8 range, though.
  - pgeorgi 8y ago
    gawk, yes. I was referring to the POSIX standard declaring it undefined.
    - kijin 8y ago
      If the standard says the behavior is undefined, EBCDIC users can't complain if the standard later becomes defined in a way that is different from the way they've been doing it.
  - laumars 8y ago
    Genuine question: does anyone actually still use EPCDIC (outside of hobbyists maintaining retro hardware)?
    - dfox 8y ago
      This question should probably be rephrased as "was there ever system that used EBCDIC in it's POSIX-compliant interface?" which I strongly believe there was not.
    - slavik81 8y ago
      From the "IBM comment on preparing for a Trigraph-adverse future in C++17" (Oct 2014): > There are real customers who use EBCDIC. We cannot reveal their names due to confidentiality agreements. One key example is some of the major banks in North America who continue to use IBM machines to perform check clearing operations. These high reliability software systems are written on IBM mainframes clearing your checks and because they have been debugged over so many years and are highly critical to daily integrity of the financial industry, they are highly reliable and will never be moved to any other platform. In a way, this makes their code unaffected by this removal of trigraphs. > Other EBCDIC users with trigraphs from a world-wide company that can’t get square brackets (i.e. [ ]) everywhere so must use trigraphs to get at them quotes: “In that capacity we have widely distributed deployments, around the world, across Windows, HP-UX, Linux, z/OS and iSeries systems. That's why we need the trigraphs, it doesn't seem you can count on the [ ] characters working everywhere. ... " Source: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4210.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n421...
      - laumars 8y ago
        Interesting read however even that was 4 years ago. A lot of European banks have been making major upgrades to their IT infrastructure (for better or worse) in recent years so it wouldn't surprise me if a few of those EBCDIC systems have been torn out of American banks as well. However even assuming they are still in use (which is a real possibility knowing how slowly banks upgrade) I would be surprised if they depended on GNU Awk let alone undefined behaviour in its Regex library. I'm not normally someone who says "move fast and break things" but I do think this is one situation where it is ok. I mean EBCDIC is hardly recent and it wasn't exactly popular even when it was "current tech" (I seem to recall it was a bit of a laughing stock?)
dwheeler 8y ago
I agree that it is terrible that this is undefined, but utf-8 is not a Locale. That is an encoding. The standard works just fine when you use C as the Locale and utf-8 as the encoding. I think they should have just defined ranges as being the encoding values, because that would make more sense, but that would be exactly the opposite of what the standard previously said.
- 1996 8y ago
  Funny enough, after plenty of locale nonsense with ls and the other tools, I decided my locale was C.UTF-8 This works at least as well as en-US, and provides me just enough internationalization to remove the most annoying americanocentric things
- kijin 8y ago
  > utf-8 is not a Locale. Of course it isn't. What I suggested above is to use neither locales nor UTF-8, but code points. "a" = U+0061 no matter which locale you're in, and no matter which encoding you use. Every locale and every encoding is based on the same universal mapping of characters to code points.
  - oldmanhorton 8y ago
    Sure, but those code points are arbitrary. For instance in German languages, you may want a-z to include umlauted vowels, or you may not. That's a locale specific setting, even though the umlauted characters come well outside of the range of ascii a to z.
    - mmt 8y ago
      Keeping ranges undefined doesn't satisfy these wants, either. Using code points would at least allow for ranges to have the possibility of being usable to someone in a standard, predictable fashion, outside of the C/POSIX locale. For example, specifying a-z plus each umlauted vowel is still shorter than specifying all letters individually. Perhaps there is some wisdom in William S. Burrough's "If you can't be just, be arbitrary."
  - 8y ago
zokier 8y ago
Codepoint ranges were also suggested in the bugzilla thread linked by rwmj https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c13 https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c13

by kijin 8y ago

pgeorgi 8y ago

your proposal changes behavior for EBCDIC

kijin 8y ago
According to the linked article, recent versions of gawk have already shown the middle finger to EBCDIC, by taking the liberty to interpret "undefined" as "we'll just go back to ASCII". It would be nice to expand that to the full UTF-8 range, though.
- pgeorgi 8y ago
  gawk, yes. I was referring to the POSIX standard declaring it undefined.
  - kijin 8y ago
    If the standard says the behavior is undefined, EBCDIC users can't complain if the standard later becomes defined in a way that is different from the way they've been doing it.
- laumars 8y ago
  Genuine question: does anyone actually still use EPCDIC (outside of hobbyists maintaining retro hardware)?
  - dfox 8y ago
    This question should probably be rephrased as "was there ever system that used EBCDIC in it's POSIX-compliant interface?" which I strongly believe there was not.
  - slavik81 8y ago
    From the "IBM comment on preparing for a Trigraph-adverse future in C++17" (Oct 2014): > There are real customers who use EBCDIC. We cannot reveal their names due to confidentiality agreements. One key example is some of the major banks in North America who continue to use IBM machines to perform check clearing operations. These high reliability software systems are written on IBM mainframes clearing your checks and because they have been debugged over so many years and are highly critical to daily integrity of the financial industry, they are highly reliable and will never be moved to any other platform. In a way, this makes their code unaffected by this removal of trigraphs. > Other EBCDIC users with trigraphs from a world-wide company that can’t get square brackets (i.e. [ ]) everywhere so must use trigraphs to get at them quotes: “In that capacity we have widely distributed deployments, around the world, across Windows, HP-UX, Linux, z/OS and iSeries systems. That's why we need the trigraphs, it doesn't seem you can count on the [ ] characters working everywhere. ... " Source: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4210.pdf http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n421...
    - laumars 8y ago
      Interesting read however even that was 4 years ago. A lot of European banks have been making major upgrades to their IT infrastructure (for better or worse) in recent years so it wouldn't surprise me if a few of those EBCDIC systems have been torn out of American banks as well. However even assuming they are still in use (which is a real possibility knowing how slowly banks upgrade) I would be surprised if they depended on GNU Awk let alone undefined behaviour in its Regex library. I'm not normally someone who says "move fast and break things" but I do think this is one situation where it is ok. I mean EBCDIC is hardly recent and it wasn't exactly popular even when it was "current tech" (I seem to recall it was a bit of a laughing stock?)

dwheeler 8y ago

I agree that it is terrible that this is undefined, but utf-8 is not a Locale. That is an encoding. The standard works just fine when you use C as the Locale and utf-8 as the encoding. I think they should have just defined ranges as being the encoding values, because that would make more sense, but that would be exactly the opposite of what the standard previously said.

1996 8y ago
Funny enough, after plenty of locale nonsense with ls and the other tools, I decided my locale was C.UTF-8 This works at least as well as en-US, and provides me just enough internationalization to remove the most annoying americanocentric things
kijin 8y ago
> utf-8 is not a Locale. Of course it isn't. What I suggested above is to use neither locales nor UTF-8, but code points. "a" = U+0061 no matter which locale you're in, and no matter which encoding you use. Every locale and every encoding is based on the same universal mapping of characters to code points.
- oldmanhorton 8y ago
  Sure, but those code points are arbitrary. For instance in German languages, you may want a-z to include umlauted vowels, or you may not. That's a locale specific setting, even though the umlauted characters come well outside of the range of ascii a to z.
  - mmt 8y ago
    Keeping ranges undefined doesn't satisfy these wants, either. Using code points would at least allow for ranges to have the possibility of being usable to someone in a standard, predictable fashion, outside of the C/POSIX locale. For example, specifying a-z plus each umlauted vowel is still shorter than specifying all letters individually. Perhaps there is some wisdom in William S. Burrough's "If you can't be just, be arbitrary."
- 8y ago

zokier 8y ago

Codepoint ranges were also suggested in the bugzilla thread linked by rwmj https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c13 https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c13