4 ms·

Ah ... it seemed like such a beautiful dream for a moment. But yeah, no way to fix this without breaking/deprecating abi /api in C,C++. Which really isn't surpr

by midjji 5y ago

Ah ... it seemed like such a beautiful dream for a moment. But yeah, no way to fix this without breaking/deprecating abi /api in C,C++. Which really isn't surprising, char and uchar should always have been byte, char should not have existed. std::string, and all string functions in C should always have contained/ required explicit if default encoding information, and the streams should always have had encoding defaults for its input and output. While the compiler should always have required a non locale based ... and so on. The problem keeps getting worse too, view is a generic and powerful meta, but it means there should never have been such a thing as string_view, except possibly as a template specialization for performance reasons, and of course that would wrapp the proper encoded string, with its explicit encoding of string litteral and so on. That said, with appropriate new api in the standard and with the compiler explicitly requiring explicit encoding specified sources, this could all be solved. The c++ streams are a very useful construct but horribly implemented, so we could just make a new one, and then deprecate the broken part of the standard lib. Its just, this is never a problem you realize when coding on your own, since it only ever hits projects big enough to be multilocal.

nly 5y ago
Most programmers should never have to worry about manipulating strings intended for human consumption at the character or code-point level. It makes about as much sense as trying to manipulate bytes inside a JPEG when you're writing a typical desktop CRUD application. Projects big enough to be multi-locale (or actually all projects) should definitely be using format strings (to account for different parameter orders), locale sensitive date, time and currency formats, and a good externalized translation framework... but I still think they should be using UTF-8 throughout, because the encoding you use is completely unrelated to these problems. C++ in particular is used less and less for human facing software, and where it is libraries like Qt are fairly excellent at handling this stuff.
- midjji 5y ago
  UTF-8 mandated everywhere is how I work atm, and I am happy the old latin... etc died off. But I think its fundamentally the wrong way to go in a sense. UTF-8 is too huge for starters, and does a great deal which it probably should not as it will never do it well. While latex is horrifying in its own right, it remains indisputably superior at writing math compared to UTF-8 and similar for html5 with styling. In both cases the extensibility and styling is built in despite not needing more than perhaps 50 characters to write. It seems to me like the mistake was adding locales in the first place, we have a chance to standardize not language, but meta language. The user will never see the low level stuff, and the programmer will mostly not see it unless they click show low level in their ide, and it will always be compressed when it matters, so why even have separate a,A. Why not have a, \{upper}{a}. shown, and even entered in the keyboard as a,A. The key difference to utf would be that the rendering and which commands exists is up to the locale which could be globally stand... Wait I just made it worse didnt I... Qt still leaves you with the problem of string litterals possibly changing when moving the code from one locale to another as the code itself is not of defined encoding, so if reparsed it will either look bad in code, or bad in output. The everpresent tr is also rather annoying.
- midjji 5y ago
  Oh and one warning regarding UTF-8 everywhere, use it but perhaps dont force it on a filesystem level. I have a file in a zfs filesystem with utf-8 and normD which everytime I try to delete it remains, and if I copy/move it in the filesystem it duplicates in size and if I try to copy it to someplace else, that fails. It has some wierd seemingly cryllic filename too long to be shown and changing with zfs version numbers and possibly randomly. I think it was a meme image originally, but its taking up 135GB of tank space by now. I mostly keep it for fun, hoping zfs will eventually either declare it broken or fix it so I can see what it was. I'd share it but the tank is huge, and I never figured out a way to move or read it without dd ing the entire tank.
  - nly 5y ago
    Can you explain this? Aren't file names on unix platforms essentially blobs, sans a few delimiters ('/', '\0', maybe '\')?
    - Filligree 5y ago
      ZFS has a 'utf8only' property which, in principle, lets you constrain all filenames on a dataset to be utf-8... only. It does not otherwise change the API, but it should make the creation of a non-utf8 file into an I/O error. This defaults to off, so by default you're right. Apparently the GP found a bug in this code. I'd be interested in seeing the github issue.
      - midjji 5y ago
        With utf8only you have to specify a way to convert non utf8 to utf8, I think the problem is a bug in the normD converter.
    - AnIdiotOnTheNet 5y ago
      IIRC the only character disallowed in EXT2 at least is '/'. Interesting things happen if you manage to get '/' into a file name, which can be done with a hex editor or a broken Windows EXT2 file browser in my case.

by midjji 5y ago

nly 5y ago

Most programmers should never have to worry about manipulating strings intended for human consumption at the character or code-point level. It makes about as much sense as trying to manipulate bytes inside a JPEG when you're writing a typical desktop CRUD application. Projects big enough to be multi-locale (or actually all projects) should definitely be using format strings (to account for different parameter orders), locale sensitive date, time and currency formats, and a good externalized translation framework... but I still think they should be using UTF-8 throughout, because the encoding you use is completely unrelated to these problems. C++ in particular is used less and less for human facing software, and where it is libraries like Qt are fairly excellent at handling this stuff.

midjji 5y ago
UTF-8 mandated everywhere is how I work atm, and I am happy the old latin... etc died off. But I think its fundamentally the wrong way to go in a sense. UTF-8 is too huge for starters, and does a great deal which it probably should not as it will never do it well. While latex is horrifying in its own right, it remains indisputably superior at writing math compared to UTF-8 and similar for html5 with styling. In both cases the extensibility and styling is built in despite not needing more than perhaps 50 characters to write. It seems to me like the mistake was adding locales in the first place, we have a chance to standardize not language, but meta language. The user will never see the low level stuff, and the programmer will mostly not see it unless they click show low level in their ide, and it will always be compressed when it matters, so why even have separate a,A. Why not have a, \{upper}{a}. shown, and even entered in the keyboard as a,A. The key difference to utf would be that the rendering and which commands exists is up to the locale which could be globally stand... Wait I just made it worse didnt I... Qt still leaves you with the problem of string litterals possibly changing when moving the code from one locale to another as the code itself is not of defined encoding, so if reparsed it will either look bad in code, or bad in output. The everpresent tr is also rather annoying.
midjji 5y ago
Oh and one warning regarding UTF-8 everywhere, use it but perhaps dont force it on a filesystem level. I have a file in a zfs filesystem with utf-8 and normD which everytime I try to delete it remains, and if I copy/move it in the filesystem it duplicates in size and if I try to copy it to someplace else, that fails. It has some wierd seemingly cryllic filename too long to be shown and changing with zfs version numbers and possibly randomly. I think it was a meme image originally, but its taking up 135GB of tank space by now. I mostly keep it for fun, hoping zfs will eventually either declare it broken or fix it so I can see what it was. I'd share it but the tank is huge, and I never figured out a way to move or read it without dd ing the entire tank.
- nly 5y ago
  Can you explain this? Aren't file names on unix platforms essentially blobs, sans a few delimiters ('/', '\0', maybe '\')?
  - Filligree 5y ago
    ZFS has a 'utf8only' property which, in principle, lets you constrain all filenames on a dataset to be utf-8... only. It does not otherwise change the API, but it should make the creation of a non-utf8 file into an I/O error. This defaults to off, so by default you're right. Apparently the GP found a bug in this code. I'd be interested in seeing the github issue.
    - midjji 5y ago
      With utf8only you have to specify a way to convert non utf8 to utf8, I think the problem is a bug in the normD converter.
  - AnIdiotOnTheNet 5y ago
    IIRC the only character disallowed in EXT2 at least is '/'. Interesting things happen if you manage to get '/' into a file name, which can be done with a hex editor or a broken Windows EXT2 file browser in my case.