9 ms·

Any Encoding, Ever – ztd.text and Unicode for C++

98 points by hasheddan 5y ago

theamk 5y ago
There was a lot of discussion with invisible opponents in that text.. Plenty of counter-arguments but only a hint of what is this replying to. Other than that, it seems to present a C++ version of encoding conversion library (like iconv), but instead of handling encodings dynamically it uses C++ types. So not very good for any place with user-specified encoding (like browser or database or text editor), but can be used when the encoding is known at compile time? I can see a few uses of it, but it seems kinda niche for all that grandiose talk about “liberation of the encoding hell”.
- aidenn0 5y ago
  ztd::text::transcode takes an encoding type, as shown in the second example. In any event, a single switch statement turns a compile-time encoding into a run-time encoding.
Animats 5y ago
If you use Go, Rust, Python 3, or Javascript, this is a non-problem, because those are all native UTF-8. Only for legacy languages is this still a problem.
- samatman 5y ago
  Would have been better to read the article, because then you'd be in a position to explain how those languages handle transcoding from ShiftJS to UTF-8.
- gameman144 5y ago
  This is explicitly called out in the article: non-UTF-8 encodings still need to be handled somehow, and this library makes those just as easy to handle as UTF-8.
- layoutIfNeeded 5y ago
  False. JavaScript uses UTF16: https://mathiasbynens.be/notes/javascript-encoding https://mathiasbynens.be/notes/javascript-encoding
- chrismorgan 5y ago
  Quite apart from the fact that you’re talking at cross-purposes to the article, you’re wrong about three of the four languages you mention. Go strings aren’t actually UTF-8; they aren’t even Unicode: they can contain arbitrary bytes. In practice, most strings will be UTF-8, but you can’t depend on it at all. Rust strings are strictly UTF-8. I appreciate this greatly. Python 3 strings are sequences of Unicode code points (as distinct from Unicode scalar values), allowing ill-formed Unicode (unmatched surrogates), and its internal representation is a disaster because they decided indexing by code point was worthwhile (it’s not, given the costs), so since CPython 3.3 strings are encoded as ISO-8859-1 (only able to represent U+0000 to U+00FF), UCS-2 (further able to represent U+0100 to U+FFFF) or UCS-4 (able to represent all Unicode code points). JavaScript strings allow ill-formed Unicode, and the public representation can be described as either UCS-2 or ill-formed UTF-16 (almost all string access techniques work in UTF-16 code units, though new stuff is now mostly working in Unicode and UTF-8 terms). I think all major engines now use an internal representation of ISO-8859-1 if the string contains only U+0000 to U+00FF, or ill-formed UTF-16 for anything else; but Servo’s work has demonstrated that it’s possible to shift to WTF-8 (UTF-8 plus allowing unmatched surrogates), saving lots of memory on some pages and simplifying and slightly speeding up some things, but at the cost of random access performance by code unit index.
  - Animats 5y ago
    Right, Python 3 looks like it's UTF-8, but CPython has that weird 1, 2 or 4 byte representation. PyPy, though, finally went UTF-8. Not sure how they do indexing. Go strings are supposed to be UTF-8, but it's not enforced. Javascript - bleah.
    - chrismorgan 5y ago
      > Python 3 looks like it's UTF-8 No, it never looked like it was UTF-8. It did look like it was Unicode (of an unspecified encoding), but Python allows unpaired surrogates, which makes for ill-formed Unicode. Python lets you have '\udead'. Well-formed encodings of Unicode (such as UTF-8, UTF-16 and UTF-32) cannot represent U+DEAD. There’s a big difference between “Unicode” and “UTF-8”. > PyPy, though, finally went UTF-8. Oh cool! I remember vaguely hearing the idea being mulled over, years ago. Good to see it happened: https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-.... Wonder what they’ve done about surrogates; the only way they can truly have switched to UTF-8 internally is if they’ve broken compatibility, which I doubt they’ve done; I’m guessing that they actually use WTF-8, not UTF-8. (In the meantime, I’ve changed “Python 3.3” in the grandparent comment to “CPython 3.3” for accuracy.)
- 1wd 5y ago
  Python 3 by default still opens text files using the legacy locale encoding.
- jcelerier 5y ago
  sure fam https://stackoverflow.com/questions/12053168/how-to-properly-output-a-string-in-a-windows-console-with-go https://github.com/intellij-rust/intellij-rust/issues/766 https://bugs.python.org/issue44275 etc etc
ncmncm 5y ago
Every single thing JeanHeyd does goes far, far beyond what we are resigned to accept from lesser individuals. That said, I am in the rowdy “Only UTF-8, ever, and nothing else” gang. Even thinking about UTF-16 gives me hives: life is too short. But Shift-JIS is OK.
- forgotmypw17 5y ago
  There's also ASCII...
- chrismorgan 5y ago
  I like Rust’s approach to UTF-8: strings are rigidly UTF-8, so the general goal is to make encoding and decoding things that you sort out at the I/O boundary, and then inside the library you can have sanity.
  - ncmncm 5y ago
    Rust deserves kudos for inventing the term "WTF-8", describing e.g. directory entries -- maybe supposed to be UTF-8, but obliged to allow invalid sequences.
    - gfody 5y ago
      isn't wtf8 'double utf8' ie the typical utf8 escape encoded in utf8 Ãƒetc? wobble transform 8bit seems like an unnecessary hijacking of a well labeled error state
      - chrismorgan 5y ago
        https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ (which I think you’ve found). People had occasionally used the label before for mojibake of various kinds, but the term was never popular under that meaning. Simon’s work is now a vastly more popular meaning.
        gfody 5y ago
siraben 5y ago
What language is/where is the title image from?
- shadowofneptune 5y ago
  The glyphs, ink, and spacing look very similar to the Zodiac Killer's famous cipher, but I cannot find any perfect matches.
  - magnio 5y ago
    It is indeed the Zodiac cipher, upside down: https://www.pexels.com/photo/photo-of-cryptic-character-codes-and-magnifying-glass-on-table-top-7319068/ https://www.pexels.com/photo/photo-of-cryptic-character-code...
forgotmypw17 5y ago
Thank you so much. I'm mostly using Perl, but I can relate to the problem. I'm working on implementing an ASCII-only and ANSI-only modes in my static html generator, and it's far trickier than I imagined, even with Perl. (My reasons for doing so are backwards compatibility support and lowering the attack surface.)
Kranar 5y ago
>In other words, this snippet of code will do exactly what you expect it to without a single surprise: That reinterpret cast from char* to a char8_t* is undefined behavior. std::u8string_view utf8_input(reinterpret_cast<const char8_t*>(argv[1])); This is not just pedantry either, it was purposely designed this way to allow compilers to better optimize code without worrying about aliasing issues. Link to the paper which explicitly calls out that char8_t be defined as a new type that does not alias with any other type (hence making that reinterpret cast undefined behavior): http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0482r6.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p048...
- 10000truths 5y ago
  `-fno-strict-aliasing` to the rescue!
- arc-in-space 5y ago
  This seems wrong. Regardless of what char8_t is, doesn't char* have a spec-given right to read it? Defining char8_t as a new type is specifically to avoid unnecessarily granting it these all-aliasing powers, but you can still read it as bytes.
  - Kranar 5y ago
    Yes, but as I mentioned earlier only char* has the right to read it, in the snippet I posted, it's char8_t* doing the reading.
- moonchild 5y ago
  char8_t* is allowed to alias char* not because of any special property of char8_t, but because char is allowed to alias any other type (including char8_t).
  - Kranar 5y ago
    As you said, char* can alias any other type, but that does not allow any other type to alias char*. reinterpret_cast<char*>(T*); // Perfectly fine. reinterpret_cast<T*>(char*); // Undefined behavior. Otherwise it would be trivial for any type to alias any other type: reinterpret_cast<T*>(reinterpret_cast<char*>(U*));
SloopJon 5y ago
I wrote some Unicode generators for RapidCheck that worked perfectly well with GCC and Clang on Unix, but not at all with Visual C++ on Windows. The idea was to generate arbitrary code points within various ranges (ASCII, Latin-1, BMP, etc. for shrinking) as UTF-32 / UCS-4 in a u32string, then convert to the datatype and encoding for the API under test--UTF-8 in a string, UTF-16 in a u16string, UTF-something in a wstring, etc. The problem was, some of the conversion facets just aren't supported in the Visual C++ runtime library. I think I ended up doing the UTF-32 to UTF-16 conversion myself. The other thing I ran into recently, is that u"" string literals that worked on GCC / Linux, Clang / Mac, and VC++ 2017 / Windows 2012 did not work for a colleague on VC++ 2019 / Windows 10. An emoji, for example, came out as four char16_t code units (one for each byte of the UTF-8 encoding, I think), instead of two. We ended up using Unicode escapes instead, although the source code is less colorful without a pile of poo. This ztd.text library looks interesting, although it's a little discouraging that the getting started section of the documentation is empty. Is this a header-only library?
nly 5y ago
Why are the examples for ztd.text writing UTF-16 to std::cout? Won't this fail on Linux where UTF-16 is rarely used and terminals typically default to UTF-8? Personally I've enjoyed using the tiny-weeny header only utfcpp[0] for simple manipulation of UTF-8 and unicode-to-unicode conversions, and typically use Boost.Locale[1] when I need to do I/O. 95% of localization problems have almost nothing to do with encoding. Encodings are boring. It's like arguing over JSON vs YAML. To do I/O properly across platforms you may need wrappers for the standard output streams that will handle conversion for you, sure, but... you also need to handle date, time and currency formatting/parsing, message formatting, and to enable translations. See [2] regarding Windows: > All of the examples that come with Boost.Locale are designed for UTF-8 and it is the default encoding used by Boost.Locale. Personally I think doing I/O as UTF-8 on Windows is the right direction, as Microsoft have been enhancing UTF-8 support in Windows for quite a while now. See[3]: > Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. [0] https://github.com/nemtrif/utfcpp https://github.com/nemtrif/utfcpp [1] https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/index.html https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/i... [2] https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/running_examples_under_windows.html https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/r... [3] https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali...
- midjji 5y ago
  Ah ... it seemed like such a beautiful dream for a moment. But yeah, no way to fix this without breaking/deprecating abi /api in C,C++. Which really isn't surprising, char and uchar should always have been byte, char should not have existed. std::string, and all string functions in C should always have contained/ required explicit if default encoding information, and the streams should always have had encoding defaults for its input and output. While the compiler should always have required a non locale based ... and so on. The problem keeps getting worse too, view is a generic and powerful meta, but it means there should never have been such a thing as string_view, except possibly as a template specialization for performance reasons, and of course that would wrapp the proper encoded string, with its explicit encoding of string litteral and so on. That said, with appropriate new api in the standard and with the compiler explicitly requiring explicit encoding specified sources, this could all be solved. The c++ streams are a very useful construct but horribly implemented, so we could just make a new one, and then deprecate the broken part of the standard lib. Its just, this is never a problem you realize when coding on your own, since it only ever hits projects big enough to be multilocal.
lifthrasiir 5y ago
I think the presented design is almost okay, even with the complexity of legacy encodings, but one thing raised my eyebrows: > If your type always has a replacement character, regardless of the situation, it can signal this by writing one of two functions: replacement_code_units() (for any failed encode step), replacement_code_points() (for any failed decode step). They (and their cousins maybe_replacement_code_units/points) do not accept any argument, and that's wrong. There are stateful encodings like ISO/IEC 2022 and HZ where you might need the current encoder state to select a correct replacement character. For example `?` might not be representable in the current state and needs a switch sequence. You can, fortunately, always switch to the desired character set in 2022 with a fixed sequence so you can do have a fixed replacement sequence there, but you can't do so in HZ. It is technically possible to force-switch to the default state when an error is occurred, but not all users will want this behavior. It should obvious to anyone that stateful encodings are in general PITA.
flohofwoe 5y ago
I'd like to see a bit more explanation why the "UTF-8 Everywhere" (https://utf8everywhere.org/ https://utf8everywhere.org/) advice is poor? Outside Windows APIs, UTF-16 is pretty much irrelevant (yes there are a few popular languages around which jumped on the UNICODE bandwagon too early and are now stuck with UTF-16, but those languages usually also offer simple conversion functions to and from other encodings - like UTF-8). UTF-32 has its place as internal runtime string representation, but not for data exchange. UTF-8 has won, and for all the right reasons. UTF-16 is a backward compatibility hack for legacy APIs, and UTF-32 is a special-case encoding for easier runtime-manipulation of string data.
- lifthrasiir 5y ago
  The author claims that the UTF-8 Everywhere manifesto makes people think that they don't need to implement anything but UTF-8 (which is false and not even what the manifesto actually says). I think that the author's claim itself is also false.
- flohofwoe 5y ago
  Btw, I'm almost certain that the first code example of expecting the command line arguments as UTF-8 strings won't work on Windows, instead the argument strings will be 8-bit-ASCII encoded with the current system code page. The only way I got this working reliably with the standard main() entry point was to call the Win32 function GetCommandLineW() and then split into args and convert to UTF-8 myself.
  - flohofwoe 5y ago
    PS: found some example code in the github project which seems to take care of the special Windows situation: https://github.com/soasis/text/blob/main/examples/documentation/source/runtime_locale_encoding.cpp https://github.com/soasis/text/blob/main/examples/documentat... Unfortunately this doesn't look so simple anymore.
    - nly 5y ago
      This code looks like a disaster to me. The issue on Windows is simply that code pages are a per-application setting (or state) and not a system setting. Programs are free to run with, and change between, any "active code page" they want and spit bytes out in that code page. This is all fine and dandy when you're talking to a -A Windows API, since Windows knows the ACP of your application, but it's disaster for I/O and making arbitrary programs talk to one another. Nothing you can do in your code can fix this, it's a contract you have to have with the outside world. On Linux calling setlocale() with a non-UTF-8 locale might not even even work because your /etc/locale.gen file will probably only have a couple of enabled entries, and doing so would be mostly pointless because your terminal and all your other programs are likely using UTF-8. The downside to the Linux approach is that if somebody does actually send you a ISO-8859-1 encoded file you will have to do some conversion. #include <boost/locale.hpp> #include <fstream> #include <iostream> #include <string> int main(int argc, char** argv) { std::ifstream ifs ("test.txt"); std::string str; boost::locale::generator gen; // No dependency on what's in /etc/locale.gen auto loc = gen ("en_US.ISO-8859-1"); ifs.imbue (loc); // Should make formatting input functions work std::getline (ifs, str); str = boost::locale::conv::to_utf<char> (str, loc); for (auto wc: str) { std::cout << sizeof(wc) << ": " << static_cast<uint64_t>(static_cast<unsigned char>(wc)) << "\n"; } }
imron 5y ago
utf16_output.size() * sizeof(char16_t) Keep on keeping on, c++.
- jstimpfle 5y ago
  what's the matter?
  - imron 5y ago
    It’s 100% correct c++. Used as an example to demonstrate a self-described best of breed library, using all of the latest and greatest features of modern c++ (c++20 features even), and yet you still have to multiply the size of the container by the size of the data type inside the container in order to have correct behaviour. It’s error prone, even for experienced and careful programmers, and despite all the advances of modern c++, this sort of thing permeates through almost every non-trivial c++ project. I didn’t read the library’s documentation so maybe there’s an overloaded output operator that prevents people from needing to do this manually, but the fact that this is considered the normal correct way of doing things, is an indictment of c++ as a whole - and I say this as someone who uses c++ daily and who considers himself a safe and careful c++ programmer, who still occasionally makes mistakes due to things like this.
    - jstimpfle 5y ago
      I find you're exaggerating what I don't recall has ever been an issue for me when doing C/C++. Having to compute sizes of dynamic objects in memory is just a normal thing to do. Here is a better practice btw: utf16_output.size()* sizeof utf16_output[0]; This removes the redundant indication of the element type (which can introduce bugs when the type of the array is changed). It gets even better when removing the naming slightly (names indicating types rarely make sense): out.size() * sizeof out[0]; With less magic: plain pointers instead of C++ vector like object, the following is total muscle memory for me: out_count * sizeof *out; I don't see how it is a problem having to do a simple multiplication when what's needed isn't the size that is normally used ("number of elements"), but the size in bytes of the slice. Adding some kind of syntax of library feature to remove this trivial multiplication would add more non-orthogonality, as one would now use sth like out.size_in_bytes() but for custom slices would have to do the calculation manually: n * sizeof out[0]; Note also that in the former case, the need to multiply is removed syntactically, but not at runtime. Personally I don't think there is a worthwhile improvement to have.
ezoe 5y ago
Well, good luck. I lost all hope and trust in C++ Standard committee. I gave up.
ivanche 5y ago
My bold claim: author didn't ever execute first 2 code snippets in the article. Proof: int main (int argv, char* argv[]), i.e. both parameters are called argv.
boxfire 5y ago
5th bullet of "Who is this library for?" at the top of the documentation [1] had me excited: > you want to work with higher-level primitives (code points, graphames) when iterating text that do not break your text apart; In reality grapheme clusters are implemented nowhere and are only in the glossary. I wanted to see how this handled the sprawling complexity that can beget. Someone call me when a library can iterate in that form, which isn't written by committee. [1]: https://ztdtext.readthedocs.io/en/latest/ https://ztdtext.readthedocs.io/en/latest/
- remexre 5y ago
  If you're fine with an extra copy, I think the UTF8PROC_CHARBOUND flag does this in https://github.com/JuliaStrings/utf8proc https://github.com/JuliaStrings/utf8proc ?
  - boxfire 5y ago
    Nice this does the job, probably could be done more efficiently, but I like that its not a kitchen-sink library. Unfortunate that the data generator is certainly not "clean C"... perl, ruby, and julia, all to generate character width table? Ugh. Otherwise this lib does a pretty good job. Thanks for the reference! edit* Yikes https://github.com/JuliaStrings/utf8proc/blob/master/utf8proc.h#L554 https://github.com/JuliaStrings/utf8proc/blob/master/utf8pro... Still this is a good template to maybe cut a (safter) state-based iterator out of their logic...
malkia 5y ago
For paths, I've been relying on this - https://en.cppreference.com/w/cpp/filesystem/path/string https://en.cppreference.com/w/cpp/filesystem/path/string to get the "utf-8" representation of uint16_t filename (UCS-2) - https://en.cppreference.com/w/cpp/filesystem/path/string https://en.cppreference.com/w/cpp/filesystem/path/string and vice versa from "utf-8" you can construct the filename back - https://en.cppreference.com/w/cpp/filesystem/path/path https://en.cppreference.com/w/cpp/filesystem/path/path

98 points by hasheddan 5y ago

There was a lot of discussion with invisible opponents in that text.. Plenty of counter-arguments but only a hint of what is this replying to. Other than that, it seems to present a C++ version of encoding conversion library (like iconv), but instead of handling encodings dynamically it uses C++ types. So not very good for any place with user-specified encoding (like browser or database or text editor), but can be used when the encoding is known at compile time? I can see a few uses of it, but it seems kinda niche for all that grandiose talk about “liberation of the encoding hell”.

aidenn0 5y ago
ztd::text::transcode takes an encoding type, as shown in the second example. In any event, a single switch statement turns a compile-time encoding into a run-time encoding.

Animats 5y ago

If you use Go, Rust, Python 3, or Javascript, this is a non-problem, because those are all native UTF-8. Only for legacy languages is this still a problem.

samatman 5y ago
Would have been better to read the article, because then you'd be in a position to explain how those languages handle transcoding from ShiftJS to UTF-8.
gameman144 5y ago
This is explicitly called out in the article: non-UTF-8 encodings still need to be handled somehow, and this library makes those just as easy to handle as UTF-8.
layoutIfNeeded 5y ago
False. JavaScript uses UTF16: https://mathiasbynens.be/notes/javascript-encoding https://mathiasbynens.be/notes/javascript-encoding
chrismorgan 5y ago
Quite apart from the fact that you’re talking at cross-purposes to the article, you’re wrong about three of the four languages you mention. Go strings aren’t actually UTF-8; they aren’t even Unicode: they can contain arbitrary bytes. In practice, most strings will be UTF-8, but you can’t depend on it at all. Rust strings are strictly UTF-8. I appreciate this greatly. Python 3 strings are sequences of Unicode code points (as distinct from Unicode scalar values), allowing ill-formed Unicode (unmatched surrogates), and its internal representation is a disaster because they decided indexing by code point was worthwhile (it’s not, given the costs), so since CPython 3.3 strings are encoded as ISO-8859-1 (only able to represent U+0000 to U+00FF), UCS-2 (further able to represent U+0100 to U+FFFF) or UCS-4 (able to represent all Unicode code points). JavaScript strings allow ill-formed Unicode, and the public representation can be described as either UCS-2 or ill-formed UTF-16 (almost all string access techniques work in UTF-16 code units, though new stuff is now mostly working in Unicode and UTF-8 terms). I think all major engines now use an internal representation of ISO-8859-1 if the string contains only U+0000 to U+00FF, or ill-formed UTF-16 for anything else; but Servo’s work has demonstrated that it’s possible to shift to WTF-8 (UTF-8 plus allowing unmatched surrogates), saving lots of memory on some pages and simplifying and slightly speeding up some things, but at the cost of random access performance by code unit index.
- Animats 5y ago
  Right, Python 3 looks like it's UTF-8, but CPython has that weird 1, 2 or 4 byte representation. PyPy, though, finally went UTF-8. Not sure how they do indexing. Go strings are supposed to be UTF-8, but it's not enforced. Javascript - bleah.
  - chrismorgan 5y ago
    > Python 3 looks like it's UTF-8 No, it never looked like it was UTF-8. It did look like it was Unicode (of an unspecified encoding), but Python allows unpaired surrogates, which makes for ill-formed Unicode. Python lets you have '\udead'. Well-formed encodings of Unicode (such as UTF-8, UTF-16 and UTF-32) cannot represent U+DEAD. There’s a big difference between “Unicode” and “UTF-8”. > PyPy, though, finally went UTF-8. Oh cool! I remember vaguely hearing the idea being mulled over, years ago. Good to see it happened: https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-.... Wonder what they’ve done about surrogates; the only way they can truly have switched to UTF-8 internally is if they’ve broken compatibility, which I doubt they’ve done; I’m guessing that they actually use WTF-8, not UTF-8. (In the meantime, I’ve changed “Python 3.3” in the grandparent comment to “CPython 3.3” for accuracy.)
1wd 5y ago
Python 3 by default still opens text files using the legacy locale encoding.
jcelerier 5y ago
sure fam https://stackoverflow.com/questions/12053168/how-to-properly-output-a-string-in-a-windows-console-with-go https://github.com/intellij-rust/intellij-rust/issues/766 https://bugs.python.org/issue44275 etc etc

ncmncm 5y ago

Every single thing JeanHeyd does goes far, far beyond what we are resigned to accept from lesser individuals. That said, I am in the rowdy “Only UTF-8, ever, and nothing else” gang. Even thinking about UTF-16 gives me hives: life is too short. But Shift-JIS is OK.

forgotmypw17 5y ago
There's also ASCII...
chrismorgan 5y ago
I like Rust’s approach to UTF-8: strings are rigidly UTF-8, so the general goal is to make encoding and decoding things that you sort out at the I/O boundary, and then inside the library you can have sanity.
- ncmncm 5y ago
  Rust deserves kudos for inventing the term "WTF-8", describing e.g. directory entries -- maybe supposed to be UTF-8, but obliged to allow invalid sequences.
  - gfody 5y ago
    isn't wtf8 'double utf8' ie the typical utf8 escape encoded in utf8 Ãƒetc? wobble transform 8bit seems like an unnecessary hijacking of a well labeled error state
    - chrismorgan 5y ago
      https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ (which I think you’ve found). People had occasionally used the label before for mojibake of various kinds, but the term was never popular under that meaning. Simon’s work is now a vastly more popular meaning.
      - gfody 5y ago

siraben 5y ago

What language is/where is the title image from?

shadowofneptune 5y ago
The glyphs, ink, and spacing look very similar to the Zodiac Killer's famous cipher, but I cannot find any perfect matches.
- magnio 5y ago
  It is indeed the Zodiac cipher, upside down: https://www.pexels.com/photo/photo-of-cryptic-character-codes-and-magnifying-glass-on-table-top-7319068/ https://www.pexels.com/photo/photo-of-cryptic-character-code...

forgotmypw17 5y ago

Thank you so much. I'm mostly using Perl, but I can relate to the problem. I'm working on implementing an ASCII-only and ANSI-only modes in my static html generator, and it's far trickier than I imagined, even with Perl. (My reasons for doing so are backwards compatibility support and lowering the attack surface.)

Kranar 5y ago

>In other words, this snippet of code will do exactly what you expect it to without a single surprise: That reinterpret cast from char* to a char8_t* is undefined behavior. std::u8string_view utf8_input(reinterpret_cast<const char8_t*>(argv[1])); This is not just pedantry either, it was purposely designed this way to allow compilers to better optimize code without worrying about aliasing issues. Link to the paper which explicitly calls out that char8_t be defined as a new type that does not alias with any other type (hence making that reinterpret cast undefined behavior): http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0482r6.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p048...

10000truths 5y ago
`-fno-strict-aliasing` to the rescue!
arc-in-space 5y ago
This seems wrong. Regardless of what char8_t is, doesn't char* have a spec-given right to read it? Defining char8_t as a new type is specifically to avoid unnecessarily granting it these all-aliasing powers, but you can still read it as bytes.
- Kranar 5y ago
  Yes, but as I mentioned earlier only char* has the right to read it, in the snippet I posted, it's char8_t* doing the reading.
moonchild 5y ago
char8_t* is allowed to alias char* not because of any special property of char8_t, but because char is allowed to alias any other type (including char8_t).
- Kranar 5y ago
  As you said, char* can alias any other type, but that does not allow any other type to alias char*. reinterpret_cast<char*>(T*); // Perfectly fine. reinterpret_cast<T*>(char*); // Undefined behavior. Otherwise it would be trivial for any type to alias any other type: reinterpret_cast<T*>(reinterpret_cast<char*>(U*));

SloopJon 5y ago

I wrote some Unicode generators for RapidCheck that worked perfectly well with GCC and Clang on Unix, but not at all with Visual C++ on Windows. The idea was to generate arbitrary code points within various ranges (ASCII, Latin-1, BMP, etc. for shrinking) as UTF-32 / UCS-4 in a u32string, then convert to the datatype and encoding for the API under test--UTF-8 in a string, UTF-16 in a u16string, UTF-something in a wstring, etc. The problem was, some of the conversion facets just aren't supported in the Visual C++ runtime library. I think I ended up doing the UTF-32 to UTF-16 conversion myself. The other thing I ran into recently, is that u"" string literals that worked on GCC / Linux, Clang / Mac, and VC++ 2017 / Windows 2012 did not work for a colleague on VC++ 2019 / Windows 10. An emoji, for example, came out as four char16_t code units (one for each byte of the UTF-8 encoding, I think), instead of two. We ended up using Unicode escapes instead, although the source code is less colorful without a pile of poo. This ztd.text library looks interesting, although it's a little discouraging that the getting started section of the documentation is empty. Is this a header-only library?

nly 5y ago

Why are the examples for ztd.text writing UTF-16 to std::cout? Won't this fail on Linux where UTF-16 is rarely used and terminals typically default to UTF-8? Personally I've enjoyed using the tiny-weeny header only utfcpp[0] for simple manipulation of UTF-8 and unicode-to-unicode conversions, and typically use Boost.Locale[1] when I need to do I/O. 95% of localization problems have almost nothing to do with encoding. Encodings are boring. It's like arguing over JSON vs YAML. To do I/O properly across platforms you may need wrappers for the standard output streams that will handle conversion for you, sure, but... you also need to handle date, time and currency formatting/parsing, message formatting, and to enable translations. See [2] regarding Windows: > All of the examples that come with Boost.Locale are designed for UTF-8 and it is the default encoding used by Boost.Locale. Personally I think doing I/O as UTF-8 on Windows is the right direction, as Microsoft have been enhancing UTF-8 support in Windows for quite a while now. See[3]: > Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. [0] https://github.com/nemtrif/utfcpp https://github.com/nemtrif/utfcpp [1] https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/index.html https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/i... [2] https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/running_examples_under_windows.html https://www.boost.org/doc/libs/1_76_0/libs/locale/doc/html/r... [3] https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali...

midjji 5y ago
Ah ... it seemed like such a beautiful dream for a moment. But yeah, no way to fix this without breaking/deprecating abi /api in C,C++. Which really isn't surprising, char and uchar should always have been byte, char should not have existed. std::string, and all string functions in C should always have contained/ required explicit if default encoding information, and the streams should always have had encoding defaults for its input and output. While the compiler should always have required a non locale based ... and so on. The problem keeps getting worse too, view is a generic and powerful meta, but it means there should never have been such a thing as string_view, except possibly as a template specialization for performance reasons, and of course that would wrapp the proper encoded string, with its explicit encoding of string litteral and so on. That said, with appropriate new api in the standard and with the compiler explicitly requiring explicit encoding specified sources, this could all be solved. The c++ streams are a very useful construct but horribly implemented, so we could just make a new one, and then deprecate the broken part of the standard lib. Its just, this is never a problem you realize when coding on your own, since it only ever hits projects big enough to be multilocal.

lifthrasiir 5y ago

I think the presented design is almost okay, even with the complexity of legacy encodings, but one thing raised my eyebrows: > If your type always has a replacement character, regardless of the situation, it can signal this by writing one of two functions: replacement_code_units() (for any failed encode step), replacement_code_points() (for any failed decode step). They (and their cousins maybe_replacement_code_units/points) do not accept any argument, and that's wrong. There are stateful encodings like ISO/IEC 2022 and HZ where you might need the current encoder state to select a correct replacement character. For example `?` might not be representable in the current state and needs a switch sequence. You can, fortunately, always switch to the desired character set in 2022 with a fixed sequence so you can do have a fixed replacement sequence there, but you can't do so in HZ. It is technically possible to force-switch to the default state when an error is occurred, but not all users will want this behavior. It should obvious to anyone that stateful encodings are in general PITA.

flohofwoe 5y ago

I'd like to see a bit more explanation why the "UTF-8 Everywhere" (https://utf8everywhere.org/ https://utf8everywhere.org/) advice is poor? Outside Windows APIs, UTF-16 is pretty much irrelevant (yes there are a few popular languages around which jumped on the UNICODE bandwagon too early and are now stuck with UTF-16, but those languages usually also offer simple conversion functions to and from other encodings - like UTF-8). UTF-32 has its place as internal runtime string representation, but not for data exchange. UTF-8 has won, and for all the right reasons. UTF-16 is a backward compatibility hack for legacy APIs, and UTF-32 is a special-case encoding for easier runtime-manipulation of string data.

lifthrasiir 5y ago
The author claims that the UTF-8 Everywhere manifesto makes people think that they don't need to implement anything but UTF-8 (which is false and not even what the manifesto actually says). I think that the author's claim itself is also false.
flohofwoe 5y ago
Btw, I'm almost certain that the first code example of expecting the command line arguments as UTF-8 strings won't work on Windows, instead the argument strings will be 8-bit-ASCII encoded with the current system code page. The only way I got this working reliably with the standard main() entry point was to call the Win32 function GetCommandLineW() and then split into args and convert to UTF-8 myself.
- flohofwoe 5y ago
  PS: found some example code in the github project which seems to take care of the special Windows situation: https://github.com/soasis/text/blob/main/examples/documentation/source/runtime_locale_encoding.cpp https://github.com/soasis/text/blob/main/examples/documentat... Unfortunately this doesn't look so simple anymore.
  - nly 5y ago
    This code looks like a disaster to me. The issue on Windows is simply that code pages are a per-application setting (or state) and not a system setting. Programs are free to run with, and change between, any "active code page" they want and spit bytes out in that code page. This is all fine and dandy when you're talking to a -A Windows API, since Windows knows the ACP of your application, but it's disaster for I/O and making arbitrary programs talk to one another. Nothing you can do in your code can fix this, it's a contract you have to have with the outside world. On Linux calling setlocale() with a non-UTF-8 locale might not even even work because your /etc/locale.gen file will probably only have a couple of enabled entries, and doing so would be mostly pointless because your terminal and all your other programs are likely using UTF-8. The downside to the Linux approach is that if somebody does actually send you a ISO-8859-1 encoded file you will have to do some conversion. #include <boost/locale.hpp> #include <fstream> #include <iostream> #include <string> int main(int argc, char** argv) { std::ifstream ifs ("test.txt"); std::string str; boost::locale::generator gen; // No dependency on what's in /etc/locale.gen auto loc = gen ("en_US.ISO-8859-1"); ifs.imbue (loc); // Should make formatting input functions work std::getline (ifs, str); str = boost::locale::conv::to_utf<char> (str, loc); for (auto wc: str) { std::cout << sizeof(wc) << ": " << static_cast<uint64_t>(static_cast<unsigned char>(wc)) << "\n"; } }

imron 5y ago

utf16_output.size() * sizeof(char16_t) Keep on keeping on, c++.

jstimpfle 5y ago
what's the matter?
- imron 5y ago
  It’s 100% correct c++. Used as an example to demonstrate a self-described best of breed library, using all of the latest and greatest features of modern c++ (c++20 features even), and yet you still have to multiply the size of the container by the size of the data type inside the container in order to have correct behaviour. It’s error prone, even for experienced and careful programmers, and despite all the advances of modern c++, this sort of thing permeates through almost every non-trivial c++ project. I didn’t read the library’s documentation so maybe there’s an overloaded output operator that prevents people from needing to do this manually, but the fact that this is considered the normal correct way of doing things, is an indictment of c++ as a whole - and I say this as someone who uses c++ daily and who considers himself a safe and careful c++ programmer, who still occasionally makes mistakes due to things like this.
  - jstimpfle 5y ago
    I find you're exaggerating what I don't recall has ever been an issue for me when doing C/C++. Having to compute sizes of dynamic objects in memory is just a normal thing to do. Here is a better practice btw: utf16_output.size()* sizeof utf16_output[0]; This removes the redundant indication of the element type (which can introduce bugs when the type of the array is changed). It gets even better when removing the naming slightly (names indicating types rarely make sense): out.size() * sizeof out[0]; With less magic: plain pointers instead of C++ vector like object, the following is total muscle memory for me: out_count * sizeof *out; I don't see how it is a problem having to do a simple multiplication when what's needed isn't the size that is normally used ("number of elements"), but the size in bytes of the slice. Adding some kind of syntax of library feature to remove this trivial multiplication would add more non-orthogonality, as one would now use sth like out.size_in_bytes() but for custom slices would have to do the calculation manually: n * sizeof out[0]; Note also that in the former case, the need to multiply is removed syntactically, but not at runtime. Personally I don't think there is a worthwhile improvement to have.

ezoe 5y ago

Well, good luck. I lost all hope and trust in C++ Standard committee. I gave up.

ivanche 5y ago

My bold claim: author didn't ever execute first 2 code snippets in the article. Proof: int main (int argv, char* argv[]), i.e. both parameters are called argv.

boxfire 5y ago

5th bullet of "Who is this library for?" at the top of the documentation [1] had me excited: > you want to work with higher-level primitives (code points, graphames) when iterating text that do not break your text apart; In reality grapheme clusters are implemented nowhere and are only in the glossary. I wanted to see how this handled the sprawling complexity that can beget. Someone call me when a library can iterate in that form, which isn't written by committee. [1]: https://ztdtext.readthedocs.io/en/latest/ https://ztdtext.readthedocs.io/en/latest/

remexre 5y ago
If you're fine with an extra copy, I think the UTF8PROC_CHARBOUND flag does this in https://github.com/JuliaStrings/utf8proc https://github.com/JuliaStrings/utf8proc ?
- boxfire 5y ago
  Nice this does the job, probably could be done more efficiently, but I like that its not a kitchen-sink library. Unfortunate that the data generator is certainly not "clean C"... perl, ruby, and julia, all to generate character width table? Ugh. Otherwise this lib does a pretty good job. Thanks for the reference! edit* Yikes https://github.com/JuliaStrings/utf8proc/blob/master/utf8proc.h#L554 https://github.com/JuliaStrings/utf8proc/blob/master/utf8pro... Still this is a good template to maybe cut a (safter) state-based iterator out of their logic...

malkia 5y ago

For paths, I've been relying on this - https://en.cppreference.com/w/cpp/filesystem/path/string https://en.cppreference.com/w/cpp/filesystem/path/string to get the "utf-8" representation of uint16_t filename (UCS-2) - https://en.cppreference.com/w/cpp/filesystem/path/string https://en.cppreference.com/w/cpp/filesystem/path/string and vice versa from "utf-8" you can construct the filename back - https://en.cppreference.com/w/cpp/filesystem/path/path https://en.cppreference.com/w/cpp/filesystem/path/path