7 ms·

And that's why I'm not a fan of C for writing any kind of program: Every time you need to touch a string, which is an effortless task in any semi-modern languag

• saagarjha 2y ago
  The code in the post does indeed have a bug if the goal is to copy UTF-8 data. However the bug is quite subtle and unlikely to occur in practice, and from your comment I suspect you don’t actually know what that problem is (because you seem to be pointing at a different issue which is actually irrelevant). Maybe you or someone else can describe what the real concern is?
  • Levitating 2y ago
    Partially copying a 2-byte character?
    • saagarjha 2y ago
      UTF-8 works fine if you truncate a codepoint because the encoding scheme lets you detect this. The problem is more subtle than that (hint: it involves a 1-byte codepoint).
      • moralestapia 2y ago
        >hint: it involves a 1-byte codepoint Which is?
        • saagarjha 2y ago
          [flagged]
      • Retr0id 2y ago
        If you're alluding to NUL, I don't really see the issue? Yes, many languages allow strings (UTF-8 or otherwise) to contain null bytes, and C's str*() functions obviously do not, but null-termination vs not is an orthogonal issue to ASCII vs UTF-8. i.e. Yes it's (depending on context) an issue that C str*() cannot handle strings with embedded null bytes, but that's not a UTF-8-specific issue.
      • lalaland1125 2y ago
        Truncating a UTF-8 codepoint is not fine because most software is not tested with partially broken UTF-8 so international users will likely run into many bugs. Especially because concatenation is a very common operation so those sliced codepoints will be everywhere, including in the middle of text.
        • saagarjha 2y ago
          Morally I view “what do I do with my truncated string” to be a separate issue from “how do I truncate the string” as described in the article. Like, yes, you absolutely should not concatenate after doing this operation. But maybe you shouldn’t be showing the user a truncated string either even if it’s all ASCII. The question of “did you make an unparseable UTF-8 string” is answered with “no” and the more complicated but also more interesting question of “did you actually want this” remains unanswered.
          • Levitating 2y ago
            This is fair, the article takes truncating a string to fit in a status bar as an example.
        • actionfromafar 2y ago
          Also consider Unicode is not only international characters, but superscripts and other stuff ♥ᵃ a: there was a list somewhere over which characters hackernews allows?
      • MathMonkeyMan 2y ago
        The null code point? That would be pedantic even by my standards.
        • saagarjha 2y ago
          Look I had to include it or someone is going to do a whole pedantic comment about how C can’t actually represent UTF-8 correctly
          • pdpi 2y ago
            By that metric, C can't represent ASCII correctly either, because there's no particular reason you couldn't have a NUL character somewhere inside a string.
            • TeMPOraL 2y ago
              Indeed it can't. Many developers were bitten by this, and still are; plenty of critical bugs and security vulnerabilities rely on this quirk too.
              • account42 2y ago
                Technically, C can. It's just C strings that are limited.
                • pdpi 2y ago
                  Sure, in the exact same way that C can handle unicode just fine too. The problem is, as always, C strings.
          • Sebb767 2y ago
            Actually, by just alluding to the bug without saying it explicitly, you managed to both be pedantic and not avoid the discussion. This is not meant as a personal attack; I just want to point out how it looks on a casual reading :)
      • Pesthuf 2y ago
        A function that can turn a correctly formatted UTF-8 string into a malformed UTF-8 string is, in my opinion, broken.
        • saagarjha 2y ago
          One problem here is that the string may not have been a correctly formatted UTF-8 string to begin with. No, not that it can contain any bytes-I mean, it might be ascribed even more than just decoding correctly. Maybe it is supposed to have the grapheme clusters preserved. Maybe the truncation should peel off the last file component because the string holds a path. The operation of “doing a dumb truncation” can be broken if you look at it from plenty of ways, and I don’t disagree with you, but I do want to make clear that the issue isn’t memcpy is breaking it but that if you need x, y, z maybe you’re reaching for the wrong tool. And conversely there is nothing inherently wrong with using it if you are going to use it in a way that is resilient to that kind of truncation.
        • account42 2y ago
          What about a function that can turn a correctly spelled english sentence into a malformed english sentence? If you truncate to a fixed length this comes with the territory.
      • deleted 2y ago
        [deleted]
    • deleted 2y ago
      [deleted]
  • keybored 2y ago
    > However the bug is quite subtle and unlikely to occur in practice, Unlikely to occur? Do you think that will put the naysayers at ease? > , and from your comment I suspect you don’t actually know what that problem is (because you seem to be pointing at a different issue which is actually irrelevant). Maybe you or someone else can describe what the real concern is? Amazing that people have the patience for this.
    • saagarjha 2y ago
      ¯\_(ツ)_/¯
• cyberpunk 2y ago
  I mean just use bstrlib why is it difficult?
  • rwmj 2y ago
    Because no one can agree on which string library / abstraction to use, if they even use one at all which is rare.
  • dark-star 2y ago
    I mean just use C++ w/ std::string why is it difficult?
    • smallstepforman 2y ago
      std::u8string
      • account42 2y ago
        No, std::string.
    • bluGill 2y ago
      Most C programs compile and work correctly compiled as C++. The exceptions are often places you shouldn't write code like that, sometimes even outright bugs that C++'s stricter type system caught for you.
  • Pesthuf 2y ago
    I'd be happy if developers did that, but they don't: So many C developer seem to think they're "smart enough" to handle strings correctly using only the standard library, not needing "training wheels" like the plebs - but then where do the countless string related memory safety errors and all the broken Unicode handling come from that plague every C program, no matter how clever the programmers, no matter how tough the review process, no matter how much static analysis is used? The issue is that many C developers simply can't seem to admit that C string handling, as it was invented many decades ago, was simply fundamentally flawed. So easy to misuse with catastrophic results, even for plain 1-byte-strings. It has nothing to do with how "smart" you are or how careful you are.
• taeric 2y ago
  Honestly, any language where you are manipulating strings is a footgun. Especially if you are manipulating text, and not character data. You can avoid many errors, but almost certainly not all. And as soon as you have someone that thinks they can piecewise localize any text, heaven help you.
  • bruce343434 2y ago
    With the up and coming browser LLM API, you could just have the LLM translate everything on the frontend. One of the few tasks that LLMs are consistently good at.
    • taeric 2y ago
      I feel this successfully upgrades a gun into a canon?
      • TeMPOraL 2y ago
        A modern artillery gun, where each shell costs upwards of $1K apiece, and lord have mercy on you if it ever goes off inside the barrel.
    • layer8 2y ago
      LLMs are not “consistently good” at translating UI text, because a proper translation requires understanding the application and the application context of each piece of text.
      • colejohnson66 2y ago
        Yes. Context is very important. Otherwise, you end up compressing files into a "postcode file".[0] [0]: https://news.ycombinator.com/item?id=36231313 https://news.ycombinator.com/item?id=36231313
• eadmund 2y ago
  > I'm not a fan of C for writing any kind of program C is okay — not fan-worthy, but okay — for one specific kind of program: a mostly-portable implementation of a real language in which to write every other kind of program. that’s not nothing: implementing a backend for every CPU-OS pair one wants to support is a pain, and a different skillset from writing, say, a web browser or text editor. I wonder how much C has held back the development of computing.
  • shermantanktop 2y ago
    C is fine. Yes, it has undefined behaviors and encourages subtle thinkos. But it’s a tool and sometimes it’s the right tool. It’s the C hackers who insist on using it for inappropriate applications who create the problems. Anything string-heavy and internationalized is a terrible place for C, even if you have spent 30 years dancing through the C minefield and are sure you’ll get it right this time.
    • fl0ki 2y ago
      The suckless.org initiative is a great example of this. Boasting about how certain software inherently sucks less because it has minimalist C code, rather than having defense-in-depth against security holes. Newsflash, a few decades ago the majority of software was the minimalist C that they claim sucks less. There are many reasons we moved on from that world, security firmly among them.
  • Zambyte 2y ago
    > I wonder how much C has held back the development of computing. Lisp is older than C. It didn't just hold computing back, it actively walked backwards.
• deleted 2y ago
  [deleted]
• Murk 2y ago
  In my opinion, and as a general rule. You can't really ever truncate a user facing string correctly because it depends on language specifics. Hence it is my suggestion not to truncate user facing strings - and in fact you may want to consider them as binary blobs. On the other hand, as a matter of practicability sometimes you may have to, but certainly avoid doing so before it hits the presentation layer, and know that it may not be language correct. There are many strings that are not user facing, which you expect to be of a certain nature, e.g. ASCII based protocols, and therefore you know what to do with them. So the multi byte situation and strcpy, or std::string, or any other "standard" string function isn't really relevant as it's some other libraries duty. the task of truncation and otherwise formatting UI strings is the preserve of the rendering layer(s).
  • hgs3 2y ago
    Yes, truncating a user facing string requires more consideration regardless of programming language. For example do you truncate at the grapheme, word, sentence, new line, paragraph, or something else? How do you indicate truncation to the user, with an ellipsis perhaps? If you use an ellipsis is it appended to the original string or drawn by the GUI toolkit? Note that the Unicode grapheme cluster, word, sentence, and line break algorithms are locale specific. Now consider how often programmers casually truncate strings, even in high-level languages, without accounting for the locale.
• smaudet 2y ago
  > like this one, only ever works with ASCII, no attempt whatsoever to handle UTF-8 I mean if your viewpoint is to handle heavy UIs then yes maybe C is not your go-to language. However, there are other solutions to the character space problem besides UTF-8. Rather than complaining that your screwdriver can't bang nails, perhaps try using a screw? No, you won't be able to support a language like chinese within a 256 bit charset, but perhaps it's not important to in all situations. The strength of ascii is that its small, simple. If that's too anglo-centric for you, maybe there's a better symbolic tiny charset that is more applicable. I'd support something like https://lojban.io/ https://lojban.io/ becoming more prevalent, but there could be advantages to having a simple(r) symbolic transmission language not burdened with anglo-centric concepts.
  • anamexis 2y ago
    Advocating a new global constructed language to accomodate C's shortcomings seems like the wrong direction to be thinking.
    • smaudet 2y ago
      Less a global constructed language, more a "better" encoding. Base64 works really well for arbitrary binary-in-text encoding, for instance.
      • estebank 2y ago
        Unicode is that better encoding. The "small and efficient per locale encoding" that you proposed was the status quo, and was an endless source of mojibake. There is a reason we moved away from that.
        • smaudet 2y ago
          I think there is a misunderstanding, which I tried to address but evidentally failed. UTF-8 is fine for a display encoding. However, not every string encoding need be a display encoding, which the parent post seems to not be considering. You could also have multiple display encodings, if it makes sense to (a tool only intended for use in a certain part of the world for instance), however that is not what I mean.
• _gabe_ 2y ago
  > It's also pretty telling that every article that tries to explain how to safely copy or concat strings in C, like this one, only ever works with ASCII, no attempt whatsoever to handle UTF-8 and keep code points together, let alone grapheme clusters. Can you expand on this? Why does it matter to keep code points and grapheme clusters together in the case of truncation? If you’re already truncating the string, then you can just copy as many bytes as possible. Then, later when you interpret that string, you’ll hit a malformed codepoint and ignore it. I guess what you might be getting at is that if you have a codepoint sequence, then you shouldn’t copy the bytes if they can’t all fit in the truncated string? I feel like this is an edge case and not “the reason almost all C software has problems with non-English strings”. 99% of the time, copying up until the null byte is fine, whether or not the string is UTF-8 or ASCII. The reason most C software doesn’t work with non-English strings is because the developer never added support. The bytes are still there, they just need to be interpreted correctly in the UI portions of the code.
  • Retr0id 2y ago
    Truncating mid-codepoint produces an invalid utf8 sequence, which some decoders will silently ignore or replace with <?> glyphs, and others will fail loudly. Truncating mid-grapheme-cluster can change semantics in unexpected ways. For example, the "family" emoji(s) can be encoded as a set of conjoined codepoints for each family member: https://stackoverflow.com/questions/49958287/printing-family-emoji-with-u200d-zero-width-joiner-directly-vs-via-list https://stackoverflow.com/questions/49958287/printing-family.... Depending on where you truncate, you might just get the father, which probably won't cause you any serious issues but it might cause confusion depending on context. (IMHO, if you're truncating a way where this would matter, you should probably be truncating in screen-space, i.e., way higher up the stack)
    • _gabe_ 2y ago
      Yep, this all makes sense and is what I was thinking as well. I also think you’re right about truncation being a better idea in the rendering state rather than in the bytes themselves. Either way, truncation of strings being displayed to the user requires extra care no matter what language you’re using. As far as this being the reason most C code doesn’t handle UTF-8 though, I’m still skeptical.
• mrkeen 2y ago
  In defence of C, it should be oblivious to different string encodings (and I thought it was). Coreutils should continue to do their job, and not need to be recompiled against whatever encoding is in fashion.
  • johnnyanmac 2y ago
    The standards, as usual for C, are paper-thin, doesn't even guarantee ordinal value for the characters specified (except 0-9) despite implementing strcmp. But GCC/MSVC do add some options on top of the standard, which includes some attempt at interpreting different encodings. Then again, I was surprised to find support for literal encodings in C23. So perhaps my knowledge was even more outdated than I anticipated.
• hifromwork 2y ago
  Well, C doesn't really have strings, just pointers. Calling them "strings" is just an abstraction for us. They don't even have to be null-terminated (even though by convention they often are)! And this is precisely what I want, in some cases. I use C when I need a low-level byte wrangling code that tightly interfaces with the operating system, or when I can't afford to allocate memory at will (like all effortless string-handling languages do under the hood), or when I don't want any runtime and need my code to behave like a shellcode, basically, or... I guess C is still used because people find it useful. No need to be a fan of it.
  • f1shy 2y ago
    > I can't afford to allocate memory at will This was my mantra for the last 30 years. But now, even the tiniest thing has 1GB, linux and virtual memory. Except for very edge cases, this excuse is rapidly dying
    • cess11 2y ago
      When I browse my local electronics web shops I see a lot of devices that couldn't boot a Linux kernel, some of which I wonder whether I could drive with a small consumer grade solar panel.
    • mrkeen 2y ago
      But the waste adds up. There are a few projects at work whose integration tests I can't run on my work machine because it doesn't have enough ram (16GB) so I run them on my home PC instead. And that's even when I close IntelliJ, my web browsers, and kill all the background processes I can.
  • ben_bai 2y ago
    by convention: if it's not NUL terminated, it's not a c-string.
• wruza 2y ago
  Not that C was wise with strings, but slicing strings in general is a bad idea unless you’re doing it strictly at ascii borders, e.g. to capitalize windows disk letter or to split by cr?lf. Modern text is yet another “falsehoods programmers believe in” material.
• deleted 2y ago
  [deleted]