9 ms·

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actual

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a proxy for something else, and whenever I'm thinking shallowly enough to want it (small scripts, mostly-ASCII, mostly-English, mostly-obvious failure modes, etc) I like grapheme cluster being the sensible default thing that people probably expect, on average.

baq 1y ago
ASCII is very convenient when it fits in the solution space (it’d better be, it was designed for a reason), but in the global international connected computing world it doesn’t fit at all. The problem is all the tutorials, especially low level ones, assume ASCII so 1) you can print something to the console and 2) to avoid mentioning that strings are hard so folks don’t get discouraged. Notably Rust did the correct thing by defining multiple slightly incompatible string types for different purposes in the standard library and regularly gets flak for it.
- eru 1y ago
  Python 3 deals with this reasonable sensibly, too, I think. They use UTF-8 by default, but allow you to specify other encodings.
  - xigoi 1y ago
    I prefer languages where strings are simply sequences of bytes and you get to decide how to interpret them.
    - afiori 1y ago
      I would like an utf-8 optimized bag of bytes where arbitrary byte operations are possible but the buffer keeps track of whether is it valid utf-8 or not (for every edit of n bytes it should be enough to check about n+8 bytes to validate) then utf-8 then utf-8 encoding/decoding becomes a noop and utf-8 specific apis can check quickly is the string is malformed or not.
      - account42 1y ago
        But why care if it's malformed UTF-8? And specifically, what do you want to happen when you get a malformed UTF-8 string. Keep in mind that UTF-8 is self-synchronizing so even if you encode strings into a larger text-based format without verifying them it will still be possible to decode the document. As a user I normally want my programs to pass on the string without mangling it further. Some tool throwing fatal errors because some string I don't actually care about contains an invalid UTF-8 byte sequence is the last thing I want. With strings being an arbitrary bag of bytes many programs can support arbitrary encodings or at least arbitrary ASCII-supersets without any additional effort.
        afiori 1y ago
        The main issue I can see is not garbage bytes in text but mixing of incompatible encoding eg splicing latin-1 bytes in a utf-8 string. My understanding of the current "always and only utf-8/unicode" zeitgeist is that is comes mostly from encoding issues among which the complexity of detecting encoding. I think that the current status quo is better than what came before, but I also think it could be improved.
    - account42 1y ago
      Yes, I always roll my eyes when people complain that C strings or C++'s std::string/string_view don't have Unicode support. They are bags of bytes with support for concatenation. Any other transformation isn't going to have a "correct" way to do it so you need to be aware of what you want anyway.
      - astrange 1y ago
        C strings are not bags of bytes because they can't contain 0x00.
    - bawolff 1y ago
      Me too. The languages that i really dont get are those that force valid utf-8 everywhere but dont enforce NFC. Which is most of them but seems like the worst of both worlds. Non normalized unicode is just as problematic as non validated unicode imo.
    - jibal 1y ago
      Python has byte arrays that allow for that, in addition to strings consisting of arrays of Unicode code points.
    - zahlman 1y ago
      Such languages do not have strings. Definitionally a string is a sequence of characters, and more than 256 characters exist. A byte sequence is just an encoding; if you are working with that encoding directly and have to do the interpretation yourself, you are not using a string. But if you do want a sequence of bytes for whatever reason, you can trivially obtain that in any version of Python.
      - capitainenemo 1y ago
        My experience personally with python3 (and repeated interactions with about a dozen python programmers, including core contributors) is that python3 does not let you trivially work with streams of bytes, esp if you need to do character set conversions, since a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. The last attempt was much larger, still failed, and they thought they could probably do it, but it would require far more code and was not worth their effort. I'll probably just use rust for that script if python2 ever gets dropped by my distro. Reminds me of https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/ https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journ...
        zahlman 1y ago
        > a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. Show me.
        capitainenemo 1y ago
        Heh. It always starts this way... then they confidently send me something that breaks on testing it, then half a dozen more iterations, then "python2 is doing the wrong thing" or, "I could get this working but it isn't worth the effort" but sure, let's do this one more time. Could be they were all missing something obvious - wouldn't know, I avoid python personally, apart from when necessary like with LLM glue. https://pastebin.com/j4Lzb5q1 https://pastebin.com/j4Lzb5q1 This is a script created by someone on #nethack a long time ago. It works great with other things as well like old BBS games. It was intended to transparently rewrite single byte encodings to multibyte with an optional conversion array.
  - ynik 1y ago
    Python 3 internally uses UTF-32. When exchanging data with the outside world, it uses the "default encoding" which it derives from various system settings. This usually ends up being UTF-8 on non-Windows systems, but on weird enough systems (and almost always on Windows), you can end up with a default encoding other than UTF-8. "UTF-8 mode" (https://peps.python.org/pep-0540/ https://peps.python.org/pep-0540/) fixes this but it's not yet enabled by default (this is planned for Python 3.15).
    - arcticbull 1y ago
      Apparently Python uses a variety of internal representations depending on the string itself. I looked it up because I saw UTF-32 and thought there's no way that's what they do -- it's pretty much always the wrong answer. It uses Latin-1 for ASCII strings, UCS-2 for strings that contain code points in the BMP and UCS-4 only for strings that contain code points outside the BMP. It would be pretty silly for them to explode all strings to 4-byte characters.
      - account42 1y ago
        It conceptually uses arrays of code points, which need up to 24 bits. Optimizing the storage to use smaller integers when possible is an implementation detail.
        jibal 1y ago
        Python3 is specified to use arrays of 8, 16, or 32 bit units, depending on the largest code point in the string. As a result, all code points in all strings are O(1) indexable. The claim that "Python 3 internally uses UTF-32" is simply false.
        zahlman 1y ago
        > code points, which need up to 24 bits They need at most 21 bits. The bits may only be available in multiples of 8, but the implementation also doesn't byte-pack them into 24-bit units, so that's moot.
      - 1y ago
- account42 1y ago
  > in the global international connected computing world it doesn’t fit at all. I disagree. Not all text is human prose. For example, there is nothing wrong with an programming language that only allows ASCII in the source code and many downsides to allowing non-ASCII characters outside string constants or comments.
  - simonask 1y ago
    This is American imperialism at its worst. I'm serious. Lots of people around the world learn programming from sources in their native language, especially early in their career, or when software development is not their actual job. Enforcing ASCII is the same as enforcing English. How would you feel if all cooking recipes were written in French? If all music theory was in Italian? If all industrial specifications were in German? It's fine to have a dominant language in a field, but ASCII is a product of technical limitations that we no longer have. UTF-8 has been an absolute godsend for human civilization, despite its flaws.
    - account42 1y ago
      Actually, it would be great to have a lingua franca in every field that all participants can understand. Are you also going to complain that biologists and doctors are expected to learn some rudimentary Latin? English being dominant in computing is absolutely a strength and we gain nothing by trying to combat that. Having support for writing your code in other languages is not going to change that most libraries will use English and and most documentation will be in English and most people you can ask for help will understand English. If you want to participate and refuse to learn English you are only shooting yourself in the foot - and if you are going to learn English you may as well do it from the beginning. Also due to the dominance of English and ASCII in computing history, most languages already have ASCII-alternatives for their writing so even if you need to refer to non-English names you can do that using only ASCII.
      - simonask 1y ago
        Well, the problem is that what you are advocating is also that knowing Latin would be a prerequisite for studying medicine, which it isn't anywhere. That's the equivalent. Doctors learn a (very limited) Latin vocabulary as they study and work. You are severely underestimate how far you can get without any real command of the English language. I agree that you can't become really good without it, just like you can't do haute cuisine without some French, but the English language is a huge and unnecessary barrier of entry that you would put in front of everyone in the world who isn't submerged in the language from an early age. Imagine learning programming using only your high school Spanish. Good luck.
- deleted 1y ago
  [deleted]
- flohofwoe 1y ago
  ASCII is totally fine as encoding for the lower 127 UNICODE code points. If you need to go above those 127 code points, use a different encoding like UTF-8. Just never ever use Extended ASCII (8-bits with codepages).
- craftkiller 1y ago
  > Notably Rust did the correct thing In addition to separate string types, they have separate iterator types that let you explicitly get the value you want. So: String.len() == number of bytes String.bytes().count() == number of bytes String.chars().count() == number of unicode scalar values String.graphemes().count() == number of graphemes (requires unicode-segmentation which is not in the stdlib) String.lines().count() == number of lines Really my only complaint is I don't think String.len() should exist, it's too ambiguous. We should have to explicitly state what we want/mean via the iterators.
  - westurner 1y ago
    String.graphemes().count() That's a real nice API. (Similarly, python has @ for matmul but there is not an implementation of matmul in stdlib. NumPy has a matmul implementation so that the `@` operator works.) ugrapheme and ucwidth are one way to get the graphene count from a string in Python. It's probably possible to get the grapheme cluster count from a string containing emoji characters with ICU?
    - dhosek 1y ago
      Any correctly designed grapheme cluster handles emoji characters. It’s part of the spec (says the guy who wrote a Unicode segmentation library for rust).
  - pron 1y ago
    Similar to Java: String.chars().count(), String.codePoints().count(), and, for historical reasons, String.getBytes(UTF-8).length
- bigstrat2003 1y ago
  > in the global international connected computing world it doesn’t fit at all. Most people aren't living in that world. If you're working at Amazon or some business that needs to interact with many countries around the globe, sure, you have to worry about text encoding quite a bit. But the majority of software is being written for a much narrower audience, probably for one single language in one single country. There is simply no reason for most programmers to obsess over text encoding the way so many people here like to.
  - rileymat2 1y ago
    Except, this is a response to emoji support, which does have encoding issues even if your user base is in the US and only speaks English. Additionally, it is easy to have issues with data that your users use from other sources via copy and paste.
  - raverbashing 1y ago
    This is naive at best Here's a better analogy, in the 70s "nobody planned" for names with 's in then. SQL injections, separators, "not in the alphabet", whatever. In the US. Where a lot of people with 's in their names live... Or double-barrelled names. It's a much simpler problem and still tripped a lot of people And then you have to support a user with a "funny name" or a business with "weird characters", or you expand your startup to Canada/Mexico and lo and behold...
    - ryandrake 1y ago
      Yea, I cringe when I hear the phrase "special characters." They're only special because you, the developer, decided to treat them as special, and that's almost surely going to come back to haunt you at some point in the form of a bug.
  - arp242 1y ago
    No one is "obsessing" over anything. Reality is there are very few cases where you can use a single 8-bit character set and not run in to problems sooner or later. Say your software is used only in Greece so you use ISO-8859-7 for Greek. That works fine, but now you want to talk to your customer Günther from Germany who has been living in Greece for the last five years, or Clément from France, or Seán from Ireland and oops, you can't. Even plain English text can't be represented with plain ASCII (although ISO-8859-1 goes a long way). There are some cases where just plain ASCII is okay, but there are quite few of them (and even those are somewhat controversial). The solution is to just use UTF-8 everywhere. Or maybe UTF-16 if you really have to.
sigmoid10 1y ago
I have wanted string length many times in production systems for language processing. And it is perfectly fine as long as whatever you are using is consistent. I rarely care how many bytes an emoji actually is unless I'm worried about extreme efficiency in storage or how many monospace characters it uses unless I do very specific UI things. This blog is more of a cautionary tale what can happen if you unconsciously mix standards e.g. by using one in the backend and another in the frontend. But this is not a problem of string lengths per se, they are just one instance where modern implementations are all over the place.
xg15 1y ago
It gets more complicated if you do substring operations. If I do s.charAt(x) or s.codePointAt(x) or s.substring(x, y), I'd like to know which values for x and y are valid and which aren't.
- mseepgood 1y ago
  The values for x and y should't come from your brain, though (with the exception of 0). They should come from previous index operations like s.indexOf(...) or s.search(regex), etc.
  - xg15 1y ago
    Indeed. Or s.length, whatever that represents.
- arcticbull 1y ago
  Substring operations (and more generally the universe of operations where there is more than one string involved) are a whole other kettle of fish. Unicode, being a byte code format more than what you think of as a logical 'string' format, has multiple ways of representing the same strings. If you take a substring of a(bc) and compare it to string (bc) are you looking for bitwise equivalence or logical equivalence? If the former it's a bit easier (you can just memcmp) but if the latter you have to perform a normalization to one of the canonical forms.
  - setr 1y ago
    I’m fairly positive the answer is trivially logical equivalence for pretty much any substring operation. I can’t imagine bitwise equivalence to ever be the “normal” use case, except to the implementer looking at it as a simpler/faster operation I feel like if you’re looking for bitwise equivalence or similar, you should have to cast to some kind of byte array and access the corresponding operations accordingly
    - arcticbull 1y ago
      Yep for a substring against its parent or other substrings of the same parent that’s definitely true, but I think this question generalizes because the case where you’re comparing strings solely within themselves is an optimization path for the more general. I’m just thinking out loud.
guappa 1y ago
What if you need to find 5 letter words to play wordle? Why do you care how many bytes they occupy or how large they are on screen?
- taneq 1y ago
  If you're playing at this level, you need to define: - letter - word - 5 :P
  - guappa 1y ago
    Eh in macedonian they have some letters that in russian are just 2 separate letters
    - CorrectHorseBat 1y ago
      In German you have the same, only within one language. ß can be written as ss if it isn't available in a font, and only in 2017 they added a capital version. So depending the font and the unicode version the number of letters can differ.
      - guappa 1y ago
        should "ß" == "ss" evaluate as true?
        birn559 1y ago
        I don't see why it should. I also believe parent is wrong as there are unambiguous rules about when to use ß or ss. Never thought of it but maybe there are rules that allow to visually present the code point for ß as ss? At least (from experience as a user) there seem to be a singular "ss" codepoint.
        guappa 1y ago
        well I don't speak german, I was asking
arcticbull 1y ago
Taking this one step further -- there's no such thing as the context-free length of a string. Strings should be thought of more like opaque blobs, and you should derive their length exclusively in the context in which you intend to use it. It's an API anti-pattern to have a context-free length property associated with a string because it implies something about the receiver that just isn't true for all relevant usages and leads you to make incorrect assumptions about the result. Refining your list, the things you usually want are: - Number of bytes in a given encoding when saving or transmitting (edit: or more generally, when serializing). - Number of code points when parsing. - Number of grapheme clusters for advancing the cursor back and forth when editing. - Bounding box in pixels or points for display with a given font. Context-free length is something we inherited from ASCII where almost all of these happened to be the same, but that's not the case anymore. Unicode is better thought of as compiled bytecode than something you can or should intuit anything about. It's like asking "what's the size of this JPEG." Answer is it depends, what are you trying to do?
- account42 1y ago
  > Number of code points when parsing. You shouldn't really ever care about the number of code points. If you do, you're probably doing something wrong.
  - deleted 1y ago
    [deleted]
  - josephg 1y ago
    It’s a bit of a niche use case, but I use the codepoint counts in CRDTs for collaborative text editing. Grapheme cluster counts can’t be used because they’re unstable across Unicode versions. Some algorithms use UTF8 byte offsets - but I think that’s a mistake because they make input validation much more complicated. Using byte offsets, there’s a whole lot of invalid states you can represent easily. Eg maybe insert “a” at position 0 is valid, but inserting at position 1 would be invalid because it might insert in the middle of a codepoint. Then inserting at position 2 is valid again. If you send me an operation which happened at some earlier point in time, I don’t necessarily have the text document you were inserting into handy. So figuring out if your insertion (and deletion!) positions are valid at all is a very complex and expensive operation. Codepoints are way easier. I can just accept any integer up to the length of the document at that point in time.
xelxebar 1y ago
> Number of monospaced font character blocks this string will take up on the screen Even this has to deal with the halfwidth/fullwidth split in CJK. Even worse, Devanagari has complex rendering rules that actually depend on font choices. AFAIU, the only globally meaningful category here is rendered bounding box, which is obviously font-dependent. But I agree with the general sentiment. What we really about how much space these text blobs take up, whether that be in a DB, in memory, or on the screen.
zwnow 1y ago
I actually want string length. Just give me the length of a word. My human brain wants a human way to think about problems. While programming I never think about bytes.
- dwb 1y ago
  The whole point is that string length doesn’t necessarily give you the “length” of a “word”, and both of those terms are not well enough defined.
- jibal 1y ago
  The point is that those terms are ambiguous ... and if you mean the length in grapheme clusters, it can be quite expensive to calculate it, and isn't the right number if you're dealing with strings as objects that are chunks of memory.
  - zwnow 1y ago
    [flagged]
    - dwb 1y ago
      You are in the wrong job if you don’t want to think about “nerd shit” while programming.
      - zwnow 1y ago
        Idk it pays my bills and I have success at work so I must do something right.
        dwb 1y ago
        Well I just hope I don’t have to use any of the software you build. What a shameful attitude.
        1y ago
thrdbndndn 1y ago
I see where you're coming from, but I disagree on some specifics, especially regarding bytes. Most people care about the length of a string in terms of the number of characters. Treating it as a proxy for the number of bytes has been incorrect ever since UTF-8 became the norm (basically forever), and if you're dealing with anything beyond ASCII (which you really should, since East Asian users alone number in the billions). Same goes to the "string width". Yes, Unicode scalar values can combine into a single glyph and cause discrepancies, as the article mentions, but that is a much rarer edge case than simply handling non-ASCII text.
- account42 1y ago
  It's not rare at all - multi-code point emojis are pretty standard these days. And before that the only thing the relative rarity did for you was that bugs with code working on UTF-8 bytes got fixed while bugs that assumed UTF-16 units or 32-bit code points represent a character were left to linger for much longer.
Semaphor 1y ago
FWIW, I frequently want the string length. Not for anything complicated, but our authors have ranges of characters they are supposed to stay in. Luckily no one uses emojis or weird unicode symbols, so in practice there’s no problem getting the right number by simply ignoring all the complexities.
- tomsmeding 1y ago
  It's not unlikely that what you would ideally use here is the number of grapheme clusters. What is the length of "ë"? Either 1 or 2 codepoints depending on the encoding (combining [1] or single codepoint [2]), and either 1 byte (Latin-1), 2 bytes (UTF-8 single-codepoint) or 3 bytes (UTF-8 combining). The metrics you care about are likely number of letters from a human perspective (1) or the number of bytes of storage (depends), possibly both. [1]: https://tomsmeding.com/unicode#U+65%20U+308 https://tomsmeding.com/unicode#U+65%20U+308 [2]: https://tomsmeding.com/unicode#U+EB https://tomsmeding.com/unicode#U+EB
bluecalm 1y ago
What about implementing text algorithms like prefix search or a suffix tree to mention the simplest ones? Don't you need a string length at various points there?
- account42 1y ago
  With UTF-8 you can implement them on top of bytes.
  - jlarocco 1y ago
    That's basically what a string data type is for.
bigstrat2003 1y ago
I have never wanted any of the things you said. I have, on the other hand, always wanted the string length. I'm not saying that we shouldn't have methods like what you state, we should! But your statement that people don't actually want string length is untrue because it's overly broad.
- zahlman 1y ago
  > I have, on the other hand, always wanted the string length. In an environment that supports advanced Unicode features, what exactly do you do with the string length?
  - PapaPalpatine 1y ago
    I don’t know about advanced Unicode features… but I use them all the time as a backend developer to validate data input. I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. This seems to have always been known as the length of the string. This thread sounds like a bunch of scientists trying to make a simple concept a lot harder to understand.
    - zahlman 1y ago
      > I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. > This seems to have always been known as the length of the string. Sure. And by this definition, the string discussed in TFA (that consists of a facepalm emoji with a skin tone set) objectively has 5 characters in it, and therefore a length of 5. And it has always had 5 characters in it, since it was first possible to create such a string. Similarly, "é" has one character in it, but "é" has two despite appearing visually identical. Furthermore, those two strings will not compare equal in any sane programming language without explicit normalization (unless HN's software has normalized them already). If you allow passwords or email addresses to contain things like this, then you have to reckon with that brute fact. None of this is new. These things have fundamentally been true since the introduction of Unicode in 1991.
capitainenemo 1y ago
FWIW, the cheap lazy way to get "number of bytes in DB" from JS, is unescape(encodeURIComponent("ə̀")).length
jlarocco 1y ago
It's definitely worth thinking about the real problem, but I wouldn't say it's never helpful. The underlying issue is unit conversion. "length" is a poor name because it's ambiguous. Replacing "length" with three functions - "lengthInBytes", "lengthInCharacters", and "lengthCombined" - would make it a lot easier to pick the right thing.
TZubiri 1y ago
How about for iterating every character in a string in order to find a specific character combination? I need (or the iterator needs) to know the length of the string and what the boundaries of each characters are.
perching_aix 1y ago
> Number of monospaced font character blocks this string will take up on the screen To predict the pixel width of a given text, right? One thing I ran into is that despite certain fonts being monospace, characters from different Unicode blocks would have unexpected lengths. Like I'd have expected half-width CJK letters to render to the same pixel dimensions as Latin letters do, but they don't. It's ever so slightly off. Same with full-width CJK letters vs two Latin letters. I'm not sure if this is due to some font fallback. I'd have expected e.g. VS Code to be able to be able to render Japanese and English monospace in an aligned way without any fallbacks. Maybe once I have energy again to waste on this I'll look into it deeper.
- oefrha 1y ago
  (Some but not all) terminal emulators are capable of rendering CJK perfectly aligned with Latin even when mixing fonts. Browsers are fundamentally incapable of that because aligning characters in different fonts wasn’t a goal at all. VS Code being a webview under the hood means it inherited this fundamental incapability.* Therefore, don’t hold your breath. * I'm talking about the DOM route, not <canvas> obviously. VS Code is powere by Monaco, which is DOM-based, not canvas-based. You can "Developer: Toggle Developer Tools" to see the DOM structure under the hood. ** I should further qualify my statement as browsers are fundamentally incapable of this if you use native text node rendering. I have built a perfectly monospace mixed CJK and Latin interface myself by wrapping each full width character in a separate span. Not exactly a performance-oriented solution. Also IIRC Safari doesn’t handle lengths in fractional pixels very well.
  - perching_aix 1y ago
    That's very informative, thanks! I guess it really wasn't a mirage or me messing up in the end then.
BobbyTables2 1y ago
Very true. Rust’s handling of strings was an eye opener for me. Seemed awkward but I eventually realized I rarely cared about number of characters. Even when dealing with substrings, I really only cared a means to describe “stuff” before/after not literal indices. Counting Unicode characters is actually a disservice.

baq 1y ago

ASCII is very convenient when it fits in the solution space (it’d better be, it was designed for a reason), but in the global international connected computing world it doesn’t fit at all. The problem is all the tutorials, especially low level ones, assume ASCII so 1) you can print something to the console and 2) to avoid mentioning that strings are hard so folks don’t get discouraged. Notably Rust did the correct thing by defining multiple slightly incompatible string types for different purposes in the standard library and regularly gets flak for it.

eru 1y ago
Python 3 deals with this reasonable sensibly, too, I think. They use UTF-8 by default, but allow you to specify other encodings.
- xigoi 1y ago
  I prefer languages where strings are simply sequences of bytes and you get to decide how to interpret them.
  - afiori 1y ago
    I would like an utf-8 optimized bag of bytes where arbitrary byte operations are possible but the buffer keeps track of whether is it valid utf-8 or not (for every edit of n bytes it should be enough to check about n+8 bytes to validate) then utf-8 then utf-8 encoding/decoding becomes a noop and utf-8 specific apis can check quickly is the string is malformed or not.
    - account42 1y ago
      But why care if it's malformed UTF-8? And specifically, what do you want to happen when you get a malformed UTF-8 string. Keep in mind that UTF-8 is self-synchronizing so even if you encode strings into a larger text-based format without verifying them it will still be possible to decode the document. As a user I normally want my programs to pass on the string without mangling it further. Some tool throwing fatal errors because some string I don't actually care about contains an invalid UTF-8 byte sequence is the last thing I want. With strings being an arbitrary bag of bytes many programs can support arbitrary encodings or at least arbitrary ASCII-supersets without any additional effort.
      - afiori 1y ago
        The main issue I can see is not garbage bytes in text but mixing of incompatible encoding eg splicing latin-1 bytes in a utf-8 string. My understanding of the current "always and only utf-8/unicode" zeitgeist is that is comes mostly from encoding issues among which the complexity of detecting encoding. I think that the current status quo is better than what came before, but I also think it could be improved.
  - account42 1y ago
    Yes, I always roll my eyes when people complain that C strings or C++'s std::string/string_view don't have Unicode support. They are bags of bytes with support for concatenation. Any other transformation isn't going to have a "correct" way to do it so you need to be aware of what you want anyway.
    - astrange 1y ago
      C strings are not bags of bytes because they can't contain 0x00.
  - bawolff 1y ago
    Me too. The languages that i really dont get are those that force valid utf-8 everywhere but dont enforce NFC. Which is most of them but seems like the worst of both worlds. Non normalized unicode is just as problematic as non validated unicode imo.
  - jibal 1y ago
    Python has byte arrays that allow for that, in addition to strings consisting of arrays of Unicode code points.
  - zahlman 1y ago
    Such languages do not have strings. Definitionally a string is a sequence of characters, and more than 256 characters exist. A byte sequence is just an encoding; if you are working with that encoding directly and have to do the interpretation yourself, you are not using a string. But if you do want a sequence of bytes for whatever reason, you can trivially obtain that in any version of Python.
    - capitainenemo 1y ago
      My experience personally with python3 (and repeated interactions with about a dozen python programmers, including core contributors) is that python3 does not let you trivially work with streams of bytes, esp if you need to do character set conversions, since a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. The last attempt was much larger, still failed, and they thought they could probably do it, but it would require far more code and was not worth their effort. I'll probably just use rust for that script if python2 ever gets dropped by my distro. Reminds me of https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/ https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journ...
      - zahlman 1y ago
        > a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. Show me.
        capitainenemo 1y ago
        Heh. It always starts this way... then they confidently send me something that breaks on testing it, then half a dozen more iterations, then "python2 is doing the wrong thing" or, "I could get this working but it isn't worth the effort" but sure, let's do this one more time. Could be they were all missing something obvious - wouldn't know, I avoid python personally, apart from when necessary like with LLM glue. https://pastebin.com/j4Lzb5q1 https://pastebin.com/j4Lzb5q1 This is a script created by someone on #nethack a long time ago. It works great with other things as well like old BBS games. It was intended to transparently rewrite single byte encodings to multibyte with an optional conversion array.
- ynik 1y ago
  Python 3 internally uses UTF-32. When exchanging data with the outside world, it uses the "default encoding" which it derives from various system settings. This usually ends up being UTF-8 on non-Windows systems, but on weird enough systems (and almost always on Windows), you can end up with a default encoding other than UTF-8. "UTF-8 mode" (https://peps.python.org/pep-0540/ https://peps.python.org/pep-0540/) fixes this but it's not yet enabled by default (this is planned for Python 3.15).
  - arcticbull 1y ago
    Apparently Python uses a variety of internal representations depending on the string itself. I looked it up because I saw UTF-32 and thought there's no way that's what they do -- it's pretty much always the wrong answer. It uses Latin-1 for ASCII strings, UCS-2 for strings that contain code points in the BMP and UCS-4 only for strings that contain code points outside the BMP. It would be pretty silly for them to explode all strings to 4-byte characters.
    - account42 1y ago
      It conceptually uses arrays of code points, which need up to 24 bits. Optimizing the storage to use smaller integers when possible is an implementation detail.
      - jibal 1y ago
        Python3 is specified to use arrays of 8, 16, or 32 bit units, depending on the largest code point in the string. As a result, all code points in all strings are O(1) indexable. The claim that "Python 3 internally uses UTF-32" is simply false.
      - zahlman 1y ago
        > code points, which need up to 24 bits They need at most 21 bits. The bits may only be available in multiples of 8, but the implementation also doesn't byte-pack them into 24-bit units, so that's moot.
    - 1y ago
account42 1y ago
> in the global international connected computing world it doesn’t fit at all. I disagree. Not all text is human prose. For example, there is nothing wrong with an programming language that only allows ASCII in the source code and many downsides to allowing non-ASCII characters outside string constants or comments.
- simonask 1y ago
  This is American imperialism at its worst. I'm serious. Lots of people around the world learn programming from sources in their native language, especially early in their career, or when software development is not their actual job. Enforcing ASCII is the same as enforcing English. How would you feel if all cooking recipes were written in French? If all music theory was in Italian? If all industrial specifications were in German? It's fine to have a dominant language in a field, but ASCII is a product of technical limitations that we no longer have. UTF-8 has been an absolute godsend for human civilization, despite its flaws.
  - account42 1y ago
    Actually, it would be great to have a lingua franca in every field that all participants can understand. Are you also going to complain that biologists and doctors are expected to learn some rudimentary Latin? English being dominant in computing is absolutely a strength and we gain nothing by trying to combat that. Having support for writing your code in other languages is not going to change that most libraries will use English and and most documentation will be in English and most people you can ask for help will understand English. If you want to participate and refuse to learn English you are only shooting yourself in the foot - and if you are going to learn English you may as well do it from the beginning. Also due to the dominance of English and ASCII in computing history, most languages already have ASCII-alternatives for their writing so even if you need to refer to non-English names you can do that using only ASCII.
    - simonask 1y ago
      Well, the problem is that what you are advocating is also that knowing Latin would be a prerequisite for studying medicine, which it isn't anywhere. That's the equivalent. Doctors learn a (very limited) Latin vocabulary as they study and work. You are severely underestimate how far you can get without any real command of the English language. I agree that you can't become really good without it, just like you can't do haute cuisine without some French, but the English language is a huge and unnecessary barrier of entry that you would put in front of everyone in the world who isn't submerged in the language from an early age. Imagine learning programming using only your high school Spanish. Good luck.
deleted 1y ago
[deleted]
flohofwoe 1y ago
ASCII is totally fine as encoding for the lower 127 UNICODE code points. If you need to go above those 127 code points, use a different encoding like UTF-8. Just never ever use Extended ASCII (8-bits with codepages).
craftkiller 1y ago
> Notably Rust did the correct thing In addition to separate string types, they have separate iterator types that let you explicitly get the value you want. So: String.len() == number of bytes String.bytes().count() == number of bytes String.chars().count() == number of unicode scalar values String.graphemes().count() == number of graphemes (requires unicode-segmentation which is not in the stdlib) String.lines().count() == number of lines Really my only complaint is I don't think String.len() should exist, it's too ambiguous. We should have to explicitly state what we want/mean via the iterators.
- westurner 1y ago
  String.graphemes().count() That's a real nice API. (Similarly, python has @ for matmul but there is not an implementation of matmul in stdlib. NumPy has a matmul implementation so that the `@` operator works.) ugrapheme and ucwidth are one way to get the graphene count from a string in Python. It's probably possible to get the grapheme cluster count from a string containing emoji characters with ICU?
  - dhosek 1y ago
    Any correctly designed grapheme cluster handles emoji characters. It’s part of the spec (says the guy who wrote a Unicode segmentation library for rust).
- pron 1y ago
  Similar to Java: String.chars().count(), String.codePoints().count(), and, for historical reasons, String.getBytes(UTF-8).length
bigstrat2003 1y ago
> in the global international connected computing world it doesn’t fit at all. Most people aren't living in that world. If you're working at Amazon or some business that needs to interact with many countries around the globe, sure, you have to worry about text encoding quite a bit. But the majority of software is being written for a much narrower audience, probably for one single language in one single country. There is simply no reason for most programmers to obsess over text encoding the way so many people here like to.
- rileymat2 1y ago
  Except, this is a response to emoji support, which does have encoding issues even if your user base is in the US and only speaks English. Additionally, it is easy to have issues with data that your users use from other sources via copy and paste.
- raverbashing 1y ago
  This is naive at best Here's a better analogy, in the 70s "nobody planned" for names with 's in then. SQL injections, separators, "not in the alphabet", whatever. In the US. Where a lot of people with 's in their names live... Or double-barrelled names. It's a much simpler problem and still tripped a lot of people And then you have to support a user with a "funny name" or a business with "weird characters", or you expand your startup to Canada/Mexico and lo and behold...
  - ryandrake 1y ago
    Yea, I cringe when I hear the phrase "special characters." They're only special because you, the developer, decided to treat them as special, and that's almost surely going to come back to haunt you at some point in the form of a bug.
- arp242 1y ago
  No one is "obsessing" over anything. Reality is there are very few cases where you can use a single 8-bit character set and not run in to problems sooner or later. Say your software is used only in Greece so you use ISO-8859-7 for Greek. That works fine, but now you want to talk to your customer Günther from Germany who has been living in Greece for the last five years, or Clément from France, or Seán from Ireland and oops, you can't. Even plain English text can't be represented with plain ASCII (although ISO-8859-1 goes a long way). There are some cases where just plain ASCII is okay, but there are quite few of them (and even those are somewhat controversial). The solution is to just use UTF-8 everywhere. Or maybe UTF-16 if you really have to.

sigmoid10 1y ago

I have wanted string length many times in production systems for language processing. And it is perfectly fine as long as whatever you are using is consistent. I rarely care how many bytes an emoji actually is unless I'm worried about extreme efficiency in storage or how many monospace characters it uses unless I do very specific UI things. This blog is more of a cautionary tale what can happen if you unconsciously mix standards e.g. by using one in the backend and another in the frontend. But this is not a problem of string lengths per se, they are just one instance where modern implementations are all over the place.

xg15 1y ago

It gets more complicated if you do substring operations. If I do s.charAt(x) or s.codePointAt(x) or s.substring(x, y), I'd like to know which values for x and y are valid and which aren't.

mseepgood 1y ago
The values for x and y should't come from your brain, though (with the exception of 0). They should come from previous index operations like s.indexOf(...) or s.search(regex), etc.
- xg15 1y ago
  Indeed. Or s.length, whatever that represents.
arcticbull 1y ago
Substring operations (and more generally the universe of operations where there is more than one string involved) are a whole other kettle of fish. Unicode, being a byte code format more than what you think of as a logical 'string' format, has multiple ways of representing the same strings. If you take a substring of a(bc) and compare it to string (bc) are you looking for bitwise equivalence or logical equivalence? If the former it's a bit easier (you can just memcmp) but if the latter you have to perform a normalization to one of the canonical forms.
- setr 1y ago
  I’m fairly positive the answer is trivially logical equivalence for pretty much any substring operation. I can’t imagine bitwise equivalence to ever be the “normal” use case, except to the implementer looking at it as a simpler/faster operation I feel like if you’re looking for bitwise equivalence or similar, you should have to cast to some kind of byte array and access the corresponding operations accordingly
  - arcticbull 1y ago
    Yep for a substring against its parent or other substrings of the same parent that’s definitely true, but I think this question generalizes because the case where you’re comparing strings solely within themselves is an optimization path for the more general. I’m just thinking out loud.

guappa 1y ago

What if you need to find 5 letter words to play wordle? Why do you care how many bytes they occupy or how large they are on screen?

taneq 1y ago
If you're playing at this level, you need to define: - letter - word - 5 :P
- guappa 1y ago
  Eh in macedonian they have some letters that in russian are just 2 separate letters
  - CorrectHorseBat 1y ago
    In German you have the same, only within one language. ß can be written as ss if it isn't available in a font, and only in 2017 they added a capital version. So depending the font and the unicode version the number of letters can differ.
    - guappa 1y ago
      should "ß" == "ss" evaluate as true?
      - birn559 1y ago
        I don't see why it should. I also believe parent is wrong as there are unambiguous rules about when to use ß or ss. Never thought of it but maybe there are rules that allow to visually present the code point for ß as ss? At least (from experience as a user) there seem to be a singular "ss" codepoint.
        guappa 1y ago
        well I don't speak german, I was asking

arcticbull 1y ago

Taking this one step further -- there's no such thing as the context-free length of a string. Strings should be thought of more like opaque blobs, and you should derive their length exclusively in the context in which you intend to use it. It's an API anti-pattern to have a context-free length property associated with a string because it implies something about the receiver that just isn't true for all relevant usages and leads you to make incorrect assumptions about the result. Refining your list, the things you usually want are: - Number of bytes in a given encoding when saving or transmitting (edit: or more generally, when serializing). - Number of code points when parsing. - Number of grapheme clusters for advancing the cursor back and forth when editing. - Bounding box in pixels or points for display with a given font. Context-free length is something we inherited from ASCII where almost all of these happened to be the same, but that's not the case anymore. Unicode is better thought of as compiled bytecode than something you can or should intuit anything about. It's like asking "what's the size of this JPEG." Answer is it depends, what are you trying to do?

account42 1y ago
> Number of code points when parsing. You shouldn't really ever care about the number of code points. If you do, you're probably doing something wrong.
- deleted 1y ago
  [deleted]
- josephg 1y ago
  It’s a bit of a niche use case, but I use the codepoint counts in CRDTs for collaborative text editing. Grapheme cluster counts can’t be used because they’re unstable across Unicode versions. Some algorithms use UTF8 byte offsets - but I think that’s a mistake because they make input validation much more complicated. Using byte offsets, there’s a whole lot of invalid states you can represent easily. Eg maybe insert “a” at position 0 is valid, but inserting at position 1 would be invalid because it might insert in the middle of a codepoint. Then inserting at position 2 is valid again. If you send me an operation which happened at some earlier point in time, I don’t necessarily have the text document you were inserting into handy. So figuring out if your insertion (and deletion!) positions are valid at all is a very complex and expensive operation. Codepoints are way easier. I can just accept any integer up to the length of the document at that point in time.

xelxebar 1y ago

> Number of monospaced font character blocks this string will take up on the screen Even this has to deal with the halfwidth/fullwidth split in CJK. Even worse, Devanagari has complex rendering rules that actually depend on font choices. AFAIU, the only globally meaningful category here is rendered bounding box, which is obviously font-dependent. But I agree with the general sentiment. What we really about how much space these text blobs take up, whether that be in a DB, in memory, or on the screen.

zwnow 1y ago

I actually want string length. Just give me the length of a word. My human brain wants a human way to think about problems. While programming I never think about bytes.

dwb 1y ago
The whole point is that string length doesn’t necessarily give you the “length” of a “word”, and both of those terms are not well enough defined.
jibal 1y ago
The point is that those terms are ambiguous ... and if you mean the length in grapheme clusters, it can be quite expensive to calculate it, and isn't the right number if you're dealing with strings as objects that are chunks of memory.
- zwnow 1y ago
  [flagged]
  - dwb 1y ago
    You are in the wrong job if you don’t want to think about “nerd shit” while programming.
    - zwnow 1y ago
      Idk it pays my bills and I have success at work so I must do something right.
      - dwb 1y ago
        Well I just hope I don’t have to use any of the software you build. What a shameful attitude.
        1y ago

thrdbndndn 1y ago

I see where you're coming from, but I disagree on some specifics, especially regarding bytes. Most people care about the length of a string in terms of the number of characters. Treating it as a proxy for the number of bytes has been incorrect ever since UTF-8 became the norm (basically forever), and if you're dealing with anything beyond ASCII (which you really should, since East Asian users alone number in the billions). Same goes to the "string width". Yes, Unicode scalar values can combine into a single glyph and cause discrepancies, as the article mentions, but that is a much rarer edge case than simply handling non-ASCII text.

account42 1y ago
It's not rare at all - multi-code point emojis are pretty standard these days. And before that the only thing the relative rarity did for you was that bugs with code working on UTF-8 bytes got fixed while bugs that assumed UTF-16 units or 32-bit code points represent a character were left to linger for much longer.

Semaphor 1y ago

FWIW, I frequently want the string length. Not for anything complicated, but our authors have ranges of characters they are supposed to stay in. Luckily no one uses emojis or weird unicode symbols, so in practice there’s no problem getting the right number by simply ignoring all the complexities.

tomsmeding 1y ago
It's not unlikely that what you would ideally use here is the number of grapheme clusters. What is the length of "ë"? Either 1 or 2 codepoints depending on the encoding (combining [1] or single codepoint [2]), and either 1 byte (Latin-1), 2 bytes (UTF-8 single-codepoint) or 3 bytes (UTF-8 combining). The metrics you care about are likely number of letters from a human perspective (1) or the number of bytes of storage (depends), possibly both. [1]: https://tomsmeding.com/unicode#U+65%20U+308 https://tomsmeding.com/unicode#U+65%20U+308 [2]: https://tomsmeding.com/unicode#U+EB https://tomsmeding.com/unicode#U+EB

bluecalm 1y ago

What about implementing text algorithms like prefix search or a suffix tree to mention the simplest ones? Don't you need a string length at various points there?

account42 1y ago
With UTF-8 you can implement them on top of bytes.
- jlarocco 1y ago
  That's basically what a string data type is for.

bigstrat2003 1y ago

I have never wanted any of the things you said. I have, on the other hand, always wanted the string length. I'm not saying that we shouldn't have methods like what you state, we should! But your statement that people don't actually want string length is untrue because it's overly broad.

zahlman 1y ago
> I have, on the other hand, always wanted the string length. In an environment that supports advanced Unicode features, what exactly do you do with the string length?
- PapaPalpatine 1y ago
  I don’t know about advanced Unicode features… but I use them all the time as a backend developer to validate data input. I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. This seems to have always been known as the length of the string. This thread sounds like a bunch of scientists trying to make a simple concept a lot harder to understand.
  - zahlman 1y ago
    > I want to make sure that the password is between a given number of characters. Same with phone numbers, email addresses, etc. > This seems to have always been known as the length of the string. Sure. And by this definition, the string discussed in TFA (that consists of a facepalm emoji with a skin tone set) objectively has 5 characters in it, and therefore a length of 5. And it has always had 5 characters in it, since it was first possible to create such a string. Similarly, "é" has one character in it, but "é" has two despite appearing visually identical. Furthermore, those two strings will not compare equal in any sane programming language without explicit normalization (unless HN's software has normalized them already). If you allow passwords or email addresses to contain things like this, then you have to reckon with that brute fact. None of this is new. These things have fundamentally been true since the introduction of Unicode in 1991.

capitainenemo 1y ago

FWIW, the cheap lazy way to get "number of bytes in DB" from JS, is unescape(encodeURIComponent("ə̀")).length

jlarocco 1y ago

It's definitely worth thinking about the real problem, but I wouldn't say it's never helpful. The underlying issue is unit conversion. "length" is a poor name because it's ambiguous. Replacing "length" with three functions - "lengthInBytes", "lengthInCharacters", and "lengthCombined" - would make it a lot easier to pick the right thing.

TZubiri 1y ago

How about for iterating every character in a string in order to find a specific character combination? I need (or the iterator needs) to know the length of the string and what the boundaries of each characters are.

perching_aix 1y ago

> Number of monospaced font character blocks this string will take up on the screen To predict the pixel width of a given text, right? One thing I ran into is that despite certain fonts being monospace, characters from different Unicode blocks would have unexpected lengths. Like I'd have expected half-width CJK letters to render to the same pixel dimensions as Latin letters do, but they don't. It's ever so slightly off. Same with full-width CJK letters vs two Latin letters. I'm not sure if this is due to some font fallback. I'd have expected e.g. VS Code to be able to be able to render Japanese and English monospace in an aligned way without any fallbacks. Maybe once I have energy again to waste on this I'll look into it deeper.

oefrha 1y ago
(Some but not all) terminal emulators are capable of rendering CJK perfectly aligned with Latin even when mixing fonts. Browsers are fundamentally incapable of that because aligning characters in different fonts wasn’t a goal at all. VS Code being a webview under the hood means it inherited this fundamental incapability.* Therefore, don’t hold your breath. * I'm talking about the DOM route, not <canvas> obviously. VS Code is powere by Monaco, which is DOM-based, not canvas-based. You can "Developer: Toggle Developer Tools" to see the DOM structure under the hood. ** I should further qualify my statement as browsers are fundamentally incapable of this if you use native text node rendering. I have built a perfectly monospace mixed CJK and Latin interface myself by wrapping each full width character in a separate span. Not exactly a performance-oriented solution. Also IIRC Safari doesn’t handle lengths in fractional pixels very well.
- perching_aix 1y ago
  That's very informative, thanks! I guess it really wasn't a mirage or me messing up in the end then.

BobbyTables2 1y ago

Very true. Rust’s handling of strings was an eye opener for me. Seemed awkward but I eventually realized I rarely cared about number of characters. Even when dealing with substrings, I really only cared a means to describe “stuff” before/after not literal indices. Counting Unicode characters is actually a disservice.