6 ms·

One way to think about text, is as the only data we use that shrinks over the years. Every other type of data continues to grow. The way this can be stretched

One way to think about text, is as the only data we use that shrinks over the years. Every other type of data continues to grow. The way this can be stretched to make sense is by thinking for example how much these have grown over 20 years. Media (music, video, images) grows staggeringly because of 200p > 4k, 128bit mp3 to flac or whatever and there's more of it all that's gone digital. Code size continues to grow quite significantly even. But text? Even the massive wikipedia's size, only a fraction is text content. Then as memory capacity also grows it's really a back and forth battle between computing resources and all other data, except text. Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps?

pg_is_a_butt 7y ago
> Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps? Parsers would be much slower.
mikeash 7y ago
String-keyed hash maps are really common, and the keys tend to be ASCII. This would increase their size by a factor of 4, resulting in substantially increased memory usage and decreased performance.
- WhitneyLand 7y ago
  Certainly, but I’m not sure myself, can you think of any scenarios that are common where the difference would be noticeable or significant?
  - mikeash 7y ago
    Boot any modern OS and you’ll have megabytes if not tens of megabytes of these strings, and spend a small but significant amount of CPU time looking them up. This may not matter on your Ryzen desktop with 97 cores and 318GB of RAM, but it’s substantial on a smartphone.
    - WhitneyLand 7y ago
      Well, that seems to be an unfounded assumption that I'm considering only powerful machines, especially since possible challenges on small devices were mentioned in the OP. In fact, I done a fair amount of ARM coding so I understand what you are trying to say. Nevertheless, your comment seems to point out smartphones may be less powerful, but sheds no light on what the delta might be for boot performance or how you can be confident it's say, more than a few hundred milliseconds. Maybe it is, I haven't isolated and investigated that single difference enough to form a reasonable guess. Have you?
  - ben509 7y ago
    Any dynamic language, as all the symbols are strings.
simias 7y ago
Conversely as CPUs get more and more powerful the overhead of a non-fixed-width encoding also becomes negligible unless you're processing a huge amount of text. And if you process enough text that it becomes a problem then maybe size would be an issue too. Furthermore 4-byte chars also have the endianess problem of UTF-16. I think the rule of thumb of "use UTF-8 unless you really have a good reason not to" still holds true today.
- WhitneyLand 7y ago
  I think that’s a valid and interesting point that the classic time versus space computer science trade-off needs to be considered in this case for large workloads. One thing that seems different is that you also hear concern from developers about coding complexity and the desire to always have things to be as easy to reason about as possible. Yes these are things a library should just handle for you but it sometimes useful to know what’s happening fundamentally or can sometimes allow for less surprises. Nevertheless, if one of the cases you mentioned became significant I’m not sure that not many would say coding complexity should take precedence over your considerations.
  - simias 7y ago
    I agree with you there, I think in the vast majority of applications these days when it comes to text processing simplicity and robustness beats other concerns. It's unclear if UTF-32 beats UTF-8 there however. UTF-8's encoding is a bit trickier to deal with of course, but it's really not that bad. On top of that you have a few advantages such as not having to worry about endianess as mentioned previously but also not having to worry too much about alignment or corrupted input (you can re-synchronize UTF-8 at every codepoint since the leading byte has a unique bit pattern). And of course ASCII/C-string compatibility means that it's unlikely to be mangled by non-unicode-aware applications. In any case the complexity probably won't be in the encoding, be it UTF-8, 16, 32 or anything else. UTF-32 might be constand-width codepoint but in general you don't really care about codepoints since they can be combined in various ways to create characters or things like country flags or complex characters with diacritics.
mntmoss 7y ago
It's not just that the raw encoding that becomes easier to relax with higher system resources - it can also be generically compressed using those same resources. I run my Windows machines with disk compression enabled, and that tames most "bloated" files and lets me fit a lot more into a 256Gb SSD. However, there is some appeal in defining the format to a fixed low fidelity, too, since it shapes the authoring process around conveying the content through the limitations - "the medium is the message." But they're more likely to be the source format than thr transmission format, in the same way that old video games designed for 8kB of working memory now end up with terabytes of video footage streamed to online audiences.
Synaesthesia 7y ago
Nitpick: it’s not shrinking, probably growing faster than ever, it’s just growing slower than other forms of media.
- WhitneyLand 7y ago
  Fair enough. Agreed, it would have been expressed more clearly as a some sort of relative perception.

pg_is_a_butt 7y ago

> Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps? Parsers would be much slower.

mikeash 7y ago

String-keyed hash maps are really common, and the keys tend to be ASCII. This would increase their size by a factor of 4, resulting in substantially increased memory usage and decreased performance.

WhitneyLand 7y ago
Certainly, but I’m not sure myself, can you think of any scenarios that are common where the difference would be noticeable or significant?
- mikeash 7y ago
  Boot any modern OS and you’ll have megabytes if not tens of megabytes of these strings, and spend a small but significant amount of CPU time looking them up. This may not matter on your Ryzen desktop with 97 cores and 318GB of RAM, but it’s substantial on a smartphone.
  - WhitneyLand 7y ago
    Well, that seems to be an unfounded assumption that I'm considering only powerful machines, especially since possible challenges on small devices were mentioned in the OP. In fact, I done a fair amount of ARM coding so I understand what you are trying to say. Nevertheless, your comment seems to point out smartphones may be less powerful, but sheds no light on what the delta might be for boot performance or how you can be confident it's say, more than a few hundred milliseconds. Maybe it is, I haven't isolated and investigated that single difference enough to form a reasonable guess. Have you?
- ben509 7y ago
  Any dynamic language, as all the symbols are strings.

simias 7y ago

Conversely as CPUs get more and more powerful the overhead of a non-fixed-width encoding also becomes negligible unless you're processing a huge amount of text. And if you process enough text that it becomes a problem then maybe size would be an issue too. Furthermore 4-byte chars also have the endianess problem of UTF-16. I think the rule of thumb of "use UTF-8 unless you really have a good reason not to" still holds true today.

WhitneyLand 7y ago
I think that’s a valid and interesting point that the classic time versus space computer science trade-off needs to be considered in this case for large workloads. One thing that seems different is that you also hear concern from developers about coding complexity and the desire to always have things to be as easy to reason about as possible. Yes these are things a library should just handle for you but it sometimes useful to know what’s happening fundamentally or can sometimes allow for less surprises. Nevertheless, if one of the cases you mentioned became significant I’m not sure that not many would say coding complexity should take precedence over your considerations.
- simias 7y ago
  I agree with you there, I think in the vast majority of applications these days when it comes to text processing simplicity and robustness beats other concerns. It's unclear if UTF-32 beats UTF-8 there however. UTF-8's encoding is a bit trickier to deal with of course, but it's really not that bad. On top of that you have a few advantages such as not having to worry about endianess as mentioned previously but also not having to worry too much about alignment or corrupted input (you can re-synchronize UTF-8 at every codepoint since the leading byte has a unique bit pattern). And of course ASCII/C-string compatibility means that it's unlikely to be mangled by non-unicode-aware applications. In any case the complexity probably won't be in the encoding, be it UTF-8, 16, 32 or anything else. UTF-32 might be constand-width codepoint but in general you don't really care about codepoints since they can be combined in various ways to create characters or things like country flags or complex characters with diacritics.

mntmoss 7y ago

It's not just that the raw encoding that becomes easier to relax with higher system resources - it can also be generically compressed using those same resources. I run my Windows machines with disk compression enabled, and that tames most "bloated" files and lets me fit a lot more into a 256Gb SSD. However, there is some appeal in defining the format to a fixed low fidelity, too, since it shapes the authoring process around conveying the content through the limitations - "the medium is the message." But they're more likely to be the source format than thr transmission format, in the same way that old video games designed for 8kB of working memory now end up with terabytes of video footage streamed to online audiences.

Synaesthesia 7y ago

Nitpick: it’s not shrinking, probably growing faster than ever, it’s just growing slower than other forms of media.

WhitneyLand 7y ago
Fair enough. Agreed, it would have been expressed more clearly as a some sort of relative perception.