3 ms·
Is it really garbage? If we want to be true to UNIX's (questionable) "Write programs to handle text streams, because that is a universal interface" ethos, our d
by geofft 11y ago
Is it really garbage? If we want to be true to UNIX's (questionable) "Write programs to handle text streams, because that is a universal interface" ethos, our definition of "text" has to admit all possible byte strings to be "universal". And the so-called C locale historically did.
- masklinn 11y ago> Is it really garbage? Invalid UTF-8 when valid UTF-8 was expected? Yes. > "Write programs to handle text streams, because that is a universal interface" ethos, our definition of "text" has to admit all possible byte strings to be "universal" Random bytes are not text. The Unix ethos is "communicate via arbitrary binary streams" but programs which only understand text understand text, not random-bytes-which-are-not-text. It seems sensible for programs to have more restrictions on their input than the general-purpose communication protocol does: would you expect jq to try and process input which is not JSON in any way, shape or form despite being it being billed a JSON processor? Because that's not what it's going to do, at least not by default.
- kragen 11y agoAlthough it was pretty common up to the mid-1990s to run into 8-bit-cleanness problems and arbitrary buffer sizes, IMHO, the Unix ethos is not for `wc`, `tr`, `dd`, `sort`, `uniq`, `read`, `diff`, `patch`, or `split` to crash with certain input data, to silently corrupt that data, or to spew warning messages about its contents. They are building blocks for your programs; it is not their business to impose unnecessary expectations on your data. They can and should correctly handle arbitrary data. When they don't do that, they limit the programs you can write with them, and with no compensating increase in anything other virtue.