3 ms·
My gripe with new languages like this is that strings are always UTF-8 and that makes it needlessly difficult to parse HTTP headers correctly. This has led to v
by tosti 2mo ago
My gripe with new languages like this is that strings are always UTF-8 and that makes it needlessly difficult to parse HTTP headers correctly. This has led to vulnerabilities in the past.
- bflesch 2mo agoI agree. At this point scripting languages should have native support for string subtypes with custom charsets (ascii/utf16) or just regexp-defined or even ENUM()'erated values. IIRC historically on Windows, a string was UTF16, on unix it was ASCII; nowadays everywhere it's UTF8 without a way to specifically limit what goes into a string. For example UTF8 opens the door to homoglyph attacks and various other things (RTL, spaces), and a program should be able to force a string to be ASCII-only so that these classes of problems are ruled out.
- tosti 2mo agoThe NT kernel predates UTF8 by at least 3 years. It originally used UCS2, which covers the basic multilingual plane of the first unicode standard.
- Georgelemental 2mo agoHomoglyph attacks and RTL work just as well in UTF-16. UTF-8 and UTF-16 are different character encodings for the same character set (Unicode). That being said, Windows permits unpaired surrogates in its "UTF-16" strings, even though that is not actually valid UTF-16. Similarly, many Linux APIs accept arbitrary bytes, not just valid UTF-8.
- bflesch 2mo agoOf course. I was trying to say that it would be nice to limit certain variables to an ASCII charset, or ASCII+umlaut+accents without any of the 10k other UTF8 shenanigans.
- josephg 2mo agoI disagree. UTF-8 is the right default. Pure ASCII strings are rarely needed in modern software. Unless you know that you need ASCII, your strings should support unicode.
- tosti 2mo agoInternet Messages have been ASCII since Internet Messages.
- jcranmer 2mo agoMost internet protocols these days have shifted to UTF-8.
- tosti 2mo agoPerhaps so, and that's a good development imho. However, the most notorious of all, HTTP, has not afaik. Different encodings have been suggested including UTF-8. I'm not entirely sure why those proposals weren't implemented, but the dot-com bubble at the time might've had something to do with it.
- josephg 2mo agoTrue, but humans have been speaking non English languages since humans.
- tosti 2mo agoThe first networked computers predate unicode by several decades. Back then, the word length could be 8 but not neccesarily. All sorts of different encodings existed and a given common standard wasn't yet agreed upon It's not like internet standards don't know there are other languages, it's just that they documented how things were done at the time. Some legacy has remained ever since.
- josephg 2mo agoSure, but most code doesn't interact with raw packets. Legacy, ascii-only internet standards probably make up much less than 1% of new lines of code written today. Programming languages should support this use case. But not at the expense of the other 99% of software. Unicode-aware strings are the right choice for 99% of code. The last 1% should be a special case.