3 ms·
I am surprised more programming language research hasn't focused on problems like this. Perl had taint mode back in the day (presumably it still exists), but i
by jrockway 7y ago
I am surprised more programming language research hasn't focused on problems like this. Perl had taint mode back in the day (presumably it still exists), but it didn't quite do enough to really be helpful. I am glad to see this idea resurfacing because I think it can solve a lot of problems, not just security-related.
A long time ago, I remember people having an insane amount of trouble with character handling; when you read binary data from a TCP socket or UNIX file, you're reading bytes, not characters. But many people would treat the bytes like characters, causing all sorts of trouble. My favorite was the double-encoding, where you read UTF-8 encoded characters as bytes, treat the bytes as Latin-1, then encode the Latin-1 characters as UTF-8. This was a perl quirk because Latin-1 was the default, but the same bug happens in other languages. Anyway, a good tainting system could prevent this sort of bug. The language can say "hey, this is a TCP socket, you can't treat those bytes as characters!" But it doesn't. And the bug occurs again and again.
(The corner cases that people don't think about are the real problems. What charset are those bytes in your URL? What about filenames? The answer is: it's often undefined. So rather than hope for the best, a compiler error would be ideal.)
The state of the art, as far as I can tell, is to just treat everything as UTF-8 these days. Since everyone seems to love UTF-8, it just works. Maybe that was the real solution. But I know there are a lot of Japanese-speakers with names that can't be encoded as UTF-8. I wonder what they're doing about that.
- deleted 7y ago[deleted]
- Sohcahtoa82 7y agoI feel like Python 3 has largely resolved the bytes vs characters problem. Byte arrays and strings are different classes in Py3. File and socket I/O deals only in byte arrays which are treated more like lists than strings. To perform a lot of string-like operations, you have to explicitly decode the byte array into a string. If you get it wrong, you'll likely get either a TypeError.
- jrockway 7y agoThis is good, but I'd prefer the error to occur at compile time. Consider the case where you do something like creating an error message because you can't open a file, and want to include the filename. The filename is bytes, the error message is a string, so instead of being able to print the error message, you instead throw an undebuggable exception. Dunno if that in particular is a problem in Python or not... but it is the sort of thing to watch out for.
- Sohcahtoa82 7y agoIn Python, filenames can be either bytes or string. Also, I made a mistake in my previous post. File I/O can deal with bytes OR strings depending on the parameters sent to the `open()` function. file_handle = open('somefile.txt') This opens the file in text mode (so `file_handle.read()` returns strings) using a platform-dependent encoding. On Windows, this will probably be CP-1252. On Linux, UTF-8. Of course, you can always explicitly choose the encoding: file_handle = open('somefile.txt', encoding='utf8') If you know you want to only deal with bytes, you can explicitly set that: file_handle = open('somefile.ext', mode='rb') I agree that it'd be nice to be able to see bytes vs strings errors at compile time, but this is difficult or even impossible with a duck-typed interpreted language like Python. IDEs and linters can make a good attempt at it, but they're not perfect.
- Eli_P 7y agoThere's another issue with Unicode: the same string can be encoded in many ways, which affects at least collation. On server side I deal with it with normalization[1]. I'm sure I can do it on the client with js but that just doesn't feel right. While rendering unsafe content still can be useful, I think Google better off added trusted-types as a tag attribute for WebComponents, like per-component CSP, and not into CSP header. [1] https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize https://docs.python.org/3/library/unicodedata.html#unicodeda...