12 ms·

I was looking for the catch. Here it is: "It's really simple: Know what encoding a certain piece of text, that is, a certain byte sequence, is in, then interpr

I was looking for the catch. Here it is: "It's really simple: Know what encoding a certain piece of text, that is, a certain byte sequence, is in, then interpret it with that encoding." That's like "knowing" the truth. How? I have received some very interesting files that made Python yack unicode errors, again and again. Why? Not only did I not "know" what encoding it was in -- the encodings changed at different points in the stream of bytes. I call this "slamming bytes together" because somewhere along the line, someone's program did exactly that. Everything is simple -- until it isn't.

banthar 6y ago
There is nothing you can do with text file with unknown encoding but treat it as an array of bytes. If you start guessing the encoding, at best it won't work in some cases, at worst you are introducing security vulnerabilities. You can try, but there is just no way to do it right. http://michaelthelin.se/security/2014/06/08/web-security-cross-site-scripting-attacks-using-utf-7.html http://michaelthelin.se/security/2014/06/08/web-security-cro...
- tialaramex 6y ago
  Generally you can safely treat text in an unknown encoding as UTF-8. Since you're expecting potential failures but want to press on anyway instead of causing an exception/ error you treat invalid sequences as U+FFFD the Replacement Character as you would in a language or API with no exception reporting mechanism. There are lots of pleasing aspects to this choice. It's ASCII compatible of course, so anything that was actually ASCII is still ASCII, anything that was almost ASCII is just ASCII with U+FFFD where it deviated. The replacement character resolutely isn't any of the specific things, nor any of the generic classes of thing you might be expected to treat differently for security reasons. It isn't a number, or a letter (of either "case"), it isn't white space, and it certainly isn't any of the separators, escapes or quote markers like ? or \ or + or . or _ or... ... yet it is still inside the BMP so it won't trigger weird (perhaps less well tested) behaviour for other planes. It's self-synchronising. If something goes wrong somehow, in a few bytes if there is UTF-8 or an ASCII-compatible encoding the decoder will synchronise properly, you never end up "out of phase" as can happen for some encodings. Most usefully, whatever you're now butted up against works with UTF-8 now. Maybe some day that'll get formally documented, maybe it won't. As the years drag on the chance of specifying _anything else_ shrink more, and the de facto popularity of UTF-8 means even if it's never formalised anywhere everybody will just assume UTF-8 anyway and you haven't to lift a finger.
  - naniwaduni 6y ago
    And now you're eating data.
    - tialaramex 6y ago
      I mean, are you though? Was it "data" ? The supposed data doesn't have metadata to tell you what it means (which encoding / character set was used), so, did it actually mean anything? Smashing the bits that don't mean anything to U+FFFD leaves humans with the unmistakable evidence that something was lost here. It's not like U+FFFD doesn't scream "Hey stuff went wrong here" - it's an inverse question mark on a diamond, short of an animated GIF that says "Uh-oh" with an anvil dropping onto a cartoon character's head we can't do much more. If you're sure it's supposed to be ISO-8859-1 then sure, treating it was UTF-8 eats data, likewise if it was supposed to be KOI-8 or something. But you don't know, so, if "Give up and demand a human fix things" isn't a sensible option, which it often won't be, this is the best we can do.
      - naniwaduni 6y ago
        > Was it "data" ? Yes. The fact that you couldn't interpret it doesn't mean that the consumer of your output couldn't have if you'd passed it through without going out of your way to destroy it. There is a large, enormous class of software that cares detects specific sequences meaningful to it that exist in ASCII, and copies other parts of it input directly into its output without really caring to modify it. If you don't intentionally destroy your input, this will silently Just Work with many encodings in actual use.
        tialaramex 6y ago
        But this involves a sleight of hand where, at first, you deny knowing the encoding so as to require we can't decode it, and then, you declare you did know the encoding so as to declare the results "destroyed" because now you can't decode them. Just tell the text processing tool the encoding. Or, don't use text. If you resent having to pick an encoding, it still works - just the encoding is UTF-8 because duh, of course it is.
- at_a_remove 6y ago
  Oh, that's what I did of course. Some substitution, some Pokemon Exception Handling. At the end of the day, it was analysis, not a random file, so I wasn't worried about security. What I am pointing out is that "know" is just doing a lot of magical work in that sentence.
BiteCode_dev 6y ago
In a sense, it is simple: simple to understand. Not simple to solve. Like to win a race Usain Bolt, it's simple, run faster! Fortunately, Python is well equipped for that. If you open a file with Python that you know it might contains mixed encoded text, you can use try/except to inform the user or open in binary mode, and just store the binary. But my favorite way of doing it is: open('file', error=strategy) Strategy can be: - "ignore": undecodable text is skipped - "replace": undecodable text is replace with "?" - "surrogateescape" (you need to use utf8): undecodable text is decoded to a special representation which makes no human sense, but can en rencoded back to it's original value. It's kinda ironic because people bashed Python for separating bytes/text, forcing them to deal with encoding correctly in Python 3. After all, this problem of "slamming bytes together" comes from languages that treat text as a bytes array, allowing this stupid mistake.
- at_a_remove 6y ago
  Yeah, that was more or less my approach to analysis for these files. Essentially it was an export that was quasi-aware of Unicode and dumped out certain fields (of course they had to be variable length) in their original encodings, whatever they were. I got more than a few of these. Unicode is great, as long as everyone upstream follows all of the rules and nothing goes wrong.

banthar 6y ago

There is nothing you can do with text file with unknown encoding but treat it as an array of bytes. If you start guessing the encoding, at best it won't work in some cases, at worst you are introducing security vulnerabilities. You can try, but there is just no way to do it right. http://michaelthelin.se/security/2014/06/08/web-security-cross-site-scripting-attacks-using-utf-7.html http://michaelthelin.se/security/2014/06/08/web-security-cro...

tialaramex 6y ago
Generally you can safely treat text in an unknown encoding as UTF-8. Since you're expecting potential failures but want to press on anyway instead of causing an exception/ error you treat invalid sequences as U+FFFD the Replacement Character as you would in a language or API with no exception reporting mechanism. There are lots of pleasing aspects to this choice. It's ASCII compatible of course, so anything that was actually ASCII is still ASCII, anything that was almost ASCII is just ASCII with U+FFFD where it deviated. The replacement character resolutely isn't any of the specific things, nor any of the generic classes of thing you might be expected to treat differently for security reasons. It isn't a number, or a letter (of either "case"), it isn't white space, and it certainly isn't any of the separators, escapes or quote markers like ? or \ or + or . or _ or... ... yet it is still inside the BMP so it won't trigger weird (perhaps less well tested) behaviour for other planes. It's self-synchronising. If something goes wrong somehow, in a few bytes if there is UTF-8 or an ASCII-compatible encoding the decoder will synchronise properly, you never end up "out of phase" as can happen for some encodings. Most usefully, whatever you're now butted up against works with UTF-8 now. Maybe some day that'll get formally documented, maybe it won't. As the years drag on the chance of specifying _anything else_ shrink more, and the de facto popularity of UTF-8 means even if it's never formalised anywhere everybody will just assume UTF-8 anyway and you haven't to lift a finger.
- naniwaduni 6y ago
  And now you're eating data.
  - tialaramex 6y ago
    I mean, are you though? Was it "data" ? The supposed data doesn't have metadata to tell you what it means (which encoding / character set was used), so, did it actually mean anything? Smashing the bits that don't mean anything to U+FFFD leaves humans with the unmistakable evidence that something was lost here. It's not like U+FFFD doesn't scream "Hey stuff went wrong here" - it's an inverse question mark on a diamond, short of an animated GIF that says "Uh-oh" with an anvil dropping onto a cartoon character's head we can't do much more. If you're sure it's supposed to be ISO-8859-1 then sure, treating it was UTF-8 eats data, likewise if it was supposed to be KOI-8 or something. But you don't know, so, if "Give up and demand a human fix things" isn't a sensible option, which it often won't be, this is the best we can do.
    - naniwaduni 6y ago
      > Was it "data" ? Yes. The fact that you couldn't interpret it doesn't mean that the consumer of your output couldn't have if you'd passed it through without going out of your way to destroy it. There is a large, enormous class of software that cares detects specific sequences meaningful to it that exist in ASCII, and copies other parts of it input directly into its output without really caring to modify it. If you don't intentionally destroy your input, this will silently Just Work with many encodings in actual use.
      - tialaramex 6y ago
        But this involves a sleight of hand where, at first, you deny knowing the encoding so as to require we can't decode it, and then, you declare you did know the encoding so as to declare the results "destroyed" because now you can't decode them. Just tell the text processing tool the encoding. Or, don't use text. If you resent having to pick an encoding, it still works - just the encoding is UTF-8 because duh, of course it is.
at_a_remove 6y ago
Oh, that's what I did of course. Some substitution, some Pokemon Exception Handling. At the end of the day, it was analysis, not a random file, so I wasn't worried about security. What I am pointing out is that "know" is just doing a lot of magical work in that sentence.

BiteCode_dev 6y ago

In a sense, it is simple: simple to understand. Not simple to solve. Like to win a race Usain Bolt, it's simple, run faster! Fortunately, Python is well equipped for that. If you open a file with Python that you know it might contains mixed encoded text, you can use try/except to inform the user or open in binary mode, and just store the binary. But my favorite way of doing it is: open('file', error=strategy) Strategy can be: - "ignore": undecodable text is skipped - "replace": undecodable text is replace with "?" - "surrogateescape" (you need to use utf8): undecodable text is decoded to a special representation which makes no human sense, but can en rencoded back to it's original value. It's kinda ironic because people bashed Python for separating bytes/text, forcing them to deal with encoding correctly in Python 3. After all, this problem of "slamming bytes together" comes from languages that treat text as a bytes array, allowing this stupid mistake.

at_a_remove 6y ago
Yeah, that was more or less my approach to analysis for these files. Essentially it was an export that was quasi-aware of Unicode and dumped out certain fields (of course they had to be variable length) in their original encodings, whatever they were. I got more than a few of these. Unicode is great, as long as everyone upstream follows all of the rules and nothing goes wrong.