5 ms·
Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players. It’s much easier to tell the people wi
by zarzavat 3y ago
Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players.
It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.
- fl7305 3y ago> It’s much easier to tell the people with old cassette tapes to rip them I assume you mean "rip them", as in transcode to a different format? In that case, you need a tool that takes the old input format(s) and convert them to the new format. For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8. Isn't the point of the article to describe how an engineer would create such a tool?
- 0xffff2 3y agoA tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
- fl7305 3y ago> should be asking the user to select explicit input and output formats It depends on the requirements. If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
- tempaccount1234 3y agoUnless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive. It becomes easier to just use https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ on the input.
- jcranmer 3y ago> Isn't the point of the article to describe how an engineer would create such a tool? Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job. If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.
- fl7305 3y agoI agree 100% on the technical issues. But the point of the article is not the same thing as how well it achieved its goals.
- javajosh 3y ago>universal charset detection isn't really a thing that works all that well. This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
- fl7305 3y agoYeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text. In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|". So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
- extraduder_ire 3y agoI'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
- taneq 3y agoNo you don’t, they (the ones with the piles of old mixtapes) do.
- fl7305 3y agoI'm an engineer. I'm sometimes hired by they to create tools like that.
- gwervc 3y agoUTF-8 uses 50% more bytes than UFT-16 to encode Chinese or Japanese texts.
- bawolff 3y agoWhich basically never matters and in any case where it actually does, gzip will make it equal again.
- deleted 3y ago[deleted]
- iraqmtpizza 3y agozip-then-encrypt leaks information about the plaintext. if it's life or death, better not to compress at all
- tnmom 3y agoHuh, never heard that before. Does it leak more information than just encrypting without zipping? Struggling to imagine how this attack works.
- iraqmtpizza 3y agoEncrypting without zipping doesn't leak any information about the content. You can't rule out certain byte sequences (other than by total length) just by looking at the ciphertext length. If "oui" compresses to two bytes and "non" compresses to one byte, and then you go over them with a stream cipher, which is which: A: ; B: *&
- lisper 3y agoThis has nothing to do with compression. If you use "yes" and "no" instead of "oui" and "non" (which just happen to be three characters each) and you compress "yes" to "T" and "no" to "F" then the uncompressed text will be the leaky one.