3 ms·
> It’s much easier to tell the people with old cassette tapes to rip them I assume you mean "rip them", as in transcode to a different format? In that case, y
by fl7305 2y ago
> It’s much easier to tell the people with old cassette tapes to rip them
I assume you mean "rip them", as in transcode to a different format?
In that case, you need a tool that takes the old input format(s) and convert them to the new format.
For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8.
Isn't the point of the article to describe how an engineer would create such a tool?
- 0xffff2 2y agoA tool whose purpose is to transcode should be asking the user to select explicit input and output formats, not guessing.
- fl7305 2y ago> should be asking the user to select explicit input and output formats It depends on the requirements. If you're hired by a company to convert millions of old textfiles, they might want you to do it as well as possible using heuristics without any human input as a starting point.
- tempaccount1234 2y agoUnless the input is such garbage that asking the user is pointless (like a lot of web sources) and fixing manually is too time expensive. It becomes easier to just use https://pypi.org/project/ftfy/ https://pypi.org/project/ftfy/ on the input.
- jcranmer 2y ago> Isn't the point of the article to describe how an engineer would create such a tool? Honestly, no, because the tool that it's suggesting how to write isn't one that will even come close to doing a good job. If you want to write such a tool, the first thing you need to do is to understand what the correct answer is. And to do that, you need to sample your input set to figure out what the correct answer should be for several inputs where it matters. There's unfortunately no easy way to avoid that work; universal charset detection isn't really a thing that works all that well.
- fl7305 2y agoI agree 100% on the technical issues. But the point of the article is not the same thing as how well it achieved its goals.
- javajosh 2y ago>universal charset detection isn't really a thing that works all that well. This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
- fl7305 2y agoYeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text. In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|". So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
- extraduder_ire 2y agoI'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.
- taneq 2y agoNo you don’t, they (the ones with the piles of old mixtapes) do.
- fl7305 2y agoI'm an engineer. I'm sometimes hired by they to create tools like that.