3 ms·
>universal charset detection isn't really a thing that works all that well. This seems like something LLMs would be good at. A mundane use of them, but I bet t
by javajosh 2y ago
>universal charset detection isn't really a thing that works all that well.
This seems like something LLMs would be good at. A mundane use of them, but I bet they'd be really good at determining that the input has the wrong encoding. Then the program would iterate through encodings, from most probable to least, and select the one that the LLM likes the most. Granted, this means your tool will be 1GB or more. But hey, thems the breaks.
- fl7305 2y agoYeah, that could be an interesting use of LLMs. It could at least tell you which languages might be present in the input text. In the 1980s, we had a version of 7-bit ASCII in Sweden where the three extra Swedish vowels "åäö" were represented by "{}|". So what might look like regular US 7-bit ASCII should be interpreted as the Swedish version if the text is in Swedish with "{}|" where "åäö" normally goes.
- extraduder_ire 2y agoI'm glad that didn't stick around, like ¥/₩ being used for directory separators like \. I can't imagine trying to read source code with those substitutions.