4 ms·
Any known web (JS) workarounds for parsing CSVs with inconsistent file encodings? Right now I am at the conclusion that you must know the encoding type before p
by prodtorok 10y ago
Any known web (JS) workarounds for parsing CSVs with inconsistent file encodings? Right now I am at the conclusion that you must know the encoding type before parsing, though I've come across some decent libraries that can "guess" the encoding type:
https://github.com/aadsm/jschardet https://github.com/aadsm/jschardet
https://github.com/sonicdoe/detect-character-encoding https://github.com/sonicdoe/detect-character-encoding
- tantalor 10y agoNot sure what you mean by "inconsistent", but in general you have to know the character set before you read the file. https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim... Guessing will lead to tears.
- prodtorok 10y agoA system reading & loading csv's on it's own
- rspeer 10y agoIf it's a mix of UTF-8 and a single-byte encoding, you can guess, because UTF-8 is nice enough to be obvious when you guessed wrong. The command-line version of ftfy [1] can handle a distinct encoding for each line with a near-zero rate of false positives. (No guarantees the output will be correct, especially if you read it as the wrong single-byte encoding initially, but it will be no worse than what you put in, because it won't replace correct text with incorrect text.) There is no JS version of ftfy, and there won't be until JS can represent all Unicode characters without surrogates. [1] https://github.com/LuminosoInsight/python-ftfy https://github.com/LuminosoInsight/python-ftfy
- pjc50 10y agoIf you load an arbitary file from the filesystem, it won't tell you, so guessing is the only possible approach.