2 ms·
Ah I see, thanks. You shouldn't require a BOM though. There are often files without a BOM, and not all files with UTF-16 in them are text either (EXEs etc.). I
by mehrdadn 7y ago
Ah I see, thanks. You shouldn't require a BOM though. There are often files without a BOM, and not all files with UTF-16 in them are text either (EXEs etc.). I would just search for all the possible UTF byte sequences (UTF-7, UTF-8, UTF-16LE/BE, UTF-32, possibly with a switch to allow specifying subsets or additional encodings if you can support that?) regardless of BOM.
- phiresky 7y agoIn ripgrep itself you can apparently only look in files of encodings other than UTF16LE with BOM by manually specifying `--encoding UTF16BE` etc. I could maybe add encoding detection myself, but I'm kind of discouraged since not even the unix `file` tool can detect those files as text, and a normal editor opens at least a UTF16BE file completely wrong. So I'm not sure if I want to spend my time on trying to write heuristic detection on those, especially since UTF16 itself is broken and shouldn't really exist at all... I'll look into what encoding_rs has to offer.
- mehrdadn 7y agoThanks! Yeah I wouldn't try to detect encodings or use heuristics either. If you could just reduce a single pattern into the OR of a bunch of byte sequences in each encoding, I think that should work? I'm not sure how easy that is with the interface you're given. (I wouldn't call UTF-16 'broken', but either way... it's a reality; a huge fraction of the time when you're searching binary files on Windows it's to find text inside executables, which on Windows are generally UTF-16.)