4 ms·

Windows support should be fairly trivial, the main problem is packaging it up and that travis doesn't like Windows. > doesn't support other UTF encodings like

Windows support should be fairly trivial, the main problem is packaging it up and that travis doesn't like Windows. > doesn't support other UTF encodings like UTF-16 UTF-16 should in fact work, since ripgrep supports it too. Looks like my binary file detection is at fault [1].. > Not sure if it can search in single-line mode either That works fine, just use `rga --multiline '\n' fname` [1]: https://github.com/phiresky/ripgrep-all/issues/5 https://github.com/phiresky/ripgrep-all/issues/5

mehrdadn 7y ago
Ah I see, thanks. You shouldn't require a BOM though. There are often files without a BOM, and not all files with UTF-16 in them are text either (EXEs etc.). I would just search for all the possible UTF byte sequences (UTF-7, UTF-8, UTF-16LE/BE, UTF-32, possibly with a switch to allow specifying subsets or additional encodings if you can support that?) regardless of BOM.
- phiresky 7y ago
  In ripgrep itself you can apparently only look in files of encodings other than UTF16LE with BOM by manually specifying `--encoding UTF16BE` etc. I could maybe add encoding detection myself, but I'm kind of discouraged since not even the unix `file` tool can detect those files as text, and a normal editor opens at least a UTF16BE file completely wrong. So I'm not sure if I want to spend my time on trying to write heuristic detection on those, especially since UTF16 itself is broken and shouldn't really exist at all... I'll look into what encoding_rs has to offer.
  - mehrdadn 7y ago
    Thanks! Yeah I wouldn't try to detect encodings or use heuristics either. If you could just reduce a single pattern into the OR of a bunch of byte sequences in each encoding, I think that should work? I'm not sure how easy that is with the interface you're given. (I wouldn't call UTF-16 'broken', but either way... it's a reality; a huge fraction of the time when you're searching binary files on Windows it's to find text inside executables, which on Windows are generally UTF-16.)
ShorsHammer 7y ago
Travis works fine for me building windows binaries. It's still beta, but wondering what problems you are having here? Have you tried windows in .travis.yml?
- phiresky 7y ago
  Nope, haven't tried it. I just saw that ripgrep is using appveyor for windows instead, so I assumed it doesn't work on travis. I was actually just trying to add appveyor to this [1], but I'm getting a weird error. [1]: https://ci.appveyor.com/project/phiresky/ripgrep-all/builds/25319098/job/j3jilrr5j6jfa9fi https://ci.appveyor.com/project/phiresky/ripgrep-all/builds/...
  - ShorsHammer 7y ago
    Give it a crack, not sure what dependencies ripgrep is pulling in but I've had good experiences so far. They seem to be doing ok with rolling it out. Just a few more lines in your travis deploy setup.

mehrdadn 7y ago

Ah I see, thanks. You shouldn't require a BOM though. There are often files without a BOM, and not all files with UTF-16 in them are text either (EXEs etc.). I would just search for all the possible UTF byte sequences (UTF-7, UTF-8, UTF-16LE/BE, UTF-32, possibly with a switch to allow specifying subsets or additional encodings if you can support that?) regardless of BOM.

phiresky 7y ago
In ripgrep itself you can apparently only look in files of encodings other than UTF16LE with BOM by manually specifying `--encoding UTF16BE` etc. I could maybe add encoding detection myself, but I'm kind of discouraged since not even the unix `file` tool can detect those files as text, and a normal editor opens at least a UTF16BE file completely wrong. So I'm not sure if I want to spend my time on trying to write heuristic detection on those, especially since UTF16 itself is broken and shouldn't really exist at all... I'll look into what encoding_rs has to offer.
- mehrdadn 7y ago
  Thanks! Yeah I wouldn't try to detect encodings or use heuristics either. If you could just reduce a single pattern into the OR of a bunch of byte sequences in each encoding, I think that should work? I'm not sure how easy that is with the interface you're given. (I wouldn't call UTF-16 'broken', but either way... it's a reality; a huge fraction of the time when you're searching binary files on Windows it's to find text inside executables, which on Windows are generally UTF-16.)

ShorsHammer 7y ago

Travis works fine for me building windows binaries. It's still beta, but wondering what problems you are having here? Have you tried windows in .travis.yml?

phiresky 7y ago
Nope, haven't tried it. I just saw that ripgrep is using appveyor for windows instead, so I assumed it doesn't work on travis. I was actually just trying to add appveyor to this [1], but I'm getting a weird error. [1]: https://ci.appveyor.com/project/phiresky/ripgrep-all/builds/25319098/job/j3jilrr5j6jfa9fi https://ci.appveyor.com/project/phiresky/ripgrep-all/builds/...
- ShorsHammer 7y ago
  Give it a crack, not sure what dependencies ripgrep is pulling in but I've had good experiences so far. They seem to be doing ok with rolling it out. Just a few more lines in your travis deploy setup.