8 ms·

Finding CSV files that start with a BOM using ripgrep

119 points by pcr910303 5y ago

asicsp 5y ago
>The --multiline option means the search spans multiple lines - I only want to match entire files that begin with my search term, so this means that ^ will match the start of the file, not the start of individual lines. That's not correct because the `m` flag gets enabled by the multiline option. $ printf 'a\nbaz\nabc\n' | rg -U '^b' baz Need to use `\A` to match start of file or disable `m` flag using `(?-m)`, but seems like there's some sort of bug though (will file an issue soon): $ printf 'a\nbaz\nabc\n' | rg -U '\Ab' baz $ printf 'a1\nbaz\nabc\n' | rg -U '\Ab' baz $ printf 'a12\nbaz\nabc\n' | rg -U '\Ab' $
- burntsushi 5y ago
  Yup, that's exactly right. '\A' or '(?-m)^' should work, but don't, because of an incorrectly applied optimization. The bug is fixed on master. Thanks for calling this to my attention! https://github.com/BurntSushi/ripgrep/issues/1878 https://github.com/BurntSushi/ripgrep/issues/1878
  - tialaramex 5y ago
    Hmm. Thanks for fixing this, but, two things about the tests 1. These seem like they're effectively integration tests. They check the entire ripgrep command line app works as intended. Is this because the bug was not where it looks like it is, in the regex crate, but elsewhere? If not, it seems like they'd be better as unit tests closer to where the bugs they're likely to detect would lie? 2. While repeating yourself exactly once isn't automatically a bad sign, it smells suspicious. It seems like there would be a lot of tests that ought to behave exactly the same with or without --mmap and so maybe that's a pattern worth extracting.
    - burntsushi 5y ago
      You're right, I probably should have added a unit test. I added one in a branch of ongoing work. I don't always add unit tests, but in this case, it was pretty easy to. (FWIW, this is in the grep-regex crate, not the regex crate. The grep-regex crate is a glue layer between regex and ripgrep that adds in a bunch of line-oriented optimizations.) > While repeating yourself exactly once isn't automatically a bad sign, it smells suspicious. It seems like there would be a lot of tests that ought to behave exactly the same with or without --mmap and so maybe that's a pattern worth extracting. Quite possibly. Not a bad idea. I already do that in the tests for PCRE2. Most tests are run with both the default regex engine and with PCRE2. (There are some tests that have intended behavioral differences with mmap enabled, but those can be handled on a case-by-case basis.)
  - Jehops 5y ago
    > That's not correct because the `m` flag gets enabled by the multiline option. Is this documented? RG(1) only says -m, --max-count <NUM> Limit the number of matching lines per files searched to NUM. Is this a different option or is there an implied <NUM> that prevents 'rg -U ^' from searching from the beginning of the file?
    - burntsushi 5y ago
      In this context the "m flag" refers to a flag inside the regex syntax. That is, when you use ripgrep's regex library as a standalone (as Rust programmers do), then '^' only matches at the beginning of text, where as '(?m)^' enables multi-line mode and thus permits to match at either the beginning of text or at the beginning of a line. ripgrep also has a -U/--multiline flag, but it's orthogonal to the regex mode called multiline. It's an unfortunate naming clash, but they are names in otherwise distinct namespaces. ripgrep always enables the regex flag 'm', regardless of whether -U/--multiline is enabled or not.
      - Jehops 5y ago
        Thank you for taking the time to explain!
tyingq 5y ago
"BOM" == UTF-8 Byte Order Mark I guess. I initially thought it was searching for "Bill of Materials" for electronics projects or similar.
- maxnoe 5y ago
  There is no utf8 bom. Utf8 has no Byte order ambiguity. Only utf-16 needs a bom.
  - codeulike 5y ago
    https://superuser.com/questions/1553666/utf-8-vs-utf-8-with-bom https://superuser.com/questions/1553666/utf-8-vs-utf-8-with-...
  - cerved 5y ago
    huh?
  - burntsushi 5y ago
    From Unicode 23.8[1]: > In UTF-8, the BOM corresponds to the byte sequence <EF BB BF>. Although there are never any questions of byte order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked. As with a BOM in UTF-16, this sequence of bytes will be extremely rare at the beginning of text files in other character encodings. For example, in systems that employ Microsoft Windows ANSI Code Page1252, <EF16 BB16 BF16> corresponds to the sequence <i diaeresis, guillemet, inverted question mark> “ï » ¿”. In practice, the UTF-8 BOM pops up. I usually see it on Windows. [1] - http://www.unicode.org/versions/Unicode13.0.0/ch23.pdf#G19635 http://www.unicode.org/versions/Unicode13.0.0/ch23.pdf#G1963...
    - cerved 5y ago
      it pops up a lot and it's annoying as it's an invisible diff in git
      - Measter 5y ago
nemetroid 5y ago
Here's a coreutils (two-liner) version: printf '\xEF\xBB\xBF' >bom.dat find . -name '*.csv' \ -exec sh -c 'head --bytes 3 {} | cmp --quiet - bom.dat' \; \ -print The -exec option for find can be used as a filter (though -exec disables the default action, -print, so it must be reenabled after). Could be made into a oneliner by replacing the 'bom.dat' argument to cmp with '<(printf ...)'.
- xorcist 5y ago
  The cmp in coreutils understands: -n, --bytes=LIMIT compare at most LIMIT bytes so head is not really necessary: find . -name '*.csv' -type f -exec cmp -sn 3 {} bom.dat \; -print Using -exec as a filter is a nice feature more people should use. That -type was put there just to avoid directories.
wodenokoto 5y ago
I don’t know if I’ll ever gonna need this, but I loved learning it!
RedShift1 5y ago
Off topic but related, why does UTF-16 and UTF-32 even exist? Doesn't UTF-8 have the capability to go up to 32 bit wide characters already?
- pwdisswordfish8 5y ago
  UTF-16 was first. Or rather, UCS-2, which was limited to the Basic Multilingual Plane, and which UTF-16 extends to the whole of Unicode.
- kbumsik 5y ago
  They exist before UTF-8 afaik.
- marcosdumay 5y ago
  UTF-16 and UTF-32 are older than UTF-8. Besides, at the beginning people were really against variable size encodings. UTF-8 won despite the Unicode consortium and all the committees effort, not because of it.
  - SCLeo 5y ago
    Do you know why UTF-8 won? I feel textual data only constitutes a very tiny portion of memory used, but working with fixed size encoding is so much more easier than variable size encodings.
    - nneonneo 5y ago
      The only universal, fixed-size encoding is UTF-32, which, as you can imagine, is very wasteful on space for ASCII text. Like it or not, most of the interesting strings in a given program are probably ASCII. UTF-16 is not a fixed-size encoding thanks to surrogate pairs. UCS-2 is a fixed-size encoding but can’t represent code points outside the BMP (such as emoji) which makes it unsuitable for many applications. Besides, most of the time individual code points aren’t what you care about anyway, so the cost of a variable-sized encoding like UTF-8 is only a small part of the overall work you need to support international text.
    - 5y ago
dazfuller 5y ago
I probably won’t ever need this, but I love the write up for a tool which I use daily
superjan 5y ago
One large source of byte order marks in utf8 is Windows. In MS DOS and later windows, 8 bit encoded files are assumed to be in the system code page, which to enable all the worlds writing systems varies from country to country. When utf8 came along, Microsoft tools disambiguated those from the local code page by prefixing them with a byte order mark. They also do this in (for instance) the .net framework Xml libraries(by default). I don’t know what .net core does. I suppose it made sense at the time but I’m sure they regret this by now.
- superjan 5y ago
  And, I bet a significant portion of the offending CSV files are from excel. Excel is really annoying because it also silently localizes CSV files: my language uses the comma as a decimal separator, so excel wil switch to semicolon for the delimiter.
nwellnhof 5y ago
Is there anything like --multiline in GNU grep?
- thijsvandien 5y ago
  See here: https://stackoverflow.com/a/7167115/1163893 https://stackoverflow.com/a/7167115/1163893.

119 points by pcr910303 5y ago

>The --multiline option means the search spans multiple lines - I only want to match entire files that begin with my search term, so this means that ^ will match the start of the file, not the start of individual lines. That's not correct because the `m` flag gets enabled by the multiline option. $ printf 'a\nbaz\nabc\n' | rg -U '^b' baz Need to use `\A` to match start of file or disable `m` flag using `(?-m)`, but seems like there's some sort of bug though (will file an issue soon): $ printf 'a\nbaz\nabc\n' | rg -U '\Ab' baz $ printf 'a1\nbaz\nabc\n' | rg -U '\Ab' baz $ printf 'a12\nbaz\nabc\n' | rg -U '\Ab' $

burntsushi 5y ago
Yup, that's exactly right. '\A' or '(?-m)^' should work, but don't, because of an incorrectly applied optimization. The bug is fixed on master. Thanks for calling this to my attention! https://github.com/BurntSushi/ripgrep/issues/1878 https://github.com/BurntSushi/ripgrep/issues/1878
- tialaramex 5y ago
  Hmm. Thanks for fixing this, but, two things about the tests 1. These seem like they're effectively integration tests. They check the entire ripgrep command line app works as intended. Is this because the bug was not where it looks like it is, in the regex crate, but elsewhere? If not, it seems like they'd be better as unit tests closer to where the bugs they're likely to detect would lie? 2. While repeating yourself exactly once isn't automatically a bad sign, it smells suspicious. It seems like there would be a lot of tests that ought to behave exactly the same with or without --mmap and so maybe that's a pattern worth extracting.
  - burntsushi 5y ago
    You're right, I probably should have added a unit test. I added one in a branch of ongoing work. I don't always add unit tests, but in this case, it was pretty easy to. (FWIW, this is in the grep-regex crate, not the regex crate. The grep-regex crate is a glue layer between regex and ripgrep that adds in a bunch of line-oriented optimizations.) > While repeating yourself exactly once isn't automatically a bad sign, it smells suspicious. It seems like there would be a lot of tests that ought to behave exactly the same with or without --mmap and so maybe that's a pattern worth extracting. Quite possibly. Not a bad idea. I already do that in the tests for PCRE2. Most tests are run with both the default regex engine and with PCRE2. (There are some tests that have intended behavioral differences with mmap enabled, but those can be handled on a case-by-case basis.)
- Jehops 5y ago
  > That's not correct because the `m` flag gets enabled by the multiline option. Is this documented? RG(1) only says -m, --max-count <NUM> Limit the number of matching lines per files searched to NUM. Is this a different option or is there an implied <NUM> that prevents 'rg -U ^' from searching from the beginning of the file?
  - burntsushi 5y ago
    In this context the "m flag" refers to a flag inside the regex syntax. That is, when you use ripgrep's regex library as a standalone (as Rust programmers do), then '^' only matches at the beginning of text, where as '(?m)^' enables multi-line mode and thus permits to match at either the beginning of text or at the beginning of a line. ripgrep also has a -U/--multiline flag, but it's orthogonal to the regex mode called multiline. It's an unfortunate naming clash, but they are names in otherwise distinct namespaces. ripgrep always enables the regex flag 'm', regardless of whether -U/--multiline is enabled or not.
    - Jehops 5y ago
      Thank you for taking the time to explain!

tyingq 5y ago

"BOM" == UTF-8 Byte Order Mark I guess. I initially thought it was searching for "Bill of Materials" for electronics projects or similar.

maxnoe 5y ago
There is no utf8 bom. Utf8 has no Byte order ambiguity. Only utf-16 needs a bom.
- codeulike 5y ago
  https://superuser.com/questions/1553666/utf-8-vs-utf-8-with-bom https://superuser.com/questions/1553666/utf-8-vs-utf-8-with-...
- cerved 5y ago
  huh?
- burntsushi 5y ago
  From Unicode 23.8[1]: > In UTF-8, the BOM corresponds to the byte sequence <EF BB BF>. Although there are never any questions of byte order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked. As with a BOM in UTF-16, this sequence of bytes will be extremely rare at the beginning of text files in other character encodings. For example, in systems that employ Microsoft Windows ANSI Code Page1252, <EF16 BB16 BF16> corresponds to the sequence <i diaeresis, guillemet, inverted question mark> “ï » ¿”. In practice, the UTF-8 BOM pops up. I usually see it on Windows. [1] - http://www.unicode.org/versions/Unicode13.0.0/ch23.pdf#G19635 http://www.unicode.org/versions/Unicode13.0.0/ch23.pdf#G1963...
  - cerved 5y ago
    it pops up a lot and it's annoying as it's an invisible diff in git
    - Measter 5y ago

nemetroid 5y ago

Here's a coreutils (two-liner) version: printf '\xEF\xBB\xBF' >bom.dat find . -name '*.csv' \ -exec sh -c 'head --bytes 3 {} | cmp --quiet - bom.dat' \; \ -print The -exec option for find can be used as a filter (though -exec disables the default action, -print, so it must be reenabled after). Could be made into a oneliner by replacing the 'bom.dat' argument to cmp with '<(printf ...)'.

xorcist 5y ago
The cmp in coreutils understands: -n, --bytes=LIMIT compare at most LIMIT bytes so head is not really necessary: find . -name '*.csv' -type f -exec cmp -sn 3 {} bom.dat \; -print Using -exec as a filter is a nice feature more people should use. That -type was put there just to avoid directories.

wodenokoto 5y ago

I don’t know if I’ll ever gonna need this, but I loved learning it!

RedShift1 5y ago

Off topic but related, why does UTF-16 and UTF-32 even exist? Doesn't UTF-8 have the capability to go up to 32 bit wide characters already?

pwdisswordfish8 5y ago
UTF-16 was first. Or rather, UCS-2, which was limited to the Basic Multilingual Plane, and which UTF-16 extends to the whole of Unicode.
kbumsik 5y ago
They exist before UTF-8 afaik.
marcosdumay 5y ago
UTF-16 and UTF-32 are older than UTF-8. Besides, at the beginning people were really against variable size encodings. UTF-8 won despite the Unicode consortium and all the committees effort, not because of it.
- SCLeo 5y ago
  Do you know why UTF-8 won? I feel textual data only constitutes a very tiny portion of memory used, but working with fixed size encoding is so much more easier than variable size encodings.
  - nneonneo 5y ago
    The only universal, fixed-size encoding is UTF-32, which, as you can imagine, is very wasteful on space for ASCII text. Like it or not, most of the interesting strings in a given program are probably ASCII. UTF-16 is not a fixed-size encoding thanks to surrogate pairs. UCS-2 is a fixed-size encoding but can’t represent code points outside the BMP (such as emoji) which makes it unsuitable for many applications. Besides, most of the time individual code points aren’t what you care about anyway, so the cost of a variable-sized encoding like UTF-8 is only a small part of the overall work you need to support international text.
  - 5y ago

dazfuller 5y ago

I probably won’t ever need this, but I love the write up for a tool which I use daily

superjan 5y ago

One large source of byte order marks in utf8 is Windows. In MS DOS and later windows, 8 bit encoded files are assumed to be in the system code page, which to enable all the worlds writing systems varies from country to country. When utf8 came along, Microsoft tools disambiguated those from the local code page by prefixing them with a byte order mark. They also do this in (for instance) the .net framework Xml libraries(by default). I don’t know what .net core does. I suppose it made sense at the time but I’m sure they regret this by now.

superjan 5y ago
And, I bet a significant portion of the offending CSV files are from excel. Excel is really annoying because it also silently localizes CSV files: my language uses the comma as a decimal separator, so excel wil switch to semicolon for the delimiter.

nwellnhof 5y ago

Is there anything like --multiline in GNU grep?

thijsvandien 5y ago
See here: https://stackoverflow.com/a/7167115/1163893 https://stackoverflow.com/a/7167115/1163893.