3 ms·

A tool doesn't need to support locales to be used in a shell pipeline. The Unix philosophy and shell pipelines were born long before POSIX locales. I even speci

A tool doesn't need to support locales to be used in a shell pipeline. The Unix philosophy and shell pipelines were born long before POSIX locales. I even specifically contrasted this with POSIX compatibility. So it seems to me like you're disingenuously interpreting my words at this point. @dang more trolling

oguz-ismail 2y ago
[flagged]
- burntsushi 2y ago
  That isn't true. ripgrep will happily pass through data it gets as-is. It just itself doesn't support locales. Commands do not need to have locale support to be used in shell pipelines. `rg foo | rg -v bar` is an example that has nothing to do with locales but demonstrates that ripgrep can be used in shell pipelines.
  - oguz-ismail 2y ago
    It is. Consider foo | grep -E '^.{4}$' | bar If foo prints $'O\u011Fuz' and LC_ALL is set to a valid UTF-8 locale, bar will receive what foo printed. If foo prints $'O\xF0uz' and LC_ALL is assigned a valid ISO-8859 locale, again, bar will receive what foo printed; because foo, bar, and grep are compatible. You can't have that with ripgrep; you'll have to modify the shell script in between or worse, manually parse LC_ALL and hope ripgrep supports the encoding
    - whytevuhuni 2y ago
      I found this interesting, so I tried to test it: On LC_ALL=en_US.UTF-8: $ echo $'O\u011Fuz' | grep -E '^.{4}$' Oğuz $ echo $'O\u011Fuz' | rg '^.{4}$' Oğuz On LC_ALL=en_US.ISO-8859-1: $ echo $'O\xF0uz' | grep -E '^.{4}$' O�uz $ echo $'O\xF0uz' | rg '^.{4}$' It strangely doesn't find anything at all: $ echo $'O\xF0uz' | rg '^.*$' | wc -c 0 It only does once the $ anchor is removed: $ echo $'O\xF0uz' | rg '^.*' | wc -c 5
      - burntsushi 2y ago
        It's not strange because ripgrep doesn't understand non-UTF-8 data (unless there's a UTF-16 BOM, in which case, ripgrep will automatically understand it). But you can tell it to: $ echo $'O\xF0uz' | rg -E iso-8859-1 '^.{4}$' Oðuz The person you're responding to has been trolling in this thread (and others) by twisting words and claiming multiple false things. When I've fixed their errors, they don't acknowledge them as mistakes and just keep on twisting words.
    - burntsushi 2y ago
      First of all, that doesn't invalidate literally anything I said. I have never claimed that ripgrep supported locales. I only said that it could be used in shell pipelines. That doesn't mean it can be used in all shell pipelines in exactly the same way that grep can. I even clarified this explicitly. Which is why you are a troll. Second of all, you're just wrong. ripgrep uses UTF-8 by default, but you don't have to use it. And when you switch your locale to ISO-8859 (who uses that!?!? why does it mater!?!?), you are no longer emitting UTF-8. As you obviously know or else you wouldn't have been able to come up with the example. As has been pointed out elsewhere in this thread, you can disable Unicode mode in ripgrep: $ echo $'O\xF0uz' | LC_ALL=en_US.ISO-8859-1 grep -E '^.{4}$' Ouz $ echo $'O\xF0uz' | rg '^(?-u:.){4}$' Ouz Or just specify the encoding you want: $ echo $'O\xF0uz' | rg -E iso-8859-1 '^.{4}$' Oðuz ripgrep doesn't do this the same way that grep does. But you can achieve it, which is perfectly in line with what I said.
      - oguz-ismail 2y ago
        > you can disable Unicode mode in ripgrep And everything becomes a byte, that's not always what you want $ export LC_ALL=tr_TR.ISO-8859-9 $ echo $'O\xD0UZ' | grep -E '^[[:upper:]]{4}$' O�UZ > Or just specify the encoding you want And what if I want to support more than one encoding? Am I supposed to modify my shell script every time I run it?
        burntsushi 2y ago
        I'm sure you can get creative. :-) You can set an environment variable to control the encoding, expose a flag or any one of a number of other things to control the encoding. Either way, now you're just shifting the goalposts. You've also continued to ignore my most substantive rebuttal: that a specific example where ripgrep is not compatible with grep or doesn't behave the same doesn't mean it can't be used in shell pipelines. I use ripgrep in shell pipelines all of the time. As do many others. Literally nothing you've said has invalidated anything I've said. All you're doing is finding things that some implementations of grep can do that ripgrep (intentionally) cannot do in exactly the same way. But that's fine, because ripgrep was never, isn't and will never be compatible with grep: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#posix4ever https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos... So if you need grep compatibility get a fucking clue and just use grep.

oguz-ismail 2y ago

[flagged]

burntsushi 2y ago
That isn't true. ripgrep will happily pass through data it gets as-is. It just itself doesn't support locales. Commands do not need to have locale support to be used in shell pipelines. `rg foo | rg -v bar` is an example that has nothing to do with locales but demonstrates that ripgrep can be used in shell pipelines.
- oguz-ismail 2y ago
  It is. Consider foo | grep -E '^.{4}$' | bar If foo prints $'O\u011Fuz' and LC_ALL is set to a valid UTF-8 locale, bar will receive what foo printed. If foo prints $'O\xF0uz' and LC_ALL is assigned a valid ISO-8859 locale, again, bar will receive what foo printed; because foo, bar, and grep are compatible. You can't have that with ripgrep; you'll have to modify the shell script in between or worse, manually parse LC_ALL and hope ripgrep supports the encoding
  - whytevuhuni 2y ago
    I found this interesting, so I tried to test it: On LC_ALL=en_US.UTF-8: $ echo $'O\u011Fuz' | grep -E '^.{4}$' Oğuz $ echo $'O\u011Fuz' | rg '^.{4}$' Oğuz On LC_ALL=en_US.ISO-8859-1: $ echo $'O\xF0uz' | grep -E '^.{4}$' O�uz $ echo $'O\xF0uz' | rg '^.{4}$' It strangely doesn't find anything at all: $ echo $'O\xF0uz' | rg '^.*$' | wc -c 0 It only does once the $ anchor is removed: $ echo $'O\xF0uz' | rg '^.*' | wc -c 5
    - burntsushi 2y ago
      It's not strange because ripgrep doesn't understand non-UTF-8 data (unless there's a UTF-16 BOM, in which case, ripgrep will automatically understand it). But you can tell it to: $ echo $'O\xF0uz' | rg -E iso-8859-1 '^.{4}$' Oðuz The person you're responding to has been trolling in this thread (and others) by twisting words and claiming multiple false things. When I've fixed their errors, they don't acknowledge them as mistakes and just keep on twisting words.
  - burntsushi 2y ago
    First of all, that doesn't invalidate literally anything I said. I have never claimed that ripgrep supported locales. I only said that it could be used in shell pipelines. That doesn't mean it can be used in all shell pipelines in exactly the same way that grep can. I even clarified this explicitly. Which is why you are a troll. Second of all, you're just wrong. ripgrep uses UTF-8 by default, but you don't have to use it. And when you switch your locale to ISO-8859 (who uses that!?!? why does it mater!?!?), you are no longer emitting UTF-8. As you obviously know or else you wouldn't have been able to come up with the example. As has been pointed out elsewhere in this thread, you can disable Unicode mode in ripgrep: $ echo $'O\xF0uz' | LC_ALL=en_US.ISO-8859-1 grep -E '^.{4}$' Ouz $ echo $'O\xF0uz' | rg '^(?-u:.){4}$' Ouz Or just specify the encoding you want: $ echo $'O\xF0uz' | rg -E iso-8859-1 '^.{4}$' Oðuz ripgrep doesn't do this the same way that grep does. But you can achieve it, which is perfectly in line with what I said.
    - oguz-ismail 2y ago
      > you can disable Unicode mode in ripgrep And everything becomes a byte, that's not always what you want $ export LC_ALL=tr_TR.ISO-8859-9 $ echo $'O\xD0UZ' | grep -E '^[[:upper:]]{4}$' O�UZ > Or just specify the encoding you want And what if I want to support more than one encoding? Am I supposed to modify my shell script every time I run it?
      - burntsushi 2y ago
        I'm sure you can get creative. :-) You can set an environment variable to control the encoding, expose a flag or any one of a number of other things to control the encoding. Either way, now you're just shifting the goalposts. You've also continued to ignore my most substantive rebuttal: that a specific example where ripgrep is not compatible with grep or doesn't behave the same doesn't mean it can't be used in shell pipelines. I use ripgrep in shell pipelines all of the time. As do many others. Literally nothing you've said has invalidated anything I've said. All you're doing is finding things that some implementations of grep can do that ripgrep (intentionally) cannot do in exactly the same way. But that's fine, because ripgrep was never, isn't and will never be compatible with grep: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#posix4ever https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos... So if you need grep compatibility get a fucking clue and just use grep.