3 ms·

The author goes to lengths to explain how bash sucks at embedded null bytes in variable values (strings). But Bash is fine with nulls in pipes: $ printf 'f

The author goes to lengths to explain how bash sucks at embedded null bytes in variable values (strings). But Bash is fine with nulls in pipes: $ printf 'foo\x00bar\x00baz' | xargs -0 -n1 | wc -l 3 The real bash solution is you'd write a command to escape HTML entities, and set up the pipeline so that bash never has to assign any values that might have nulls. If I wanted a shell that was better at programming & manipulating test data, I'd use perl.

chubot 9y ago
(author here) Yes, bash can write NUL to stdout. But the point is that it can't store NUL in a string. You can't just replace \x01 from the last solution with \x00, e.g. Try: git log --pretty=format:$'%h\x00%s%\x00' The string is truncated at the first NUL. Compare with: git log --pretty=format:'%h%x00%s%x00' This matters because not all tools support something like %x00 -- it's specific to git. printf works because it's not storing NUL in a string, rather an escaped version, similar to echo -e '\x00' | od -c. Also I don't see the difference between "real bash solution" and what I described. That's exactly what I do, but you can't escape the entire string because it would remove your markup. I'd like to see a solution that is shorter but also correct against adversarial input like mine is.
- chubot 9y ago
  EDIT: I should say that the first command wouldn't work anyway because argv is NUL-terminated -- I mentioned this in "Note 2" with respect to grep. But still I'm failing to see what your point is about the "real bash solution". That's what this is, at least if you want to be 100% correct about escaping. I criticized the "normal bash solution" in the section on "The Worst Parts of Shell".
  - falsedan 9y ago
    > But still I'm failing to see what your point is about the "real bash solution" It's accepting that bash sucks at strings, and trying to keep it as far away as possible from the data in the pipeline. You keep pointing to the nulls and say, bash is bad at this and oil is good; but it's a strawman since bash doesn't have to even see the nulls for a bash script to be 100% correct at escaping the git log messages.
    - chubot 9y ago
      As noted in the article, I'm suggesting that the shell itself should able to handle searching a stream for NULs, without this roundabout way of doing things: count-nul() { # -o puts every match on its own line. (grep -o -c doesn't work.) od -A n -t x1 | grep -o '00' | wc -l } You can do this in Python or Perl, but it's not a stretch for the shell to do it directly.
      - falsedan 9y ago
        And I’m saying, this isn’t a feature I feel the lack of in bash. If I’m working with non-text data, I’ll move to a language with better handling of complex data structures like perl & explicitly handle concurrency myself. To be clear: oil has to be a compelling alternative to perl, not bash.

chubot 9y ago

(author here) Yes, bash can write NUL to stdout. But the point is that it can't store NUL in a string. You can't just replace \x01 from the last solution with \x00, e.g. Try: git log --pretty=format:$'%h\x00%s%\x00' The string is truncated at the first NUL. Compare with: git log --pretty=format:'%h%x00%s%x00' This matters because not all tools support something like %x00 -- it's specific to git. printf works because it's not storing NUL in a string, rather an escaped version, similar to echo -e '\x00' | od -c. Also I don't see the difference between "real bash solution" and what I described. That's exactly what I do, but you can't escape the entire string because it would remove your markup. I'd like to see a solution that is shorter but also correct against adversarial input like mine is.

chubot 9y ago
EDIT: I should say that the first command wouldn't work anyway because argv is NUL-terminated -- I mentioned this in "Note 2" with respect to grep. But still I'm failing to see what your point is about the "real bash solution". That's what this is, at least if you want to be 100% correct about escaping. I criticized the "normal bash solution" in the section on "The Worst Parts of Shell".
- falsedan 9y ago
  > But still I'm failing to see what your point is about the "real bash solution" It's accepting that bash sucks at strings, and trying to keep it as far away as possible from the data in the pipeline. You keep pointing to the nulls and say, bash is bad at this and oil is good; but it's a strawman since bash doesn't have to even see the nulls for a bash script to be 100% correct at escaping the git log messages.
  - chubot 9y ago
    As noted in the article, I'm suggesting that the shell itself should able to handle searching a stream for NULs, without this roundabout way of doing things: count-nul() { # -o puts every match on its own line. (grep -o -c doesn't work.) od -A n -t x1 | grep -o '00' | wc -l } You can do this in Python or Perl, but it's not a stretch for the shell to do it directly.
    - falsedan 9y ago
      And I’m saying, this isn’t a feature I feel the lack of in bash. If I’m working with non-text data, I’ll move to a language with better handling of complex data structures like perl & explicitly handle concurrency myself. To be clear: oil has to be a compelling alternative to perl, not bash.