4 ms·

What a coincidence to see CRLF discussed in HN front page, this bit me just yesterday! While using the aws cli tool on Linux (the Docker containerized version)

by j1elo 5y ago

What a coincidence to see CRLF discussed in HN front page, this bit me just yesterday! While using the aws cli tool on Linux (the Docker containerized version) with the "text" output format, it turns out each output line ends with CRLF ("\r\n") instead of simply LF ("\n") as is common for any Linux cli tool, so I had to add a post-process step in my commands to remove CRs (a chained call to tr -d '\r'). Otherwise, my bash variables would end up containing those pesky carriage returns, and debug logs would show up unexpectedly bizarre. In retrospect it's obvious what happened, but it got me confused for a good while. Why would it output CRLF at all? Also of course, not a single mention of this in any of the docs I was reading.

bee_rider 5y ago
Vague memory, so take it with a grain of salt, but I think this might be a way of getting the break to show up in an editor that expects \n or one that expects \r. Maybe most editors that expect one simply ignore the other?
CornCobs 5y ago
The dos2unix tool is purpose-built for this!
- Josteniok 5y ago
  Or flip (https://ccrma.stanford.edu/~craig/utility/flip/ https://ccrma.stanford.edu/~craig/utility/flip/)! I've been carrying this thing around with me for years.
- mananaysiempre 5y ago
  Note that dos2unix and unix2dos do what you expect when used as filters, but overwrite the specified file in place when given a file name. This is a footgun second only to one in pdftotext, arguably, which (unsurprisingly) refuses to work as a filter and (surprisingly) will silently create a matching .txt file when passed a single .pdf as an argument.
  - spc476 5y ago
    Why is that a footgun? If it just piped the output to stdout, then to convert a file, you then have to $ dos2unix file >x $ mv x file In my mind, overwriting the file is the Right Thing.
    - romwell 5y ago
      The Right Thing would be to require an explicit -o <output filename> option to avoid modifying files without getting user's consent first. Especially because the changes it makes are irreversible (e. g. when a text file has mixed unix/dos line endings).
- j1elo 5y ago
  You are right, that's the best tool to use for the job! However, as it usually happens with scripts, adding non-coreutils dependencies becomes an issue (now the script needs to check for existence, request users to install if missing, etc...) For this use case which is so well scoped, a single character removal seems enough but in general I totally agree with dos2unix as the better choice.
thristian 5y ago
The Unix convention is for "\n" alone to signify a line-break; when Unix programs talk amongst themselves (such as piping one to another), that's what they do. The terminal convention (as described in the linked blog post) is for "\r\n" to signify a line-break. When a Unix program talks to a terminal, the kernel will (by default) automatically add or strip "\r" characters as needed. Docker messes this up, since the tool inside the container is isolated from the environment where the "docker" command is run. Since it can't directly talk to the "docker" command's standard output, you have to manually choose whether it's connected to a pseudo-terminal ("docker run -t ...") or with a pipe ("docker run ..."). If the command runs with a pipe, no output translation will be done, but the program will likely be difficult to use interactively - there's likely to be no interactive prompts, no line-editing, etc. If the command runs with a pseudo-terminal, then it will behave as if it's attached to a terminal - you're likely to get input prompts, ANSI colour codes, and "\r\n" line endings, even if the output of "docker run -t ..." is being piped to a file. I don't know if this is the behaviour you hit, but I can easily imagine somebody running a tool in a docker container interactively with "docker run -it ..." until they got the behaviour they wanted, then just sticking that command in a shell pipeline like they would any other Unix tool, and getting a nasty surprise.
- j1elo 5y ago
  Thank you a lot for this explanation. I have a good understanding of the difference between using or not a pseudo-terminal (and its effects on Docker commands) but I had never learned before about this distinction on behavior of line breaks. I'm launching the command with -it as suggested on https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-docker.html#cliv2-docker-install https://docs.aws.amazon.com/cli/latest/userguide/install-cli... However, now that you mention it, this page does comment on the usage of -it, so that might be it... thanks again for the hint!
- gnubison 5y ago
  For the record, ONLRET is the relevant flag and you can disable it with “stty -onlret”. Then output newlines will stay newlines (which typically results in bizarre behavior if those raw newlines are sent to a terminal).
mdaniel 5y ago
Related to that, the ansible "community.aws.aws_ssm" connection plugin[0] always returns the output of "raw:" tasks with "\r" appended to it, too, and I'd guess it's for a similar reason 0: https://github.com/ansible-collections/community.aws/blob/3.0.1/plugins/connection/aws_ssm.py#L14 https://github.com/ansible-collections/community.aws/blob/3....
ghshephard 5y ago
Weird - I also had to write this exact thing: Creds () { kubectl exec -ti $(getpod cred-api) -- printenv | grep --color=auto --color=auto RAWDATA | tr -d '\r' }
- j1elo 5y ago
  See my "UPDATE" comment; basically the issue seems to be that the output from "docker run" will contain DOS-type line breaks ("\r\n") if running with a pseudo-tty attached, so, with the "-t" argument. If that is not needed (e.g. when scripting) this argument can be dropped, and then the stdout comes clean with Linux style line breaks ("\n"). I don't know about kubectl, but if those "-ti" mean the same as in plain Docker, you could check if dropping the "-t" helps. After all, it looks like you are also scripting, so any kind of interactive input/output would probably be unexpected and undesired in your command.
j1elo 5y ago
UPDATE: The comment by user thristian [0] did indeed contain an explanation for this and hinted to what is the solution to this problem! "docker run ..." should be scripted without "-t", in order to run the command without a pseudo-terminal, which then yields output with Linux style line breaks ("\n"). [0]: https://news.ycombinator.com/item?id=30255017 https://news.ycombinator.com/item?id=30255017
- kps 5y ago
  If for some reason you do want a terminal, you can run `stty -onlcr` to turn off the NL → CR LF conversion.

by j1elo 5y ago

bee_rider 5y ago

Vague memory, so take it with a grain of salt, but I think this might be a way of getting the break to show up in an editor that expects \n or one that expects \r. Maybe most editors that expect one simply ignore the other?

CornCobs 5y ago

The dos2unix tool is purpose-built for this!

Josteniok 5y ago
Or flip (https://ccrma.stanford.edu/~craig/utility/flip/ https://ccrma.stanford.edu/~craig/utility/flip/)! I've been carrying this thing around with me for years.
mananaysiempre 5y ago
Note that dos2unix and unix2dos do what you expect when used as filters, but overwrite the specified file in place when given a file name. This is a footgun second only to one in pdftotext, arguably, which (unsurprisingly) refuses to work as a filter and (surprisingly) will silently create a matching .txt file when passed a single .pdf as an argument.
- spc476 5y ago
  Why is that a footgun? If it just piped the output to stdout, then to convert a file, you then have to $ dos2unix file >x $ mv x file In my mind, overwriting the file is the Right Thing.
  - romwell 5y ago
    The Right Thing would be to require an explicit -o <output filename> option to avoid modifying files without getting user's consent first. Especially because the changes it makes are irreversible (e. g. when a text file has mixed unix/dos line endings).
j1elo 5y ago
You are right, that's the best tool to use for the job! However, as it usually happens with scripts, adding non-coreutils dependencies becomes an issue (now the script needs to check for existence, request users to install if missing, etc...) For this use case which is so well scoped, a single character removal seems enough but in general I totally agree with dos2unix as the better choice.

thristian 5y ago

The Unix convention is for "\n" alone to signify a line-break; when Unix programs talk amongst themselves (such as piping one to another), that's what they do. The terminal convention (as described in the linked blog post) is for "\r\n" to signify a line-break. When a Unix program talks to a terminal, the kernel will (by default) automatically add or strip "\r" characters as needed. Docker messes this up, since the tool inside the container is isolated from the environment where the "docker" command is run. Since it can't directly talk to the "docker" command's standard output, you have to manually choose whether it's connected to a pseudo-terminal ("docker run -t ...") or with a pipe ("docker run ..."). If the command runs with a pipe, no output translation will be done, but the program will likely be difficult to use interactively - there's likely to be no interactive prompts, no line-editing, etc. If the command runs with a pseudo-terminal, then it will behave as if it's attached to a terminal - you're likely to get input prompts, ANSI colour codes, and "\r\n" line endings, even if the output of "docker run -t ..." is being piped to a file. I don't know if this is the behaviour you hit, but I can easily imagine somebody running a tool in a docker container interactively with "docker run -it ..." until they got the behaviour they wanted, then just sticking that command in a shell pipeline like they would any other Unix tool, and getting a nasty surprise.

j1elo 5y ago
Thank you a lot for this explanation. I have a good understanding of the difference between using or not a pseudo-terminal (and its effects on Docker commands) but I had never learned before about this distinction on behavior of line breaks. I'm launching the command with -it as suggested on https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-docker.html#cliv2-docker-install https://docs.aws.amazon.com/cli/latest/userguide/install-cli... However, now that you mention it, this page does comment on the usage of -it, so that might be it... thanks again for the hint!
gnubison 5y ago
For the record, ONLRET is the relevant flag and you can disable it with “stty -onlret”. Then output newlines will stay newlines (which typically results in bizarre behavior if those raw newlines are sent to a terminal).

mdaniel 5y ago

Related to that, the ansible "community.aws.aws_ssm" connection plugin[0] always returns the output of "raw:" tasks with "\r" appended to it, too, and I'd guess it's for a similar reason 0: https://github.com/ansible-collections/community.aws/blob/3.0.1/plugins/connection/aws_ssm.py#L14 https://github.com/ansible-collections/community.aws/blob/3....

ghshephard 5y ago

Weird - I also had to write this exact thing: Creds () { kubectl exec -ti $(getpod cred-api) -- printenv | grep --color=auto --color=auto RAWDATA | tr -d '\r' }

j1elo 5y ago
See my "UPDATE" comment; basically the issue seems to be that the output from "docker run" will contain DOS-type line breaks ("\r\n") if running with a pseudo-tty attached, so, with the "-t" argument. If that is not needed (e.g. when scripting) this argument can be dropped, and then the stdout comes clean with Linux style line breaks ("\n"). I don't know about kubectl, but if those "-ti" mean the same as in plain Docker, you could check if dropping the "-t" helps. After all, it looks like you are also scripting, so any kind of interactive input/output would probably be unexpected and undesired in your command.

j1elo 5y ago

UPDATE: The comment by user thristian [0] did indeed contain an explanation for this and hinted to what is the solution to this problem! "docker run ..." should be scripted without "-t", in order to run the command without a pseudo-terminal, which then yields output with Linux style line breaks ("\n"). [0]: https://news.ycombinator.com/item?id=30255017 https://news.ycombinator.com/item?id=30255017

kps 5y ago
If for some reason you do want a terminal, you can run `stty -onlcr` to turn off the NL → CR LF conversion.