15 ms·
Curl doesn’t spew binary anymore
- Retr0spectrum 9y agoHow can a program determine whether its output is being piped to a file/program, or the terminal? Edit: It would appear that the answer is `isatty()`
- LogicX 9y agoCame here to say this. I can imagine a lot of people instead of doing -o were just redirecting output... if this changes the behavior of that, then a lot of people are not going to be happy when a curl upgrade breaks scripts.
- jacquesm 9y agoIt won't because 'isatty' returns false for a pipe. So that should not break anything unless you plan on sending binary output to your terminal which is a bad idea anyway, but they allow you to override the default as per TFA.
- arghwhat 9y agoYou can trick isatty, but only if you're very, very set on doing so. It requires allocating a psuedo-TTY, and attaching it. cURL already use isatty checks elsewhere, such as to determine if progress info should be printed (will only show if you're not outputting to tty).
- jacquesm 9y agoSure, but that's exactly what a pseudo-TTY is supposed to do, pretend it is a TTY when actually it is a pipe so isatty will return true. So that's not so much tricking isatty as much as using the system in the way it is intended. If you do that without the required background information then you have only yourself to blame.
- Asooka 9y agoA pipe is different from a pseudo-TTY. Pseudo-TTYs are how all graphical terminals work, but all shells use simple pipes on redirection and pipe-composition. TTYs and pseudo-TTYs are supposed to be interactive, with a user sitting on the other end with a keyboard. Plenty of tools make their output different based on if they're outputting to a tty or not, e.g. ls and others do not colourise by default if not outputting to a tty, as colour control characters are usually not what you want dumped to a text file or piped to grep. I feel this change brings curl more in-line with other terminal programs that try to be good citizens.
- jacquesm 9y agoPsuedo-TTY's come in pairs, one end for the application to connect to, the other for whatever is consuming the output or producing input to the application. So, yes, of course they are not pipes but they are similar in that you have two end-points. So pipes and pseudo-tty's have quite a bit in common, thinking of a pseudo-tty pair as a pipe that knows how to deal with a bunch of terminal specific ioctl calls is a pretty good approximation.
- arghwhat 9y agoConsidering it a pipe is a very rough approximation, though. The only purpose of PTYs is to provide access to a TTY with all its bells and whistles (kernel provided line editing, yay!) in a world without hard terminals.
- arghwhat 9y agoThe psuedo-TTY subsystem is designed to allow user sessions with virtual, rather than hard terminals. Telnet, ssh, terminal emulators. It provides everything you would expect from a tty. Cooked mode, baudrate control (despite redundant), all that jazz. Its purpose is not just to make isatty return true. If you spin up that giant subsystem to trick an application into a different behavior on a simple pipe, then I believe it is appropriate to call it a hack, trick, workaround or similar.
- eigengrau 9y ago> You can trick isatty, but only if you're very, very set on doing so. It requires allocating a psuedo-TTY, and attaching it. You can also LD_PRELOAD an isatty shim that returns true. This is occasionally useful when piping the output of tools which disable ANSI colorization with no switch to re-enable that.
- deathanatos 9y agoAnd note that isatty() works on a file descriptor, and that you might be outputting to both stderr and stdout, and both, neither, or some weird combination might be a tty. (i.e., you need to check for the file descriptor you're going to output to.) Some programs will check if stdout is a tty prior to writing to stderr, for example, which means that if you're doing, foo | bar foo's stderr will go to the tty, but would lose coloring, for example. For example, `babel`, a JS transpiler, loses the color in its error messages when you pipe its output.
- rrdharan 9y agoHooray! I'm gonna go ahead and claim credit for the festure request (though I never attempted to implement it :) https://curl.haxx.se/mail/archive-2014-02/0032.html https://curl.haxx.se/mail/archive-2014-02/0032.html
- TekMol 9y agoI think this is a bad idea. Trying to make programs intelligent and have them behave in unexpected ways usually is. It will make curl fail if a 0 byte is outputted within the text. There are many reasons why this will happen. Software errors, UTF-Encoding, special use cases... In fact, I run into this problem with grep from time to time and it is super annoying: > grep somefile.txt stuff Binary file somefile.txt matches So this change makes the code base more complex and the behavior fuzzy. I prefer elegant software that behaves predictably.
- Crontab 9y agoI am sympathetic to your comment because it feels like we are babying users. To me, this is no different than when FreeBSD and GNU modified the rm command to prevent people from shooting themselves in the foot.
- shawnz 9y agoI thought they changed it to prevent buggy scripts from executing rm with empty variables? That's not necessarily the fault of the user.
- pikzen 9y agoWhich, by any standard, is a good thing unless you're an elitist, gatekeeping prick.
- pcwalton 9y ago> To me, this is no different than when FreeBSD and GNU modified the rm command to prevent people from shooting themselves in the foot. Which was the right change. It prevents data loss for negligible added complexity.
- arghwhat 9y agoA NUL byte can't print in a terminal or show in an editor, so treating it as binary is a sensible thing to do when you are asking cURL to print it to your terminal. It is no longer text if a NUL byte is present (unless you use UTF-16/32, in which case, WHY?!). I assume that you also inserted a NUL byte into your somefile.txt to make grep treat it as binary. cURL behaves predictably now as well as before. It will warn you if you try to print unprintable data to your terminal, but can be told to ignore it, and won't affect output redirects.
- x0 9y agoIs there going to be a ~/.curlrc option so I can turn this off? Else I suppose I can add -o- to my curl alias.
- Daviey 9y agowhat situation would you want this?
- arghwhat 9y ago-o- seems like a terrible idea in a curl alias. Why do you want cURL to print binary data to your terminal? The check disables if you're redirecting output (> or |).
- dullgiulio 9y agoYou could simply write a wrapper that reads an options file. In general, it's a bad idea to have a hidden state such as a configuration file. Imagine, for examples, scripts that expect the standard curl behaviour, but your configuration file changes that. (Of course setting shell aliases has the same problem...)
- slrz 9y agoShell aliases do not have that problem as they're generally used with interactive shells only. Non-interactive shells tend to read a different set of configuration files for that reason.
- tyingq 9y agoI'll probably just add "| cat" on in the rare case when I want unfiltered output, then I don't need to memorize any options. That's what I do with ls when I temporarily don't want colorized output.
- deleted 9y ago[deleted]
- baby 9y agoI just looked at the commit and I was surprised at how nice the code is! The relevant line to my question is: if(isatty && (outs->bytes < 2000) && terminal_binary_ok) { if(memchr(buffer, 0, bytes)) { memchr() returns the first occurrence of the byte 0 (your second argument), or NULL. So a few things: * what if your output is more than 2000 bytes? * what if your output is binary but doesn’t contain a byte 0? * what if your output is a normal UTF-8 string but contains a byte 0? ( see https://stackoverflow.com/questions/6907297/can-utf-8-contain-zero-byte https://stackoverflow.com/questions/6907297/can-utf-8-contai... ) This is interesting to me as I am developing a tool that parses files in search for bugs and I need to ignore binary files. What I am doing right now: when checking a line, if it is not a valid UTF-8 string I skip the file. It's not really nice as I am doing this verification for every file's lines...
- grenoire 9y agoThere is no other Unicode code point that will be encoded in UTF8 with a zero byte anywhere within it. From what I understand, this means that no wild UTF-8 string would have a NUL byte anywhere in it, no?
- arghwhat 9y agoIt means that a UTF-8 string can contain the "NUL" character, which is a single null byte. While valid UTF-8, it is still an unprintable character, making it "binary" for most intents and purposes.
- mikeash 9y agoIt's worth noting that, while NUL is valid UTF-8, there's a lot of software which will fail strangely when presented with it, since UTF-8 is so often used to interoperate with NUL-terminated C strings. If you write such a string to a C string then it will be silently truncated. I ran into this once with an Objective-C program that created filenames from strings found in files. When presented with a string containing NUL, the code ran, but didn't really work. I'd get "foo" and then ask it to append ".txt" and the result would come back as "foo" still! And it depends on the context in which you use it. Using it as a filename truncated, because you ultimately go through the POSIX-level calls that use C strings. Printing it in the debugger truncated, as evidently it used C strings at some point in that process. Displaying it in a text field in the UI worked perfectly fine, though, as apparently that path never uses C strings and the NUL character is just an invisible zero-space character in the middle.
- agentgt 9y agoI have wondered if there are security hazards to piping binary to your terminal. I guess what really bad things can happen (and no I'm not talking about blindly piping to a shell to eval)? Sure it messes up the terminal of which I usually type clear and/or reset. If it's really bad I usually just kill the terminal.
- jwilk 9y agoMany terminal emulators are not very good at dealing with unstrusted input. http://www.openwall.com/lists/oss-security/2017/05/01/13 http://www.openwall.com/lists/oss-security/2017/05/01/13
- tyingq 9y agoThere used to be some terminal emulators that supported "screen dump", where the scrollback buffer would be written to an arbitrary named file. echo -e "\ec\n\e]55;/tmp/ouch.php\a" I believe they've all removed that functionality by now, but there might be some older copies around here and there.
- eigengrau 9y agoIIRC, another angle that has been proposed is to switch between the primary and secondary terminal buffers. This way, one could create a file that looks harmless when piped to the terminal but contains a hidden, malicious payload. The victim would, after manual inspection, finally pipe the file to a shell, where the payload would do something evil.
- joombaga 9y ago> The victim would, after manual inspection, finally pipe the file to a shell Why would anyone do this? If I see some unknown, interesting file, I might run cat, head, tail, less or vim on it. If it's binary then maybe I'll use xxd. But it wouldn't even occur for me to pipe it to a shell.
- lathiat 9y ago
- est 9y agoshould have used `-O - ` so the options is consistent with wget
- heinrich5991 9y ago`-O` is already used. `curl -O <URL>` is roughly equivalent to `wget <URL>`.
- kwvarga 9y agoThere are a number of download urls that are "one-time" use. Shouldn't this at least download the output to a /tmp file?
- avaer 9y agoNo. This would open an enormous can of worms, but I'll just pick one: security. And if you're the kind of person that needs this, the solution is a one-line alias.
- pebers 9y agoIf you're relying on the previous behaviour of curl on your one-time URL, you're already in trouble, because it's been printed to a terminal and the real contents are probably irretrievable.
- dstroot 9y agothe fact that cURL is hosted on haxx.se always throws me. It makes me immediately associate haxx === hacking and I feel uncomfortable. This is honestly why I use other tools instead even though it's more an unconscious decision than a conscious one.
- kam 9y agoYet you post on a site called Hacker News.
- ComputerGuru 9y agoI would never hire someone that properly doesn't use industry standard software that's purpose-built for the task just because the site name "makes them uncomfortable."
- dstroot 9y agocURL is a good utility but in my view calling it industry standard is a stretch. Personally I like httpie and postman.
- vacri 9y agoI'd prefer they stop curl from detecting a pipe on STDOUT and assuming that the user must want download progress garbage. Why would I want it in a pipe any more than I'd want it standalone? Inconsistent behaviour violates POLS.
- teddyh 9y agoThe GNU Coding Standards adresses this problem: “please don’t make the behavior of a command-line program depend on the type of output device it gets as standard output or standard input. […] There is an exception for programs whose output in certain cases is binary data. Sending such output to a terminal is useless and can cause trouble. If such a program normally sends its output to stdout, it should detect, in these cases, when the output is a terminal and give an error message instead. The -f option should override this exception, thus permitting the output to go to the terminal. ”¹ 1. https://www.gnu.org/prep/standards/standards.html#User-Interfaces https://www.gnu.org/prep/standards/standards.html#User-Inter...
- vacri 9y agoI don't know about the version of curl you use, but mine doesn't give download progress in binary format, and doesn't contain any content of the target you're curling. $ curl news.ycombinator.com > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 178 0 178 0 0 369 0 --:--:-- --:--:-- --:--:-- 370 You have to add an extra arg to get rid of that progress meter.
- joombaga 9y agoThe behavior doesn't exactly switch on pipe detection though. The progress bar consistently shows when outputting to a file. $ curl news.ycombinator.com -o /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 178 0 178 0 0 948 0 --:--:-- --:--:-- --:--:-- 951 I mean, it's effectively doing tty detection, but I think it's fair to argue a semantic difference, and it's very similar to how ls strips colors and disables listing format when piped, which is mentioned by the next paragraph in the coding standards. Compatibility requires certain programs to depend on the type of output device. It would be disastrous if ls or sh did not do so in the way all users expect. In some of these cases, we supplement the program with a preferred alternate version that does not depend on the output device type. For example, we provide a dir program much like ls except that its default output format is always multi-column format.
- teddyh 9y agoChecking for the NUL byte seems like a very arbitrary test. Shouldn’t it be using isprint_l()? Isn’t that, in fact, what isprint_l() is for?
- aaron_m04 9y agoWhy not check for ESC (0x1b, ^[) bytes instead? Without those, you can't mess up your terminal that badly.
- mlgh 9y agoNow you can annoy curl users by putting null byte in the beginning of the document.
- yaronn01 9y agoThere are some valid use cases here, I hope this will not affect users of wopr (https://github.com/yaronn/wopr https://github.com/yaronn/wopr).
- jkbr 9y agoGreat feature. HTTPie, a CLI HTTP client I created, has always had a similar one: No binary data in terminal output, and on top of that, you get to see the response headers by default as well. https://httpie.org/doc#binary-data https://httpie.org/doc#binary-data