5 ms·

Curl says bye bye to pipelining

74 points by okket 8y ago

skunkworker 8y ago
Note, this is about HTTP/1.1 Pipelining being removed not *nix Pipeline. HTTP/1.1 Pipelining was badly supported by curl for a long time in the sense that we had a series of known bugs and it was a fragile feature without enough tests.
- thunderbong 8y ago
  Thanks for that! I got a real fright!
- markmm 8y ago
  *nix pipes are part of POSIX and the OS/Shell implements them, if the app takes input and produces output it can be a component of a pipeline. In short, the Curl developers couldn't remove pipes without crippling the app.
  - jstanley 8y ago
    Which is exactly why the prospect of removing pipelining was so surprising!
  - DougMerritt 8y ago
    Everything you say is correct, and may even be a word to the wise, as it were -- but there are always new command line apps made by people who haven't been thoroughly exposed to the true Unix/Linux/Posix way, and need to have that kind of thing explained. [1] [2] [3] Which is to say that your comments are not self-evident to everyone, by any means. Example: less(1) was eventually fitted with the -R flag to allow highlighting to be passed by pipe, to make it friendly to piping from highlight-producing apps rather than its default highlight-unfriendly stripping of input controls/escapes. Benign intentions with regard to files sometimes require a little more thought to be simultaneously friendly to pipes, except for the simplest of plain text apps. (I'm not trying to make a big point, but I just rediscovered 'less -R' out of necessity today. So, I was pondering some of these issues) Worth a re-read for everyone (and I worry that these things may be in the process of being forgotten): [1] https://en.wikipedia.org/wiki/Unix_philosophy https://en.wikipedia.org/wiki/Unix_philosophy [2] https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond's_17_Unix_Rules https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond's... and the book: [3] https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming [4] Raymond didn't invent any of this, but he explained it very nicely, IMHO. Edit: P.S. the above is somewhat meandering, and is sure to be criticized because simple equivalence of piped and file input is the most important thing, which I agree with -- but I was mixing that with a secondary point because of what I was doing today. Sorry. "I did not have time to make this letter shorter" etc.
tomchristie 8y ago
Good call, given that it's not enabled in modern browsers, and is unreliably supported in proxies and servers. It's for the best that it becomes widely treated as a non-feature of HTTP/1.1.
- iforgotpassword 8y ago
  Still somewhat baffling that it's apparently so buggy across servers and proxies that it was for the best to disable it across browsers. In comparison to http2 multiplexing it seems trivial to get right, yet posts like this one suggest that http2 works fine everywhere and in every regard, and http1.x doesn't.
  - rakoo 8y ago
    It's because real life http2 makes encryption mandatory, so that middle boxes can't fiddle with the stream. Thanks to that everything arrives as it was intended on the server
sohkamyung 8y ago
Maybe the headline should be edited to "Curl says bye bye to HTTP/1.1 Pipelining" to prevent giving people scanning the HN headlines a heart attack. :-)
- izietto 8y ago
  Yeah, this title is a bit "click-baity"
- karmakaze 8y ago
  I had to read a many comments to see how this could be interpreted as alarming. I guess I assume more context of HM readers.
amelius 8y ago
Why isn't pipelining part of TCP/IP, so the OS can handle it instead of the application?
- vbezhenar 8y ago
  Because TCP was designed as a single-stream protocol. There's SCTP which operates with messages rather than streams and can send those messages simultaneously.
  - amelius 8y ago
    Sounds interesting. Is that implemented in kernels? And why wasn't HTTP1.1+ built on top of that?
    - sanxiyn 8y ago
      Yes, SCTP is implemented in kernels.
    - deleted 8y ago
      [deleted]
    - Volundr 8y ago
      Pretty much any modern OS supports SCTP. HTTP wasn't built off it because HTTP came long before SCTP, but you won't find many protocols built off SCTP anyway, because it's poorly supported by middle-boxes and firewalls. Just look at any cloud firewall for example, you can open TCP or UDP, but not SCP or any of the "lesser known" protocols. It's currently in an unfortunate chicken and egg problem, where crappy firewalls don't support it because it's rarely used, and it's rarely used because crappy firewalls don't support it, but are everywhere on the public internet. HTTP3 actually does propose changing the transport protocol, but to QUIC, which is a transport over UDP.
      - lttlrck 8y ago
        SCTP can be tunneled over UDP - which WebRTC does. It didn’t get the right traction. It isn’t built into windows, it was even a second class citizen on Linux not that long ago requiring lksctp to be installed (maybe it still does).

74 points by okket 8y ago

Note, this is about HTTP/1.1 Pipelining being removed not *nix Pipeline. HTTP/1.1 Pipelining was badly supported by curl for a long time in the sense that we had a series of known bugs and it was a fragile feature without enough tests.

thunderbong 8y ago
Thanks for that! I got a real fright!
markmm 8y ago
*nix pipes are part of POSIX and the OS/Shell implements them, if the app takes input and produces output it can be a component of a pipeline. In short, the Curl developers couldn't remove pipes without crippling the app.
- jstanley 8y ago
  Which is exactly why the prospect of removing pipelining was so surprising!
- DougMerritt 8y ago
  Everything you say is correct, and may even be a word to the wise, as it were -- but there are always new command line apps made by people who haven't been thoroughly exposed to the true Unix/Linux/Posix way, and need to have that kind of thing explained. [1] [2] [3] Which is to say that your comments are not self-evident to everyone, by any means. Example: less(1) was eventually fitted with the -R flag to allow highlighting to be passed by pipe, to make it friendly to piping from highlight-producing apps rather than its default highlight-unfriendly stripping of input controls/escapes. Benign intentions with regard to files sometimes require a little more thought to be simultaneously friendly to pipes, except for the simplest of plain text apps. (I'm not trying to make a big point, but I just rediscovered 'less -R' out of necessity today. So, I was pondering some of these issues) Worth a re-read for everyone (and I worry that these things may be in the process of being forgotten): [1] https://en.wikipedia.org/wiki/Unix_philosophy https://en.wikipedia.org/wiki/Unix_philosophy [2] https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond's_17_Unix_Rules https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond's... and the book: [3] https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming [4] Raymond didn't invent any of this, but he explained it very nicely, IMHO. Edit: P.S. the above is somewhat meandering, and is sure to be criticized because simple equivalence of piped and file input is the most important thing, which I agree with -- but I was mixing that with a secondary point because of what I was doing today. Sorry. "I did not have time to make this letter shorter" etc.

tomchristie 8y ago

Good call, given that it's not enabled in modern browsers, and is unreliably supported in proxies and servers. It's for the best that it becomes widely treated as a non-feature of HTTP/1.1.

iforgotpassword 8y ago
Still somewhat baffling that it's apparently so buggy across servers and proxies that it was for the best to disable it across browsers. In comparison to http2 multiplexing it seems trivial to get right, yet posts like this one suggest that http2 works fine everywhere and in every regard, and http1.x doesn't.
- rakoo 8y ago
  It's because real life http2 makes encryption mandatory, so that middle boxes can't fiddle with the stream. Thanks to that everything arrives as it was intended on the server

sohkamyung 8y ago

Maybe the headline should be edited to "Curl says bye bye to HTTP/1.1 Pipelining" to prevent giving people scanning the HN headlines a heart attack. :-)

izietto 8y ago
Yeah, this title is a bit "click-baity"
karmakaze 8y ago
I had to read a many comments to see how this could be interpreted as alarming. I guess I assume more context of HM readers.

amelius 8y ago

Why isn't pipelining part of TCP/IP, so the OS can handle it instead of the application?

vbezhenar 8y ago
Because TCP was designed as a single-stream protocol. There's SCTP which operates with messages rather than streams and can send those messages simultaneously.
- amelius 8y ago
  Sounds interesting. Is that implemented in kernels? And why wasn't HTTP1.1+ built on top of that?
  - sanxiyn 8y ago
    Yes, SCTP is implemented in kernels.
  - deleted 8y ago
    [deleted]
  - Volundr 8y ago
    Pretty much any modern OS supports SCTP. HTTP wasn't built off it because HTTP came long before SCTP, but you won't find many protocols built off SCTP anyway, because it's poorly supported by middle-boxes and firewalls. Just look at any cloud firewall for example, you can open TCP or UDP, but not SCP or any of the "lesser known" protocols. It's currently in an unfortunate chicken and egg problem, where crappy firewalls don't support it because it's rarely used, and it's rarely used because crappy firewalls don't support it, but are everywhere on the public internet. HTTP3 actually does propose changing the transport protocol, but to QUIC, which is a transport over UDP.
    - lttlrck 8y ago
      SCTP can be tunneled over UDP - which WebRTC does. It didn’t get the right traction. It isn’t built into windows, it was even a second class citizen on Linux not that long ago requiring lksctp to be installed (maybe it still does).