3 ms·
To conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining"). The cURL project said it nev
by 1vuio0pswjnm7 5y ago
To conserve host resources RFC 2616 recommends making multiple HTTP requests over a single TCP connection ("HTTP/1.1 pipelining").
The cURL project said it never properly suported HTTP/1.1 pipelining and in 2019 it said it was removed once and for all.
https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-pipelining/ https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-...
Anyway, curl is not needed. One can write a small program in their language of choice to generate HTTP/1.1, but even a simple shell script will work. Even more, we get easy control over SNI which curl binary does have have.
There are different and more concise ways, but below is an example, using the IFS technique.
This also shows the use of sed's "P" and "D" commands (credit: Eric Pement's sed one-liners).
Assumes valid, non-malicious URLs, all with same host.
Usage: 1.sh < URLs.txt
#!/bin/sh
(IFS=/;while read w x y z;do
case $w in http:|https:);;*)exit;esac;
case $x in "");;*)exit;esac;
echo $y > .host
printf '%s\r\n' "GET /$z HTTP/1.1";
printf '%s\r\n' "Host: $y";
# add more headers here if desired;
printf 'Connection: keep-alive\r\n\r\n';done|sed 'N;$!P;$!D;$d';
printf 'Connection: close\r\n\r\n';
) >.http
read x < .host;
# SNI;
#openssl s_client -connect $x:443 -ign_eof -servername $x < .http;
# no SNI;
openssl s_client -connect $x:443 -ign_eof -noservername < .http;
exec rm .host .http;
- themk 5y agoThe curl binary will reuse the TCP connection when fed multiple URLs. Infact it can even use HTTP2 and make the requests in parallel over a single TCP connection. Common pattern I use is to construct URLs with a script and use xargs to feed to curl.
- 1vuio0pswjnm7 5y agoFor HTTP/1.1 pipelining, not HTTP/2 which not all websites support, the curl binary must be slower because the program tries to do more than just make HTTP from URLs and send the text over TCP. It tries to be "smart" and that slows it down. But dont't take my word for it, test it. For example, compare the retrieval speed of the above to something like sed -n '/^http/s/^/url=/p' URLs.txt|curl -K- --http1.1
- themk 5y agoI was responding to your original comment, which has since been edited: > When fed mutiple URLs, the curl binary will open multiple TCP connections, consuming more resources on the host. Which I felt was a bit of an unfair thing to say. I have no issue with the rest :)
- 1vuio0pswjnm7 5y agoAlthough historically it was true, I agree it was an unfair statement thus I removed it. Thank you for the correction. I will be testing curl a bit more; I rarely ever use it. I could never trust cURL for doing fast HTTP/1.1 pipelining so I am admittedly skeptical. It is a moving target that is constantly changing. The author clearly has a bias toward HTTP/2, and never really focused on HTTP/1.1 pipelining support even though HTTP/1.1 has, IME, worked really well for bulk text retrieval from a single host using a wide variety of smaller simpler programs,^1 not a graphical web browser, for at least fifteen years.^2 HTTP/2 is designed for graphical web pages full of advertising and tracking; introduced as a "standard" by a trillion dollar web advertising company who also controls the majority share web browser. Thankfully, HTTP/1.1 does not have that baggage. 1. Original netcat, djb's tcpclient, socat, etc. 2. It could be used to retrieve lots of small binary files too. See phttpget.
- saltcured 5y agoI'm not sure if you realize it but connection reuse and pipelining are not the same thing. Curl does connection reuse with HTTP/1.1 but not pipelining. Connection reuse is what was described up-thread, using a single connection to convey many requests and responses. Pipelining is a step further where the client pushes multiple requests down the connection before receiving responses. Pipelining is problematic with HTTP/1.1 because servers are allowed to close a connection to signal the end of a response, rather than announcing the Content-Length in the response header. The 1.1 protocol also requires that responses to pipelined requests come in the same order as the original requests. This is awkward and most servers do not bother to parallelize request processing and response writing. Even if the client sends requests back-to-back, the responses will come with delays between them. With HTTP/1.1 curl (like most clients) will do non-pipelined connection reuse. They push one request, read one response, then push the next request, etc. The network traffic will be small bursts with gaps between them, as each request-response cycle takes at least a round-trip delay. This is still faster than using independent connections, where extra TCP connection setup happens per request, and this is even more significant for HTTPS.
- 1vuio0pswjnm7 5y agoHeres another way to do it without the subshell, using tr. #!/bin/sh IFS=/;while read w x y z;do v=$(echo x|tr x '\34'); case $w in http:|https:);;*)exit;esac; case $x in "");;*)exit;esac; echo $y > .host printf '%s\r\n' "GET /$z HTTP/1.1"; printf '%s\r\n' "Host: $y"; printf 'Connection: keep-alive'$v$v;done \ |sed '$s/keep-alive/close/'|tr '\34\34' '\r\n' > .http; read x < .host; # SNI; #openssl s_client -connect $x:443 -ign_eof -servername $x < .http; # no SNI; openssl s_client -connect $x:443 -ign_eof -noservername < .http; exec rm .host .http;
- 1vuio0pswjnm7 5y agocase $x in "");;*)exit;esac is better written as test ${#x} = 0||exit