12 ms·
The HTTP crash course nobody asked for
- juped 4y agoIf you're using OpenBSD nc already, just use nc -c for TLS.
- stevewatson301 4y agoDepending on your version of nc, -c is for sending CRLFs or executing sent data as commands. You might be looking for ncat instead.
- Denvercoder9 4y agoIn OpenBSD nc (as GP mentioned), -c is for a TLS connection: https://man.openbsd.org/nc.1 https://man.openbsd.org/nc.1
- Aissen 4y agoReminder, there are many different netcats, here are some of the most commons: - netcat-traditional http://www.stearns.org/nc/ http://www.stearns.org/nc/ - netcat-openbsd : https://github.com/openbsd/src/blob/master/usr.bin/nc/netcat.c https://github.com/openbsd/src/blob/master/usr.bin/nc/netcat... (also packaged in Debian) - ncat https://nmap.org/ncat/ https://nmap.org/ncat/ - netcat GNU: https://netcat.sourceforge.net/ https://netcat.sourceforge.net/ (quite rare) To prevent any confusion, I like to recommend socat: http://www.dest-unreach.org/socat/ http://www.dest-unreach.org/socat/
- silon42 4y agoMy nc has that as -C, no -c option.
- est 4y agoI hope there's a h2 or TLS crash course.
- fasterthanlime 4y agoAgainst my better judgement, the article /does/ go over H2 (although H3 is all the rage right now). For TLS, I recommend The Illustrated TLS 1.3 Connection (Every byte explained and reproduced): https://tls13.xargs.org/ https://tls13.xargs.org/
- keewee7 4y agoThanks for the link! Are there other good crash courses on various protocols and standards? Directly jumping into the dry official specs is just too overwhelmingly sometimes.
- Icathian 4y agoI recently crammed a bunch on DNS for an interview, and I can recommend the cloudflare blogs on that topic as being quite good.
- tehmillhouse 4y agoI'd like to thank you for the time and effort it must take to research, write and edit these articles. The tone you strike with these articles is a delight to read, and I find myself gobbling these things up even for topics about which I (falsely, it usually turns out) consider myself fairly knowledgeable.
- tomcam 4y agoFunny and very helpful. Thank you.
- raydiatian 4y ago
- deleted 4y ago[deleted]
- Donckele 4y ago“By contrast, I think about Bluetooth a lot. I wish I didn't.” LOL, yes same here. Can’t wait for Bluetooths b̶a̶l̶l̶s̶ baggage to be chopped.
- danuker 4y agoHow is WiFi so much more reliable than Bluetooth? I installed a web server on my phone and send files this way much faster (and Android -> Apple works): https://f-droid.org/en/packages/net.basov.lws.fdroid/ https://f-droid.org/en/packages/net.basov.lws.fdroid/ I wish there were a standard for streaming (headphones could connect to your network via WPS, and stream some canonical URL with no configuration needed).
- 411111111111111 4y agoWiFi supposedly needs more power and has higher latency. Not sure how true that remains post WiFi6 though
- pletnes 4y agoRange and bandwidth is orders of magnitude larger, and both have direct limitations in terms of energy budget.
- masklinn 4y ago> How is WiFi so much more reliable than Bluetooth? WiFi uses near 10x the power Bluetooth does when active (and that’s before factoring in BLE which cuts that down in half). WiFi also has access to the much less crowded 5GHz band. IIRC WiFi is also a much simpler protocol, it’s just a data channel (its aim being to replace LAN cables). Plus in order to support cheap and specialised devices Bluetooth supports all sorts of profiles and applications. This makes the devices simpler, and means all the configuration can be automated to pairing, but it makes the generic hosts a lot more complicated.
- Reventlov 4y ago
- Andys 4y agoI learned HTTP1 pretty well but not much of 2. Since playing with QUIC, I've lost all interest in learning HTTP/2, it feels like something already outdated that we're collectively going to skip over soon.
- masklinn 4y agoFWIW HTTP/3 very much builds upon / reframes HTTP/2’s semantics, so it might be useful to get a handle on /2, as I’m not sure all the /3 documentation will frame it in /1.1 terms.
- deleted 4y ago[deleted]
- Joker_vD 4y agoIt's pretty much a transport-level protocol, just like QUIC.
- pcthrowaway 4y agoHTTP1 is definitely outdated (it was expeditiously replaced by HTTP 1.1), but I'd argue ignoring HTTP/2 might be more like ignoring IPv4 because we have IPv6 now
- fasterthanlime 4y agoI tend to agree with you there, however the thing I'm replacing does HTTP/2, and HTTP/3 is yet another can of worms as far as "production multitenant deployment" goes, so, that's what my life is right now. As far as learning goes, I do think HTTP/2 is interesting as a step towards understanding HTTP/3 better, because a lot of the concepts are refined: HPACK evolves into QPACK, flow control still exists but is neatly separated into QUIC, I've only taken a cursory look at H3 so far but it seems like a logical progression that I'm excited to dig into deeper, after I've gotten a lot more sleep.
- I_complete_me 4y agoThat was an excellent, well-written, well-thought out, well presented, interesting, humorous, enjoyable read. Coincidentally I recently did a Rust crash course so it all made perfect sense - I am not an IT pro. Anyhows, thanks.
- q-base 4y agoI will piggyback on your comment as I totally agree. I am amazed at the amount of work that must go into not just writing the article itself but all the implementations along the way. Really amazing job!
- pohuing 4y agoI highly recommend taking a look at the other writeups on fasterthanli.me they're almost all excellent
- mihneawalker 4y agoI'd like to ask you what crash course on Rust did you take, as there are quite a few out there, and it would help if someone recommends a certain course.
- atfzl 4y agoTry https://fasterthanli.me/articles/a-half-hour-to-learn-rust https://fasterthanli.me/articles/a-half-hour-to-learn-rust which is also written by the same author.
- I_complete_me 4y agoYou Tube Let's Get Rusty - ULTIMATE Rust Lang Tutorial! - Getting Started
- phenylene 4y agoPlaylist link: https://youtube.com/playlist?list=PLai5B987bZ9CoVR-QEIN9foz4QCJ0H2Y8 https://youtube.com/playlist?list=PLai5B987bZ9CoVR-QEIN9foz4...
- 4y ago
- Joker_vD 4y ago> HTTP/1.1 is a delightfully simple protocol, if you ignore most of it. As someone who had to write a couple of proxy servers, I can't express how so sadly accurate it is.
- chrismorgan 4y agoAnd this is why I expect HTTP/2 and HTTP/3 to be much more robust in the long term: the implementations are harder to write, and you won’t get anywhere without reading at least a some spec, whereas HTTP/1 is deceptively simple with therefore a lot of badly incorrect implementations, often with corresponding security problems.
- arjvik 4y agoAs someone who has not read the HTTP/1.1 spec, what are some pitfalls that could actually become security issues?
- pwdisswordfish9 4y agoBeing text-based. Which leads to people constructing protocol messages by printf and therefore tons of injection bugs.
- chrismorgan 4y agoThe most common proximate cause of security issues in format handling (parsing and emitting) comes from implementations differing in their parsing, or implementations emitting invalid values in a way that will be parsed differently. Probably the most common type of security issue then comes from smuggling values through, bypassing checks or triggering injection. (This is the essence of injection attacks as a broad class.) One of the easiest demonstrations of this in HTTP specifically is called HTTP request smuggling: https://portswigger.net/web-security/request-smuggling https://portswigger.net/web-security/request-smuggling. And the solution for that is pretty much: “stop using a text protocol, they’re too hard to use correctly”.
- jefftk 4y ago
- sireat 4y agoIs HTTP always the same protocol as HTTPS - given the same version - and ignoring the encryption from TLS? Theoretically yes, but in practice? I've done my share of nc testing even simpler protocols than HTTP/1.1 For some reason the migration to HTTPS scared me despite the security assurances. I could not see anything useful in wireshark anymore. I now had to trust one more layer of abstraction.
- dochtman 4y agoFor 1.1 and 2, the byte stream is the same for TCP vs TLS over TCP. For 3, it uses one stream per request over a QUIC connection which is always encrypted.
- st_goliath 4y ago> Is HTTP always the same protocol as HTTPS - given the same version - and ignoring the encryption from TLS? > Theoretically yes, but in practice? Yes, that's the whole point of encapsulation. The protocol is blissfully unaware of encryption and doesn't even have to be. It has no STARTTLS mechanism either. Your HTTPS traffic consists of a TCP handshake to establishes a TCP connection, a TLS handshake across that TCP connection to exchange keys and establish a TLS session, and the exact, same HTTP request/response traffic, inside the encrypted/authenticated TLS session. The wonderful magic of solving a problem by layering/encapsulating. > I could not see anything useful in wireshark anymore Wireshark supports importing private keys for that, see: https://wiki.wireshark.org/TLS https://wiki.wireshark.org/TLS
- fasterthanlime 4y agoThe article covers using Wireshark to decrypt TLS traffic using Pre-Shared Master Secrets!
- ok123456 4y agoThe encapsulation isn't complete because of SNI.
- Too 4y agoThe protocol is the same, but semantics in the applications can differ. Secure cookies only working on https to give one example.
- mustak_im 4y agoYay! this is going to be a great read for the weekend!
- mahdi7d1 4y agoI didn't ask but I needed it.
- cph123 4y agoFor a crash course would the code examples have been better in something like Python rather than Rust?
- rk06 4y agoNope, that’s the author’s favourite language. A regular reader would expect rust to be used like in previous articles
- fasterthanlime 4y agoMy whole thing is that I'm teaching Rust /while/ solving interesting, real-world problems (instead of looking at artificial code samples), so, if someone wants to write the equivalent article with Python, they should! I won't.
- leinadho 4y agoThe humorous style is very refreshing, if only my networking lecturers had been more witty I might remember more of this
- yfiapo 4y ago> We're not done with our request payload yet! We sent: > Host: neverssl.com > This is actually a requirement for HTTP/1.1, and was one of its big selling points compared to, uh... > AhAH! Drew yourself into a corner didn't you. > ...Gopher? I guess? I feel like the author must know this.. HTTP/1.0 supported but didn't require the Host header and thus HTTP/1.1 allowed consistent name-based virtual hosting on web servers. I did appreciate the simple natures of the early protocols, although it is hard to argue against the many improvements in newer protocols. It was so easy to use nc to test SMTP and HTTP in particular. I did enjoy the article's notes on the protocols however the huge sections of code snippets lost my attention midway.
- proto_lambda 4y ago> I feel like the author must know this The author does know this, it's a reference to a couple paragraphs above: > [...] and the HTTP protocol version, which is a fixed string which is always set to HTTP/1.1 and nothing else. > (cool bear) But what ab- > IT'S SET TO HTTP/1.1 AND NOTHING ELSE.
- yfiapo 4y agoThanks, missed that.
- fasterthanlime 4y agoYou know how some movie fans will sometimes pretend the sequels to some franchise don't exist? HTTP is the opposite.
- bak3y 4y agoSo HTTP is the old school Star Wars fans pretending the prequels don't exist?
- HideousKojima 4y agoNah it's the Buffy TV show fans ignoring the movie
- Icathian 4y agoAmos' writing style is just so incredibly good. I don't know anyone else doing these very long-form, conversational style articles. Plus, you know, just an awesome dev who knows his stuff. Huge fan.
- tmountain 4y agoThis is gold.
- photochemsyn 4y agoWhat a great overall site. Hopping down the links I found the section on files with code examples in JS, Rust and C, plus strace, really the best short explanation I've ever found online. https://fasterthanli.me/series/reading-files-the-hard-way/part-1 https://fasterthanli.me/series/reading-files-the-hard-way/pa...
- rpigab 4y agoThis is awesome, didn't read all of it yet, but I will for sure, I use HTTP way too much and too often to ignore some of these underlying concepts, and when I try to look it up, there's always way too much abstraction and the claims aren't proven to me with a simple example, and this article is full of simple examples. Thanks Amos!
- X-Istence 4y ago> This is not the same as HTTP pipelining, which I will not discuss, out of spite. That is cause HTTP pipelining was and is a mistake and is responsible for a ton of http request smuggling vulnerabilities because the http 1.1 protocol has no framing. No browser supports it anymore, thankfully.
- mgaunard 4y agoIsn't "HTTP pipelining" just normal usage of HTTP/1.1? Anyone that doesn't support this is broken. My own code definitely does not wait for responses before sending more requests, that's just basic usage of TCP.
- deathanatos 4y ago> Isn't "HTTP pipelining" just normal usage of HTTP/1.1? > Anyone that doesn't support this is broken. My own code definitely does not wait for responses before sending more requests, that's just basic usage of TCP. Yep. There is some "support" a server could do, in the form of processing multiple requests in parallel¹, e.g., if it gets two GET requests back to back, it could queue up the second GET's data in memory, or so. The responses still have to be streamed out in the order they came in, of course. Given how complex I imagine such an implementation would be, I'd expect that to be implemented almost never, though; if you're just doing a simple "read request from socket, process request, write response" loop, then like you say, pipelined requests aren't a problem: they're just buffered on the socket or in the read portion's buffers. ¹this seems fraught with peril. I doubt you'd want to parallelize anything that wasn't GET/HEAD for risk of side-effects happening in unexpected orders.
- X-Istence 4y agoHTTP pipelining is not normal usage of HTTP/1.1. And it means that if request number 1 fails, usually request number 2 and 3 are lost because servers will slam the door shut because of the lack of framing around HTTP it is too dangerous to try and continue parsing the HTTP requests that are incoming without potentially leading to a territory where they are parsing the incoming text stream wrong. This is what led to the many request smuggling, its because the front-end proxy treats the request different from the backend proxy and parses the same HTTP text stream differently. Since there is no framing there is no one valid way to say "this is where a request starts, and this is where a request ends and it is safe to continue parsing past the end of this request for the next request". Servers are also allowed to close the connection at will. So let's say I pipeline Request 1, 2, and 3. The server can respond to Request 1 with Connection: close, and now request 2 and 3 are lost. That's the reason HTTP pipelining is not supported by browsers/most clients. Curl removed it and there's a blog post about it: https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-pipelining/ https://daniel.haxx.se/blog/2019/04/06/curl-says-bye-bye-to-...
- danesparza 4y agoMore articles should be written in the style of this article. Thank you for this.
- stefs 4y agomost of his articles are written in this style. they're great!
- mlindner 4y agoIs there a way to get this guide without the annoying side-commentary?
- fasterthanlime 4y agoThe RFCs themselves are pretty dry, if that's your thing — https://httpwg.org/ https://httpwg.org/ has the freshest ones.
- antonvs 4y ago> Where every line ends with \r\n, also known as CRLF, for Carriage Return + Line Feed, that's right, HTTP is based on teletypes, which are just remote typewriters Does it need to be pointed out that this is complete bullshit?
- deleted 4y ago[deleted]
- tripa 4y agoKind of. Which part of it do you think is wrong?
- antonvs 4y agoHTTP is not “based on teletypes”. That’s just nerd hyperbole for a technical choice they don’t like, for irrational reasons.
- a1369209993 4y agoWell, I've definitely seen a lot of people claim (generally not word-for-word) that using a pointlessly-overlong encoding of newline that exists to cater to the design deficiencies of hardware from the nineteen-sixties is not bullshit, so... maybe? But only for rather mushy values of "need".
- kortex 4y agoIt's not totally right, but it's not totally wrong, either, kind of like the way the dimensions of the space shuttle booster are directly affected by the size of a pair of Roman war horses' asses. CRLF was used verily heavily and thus got baked into a lot of different places. Namely, it conveniently sidesteps the ambiguity of "some systems use CR, others use LF" by just putting both in, and since they are whitespace, there's not much downside other than the extra byte. Beyond that, there are many other clear and obvious connections between Hypertext Transfer Protocol and teletype machines. Many early web browsers were expected to be teletype machines [0]. So while it might be a bit of a stretch, I'd say this is far from "complete bullshit". [0] - http://info.cern.ch/hypertext/WWW/Proposal.html#:~:text=it%20performs%20the%20negotiation%20of%20formats%20in%20dialog%20with%20the%20server.%20for%20example%2C%20a%20browser%20for%20a%20vt100%20type%20display%20will%20always%20negotiate%20ascii%20text%20only%2C%20whereas%20a%20macintosh%20browser%20might%20be%20constructed%20to%20accept%20postscript%20or%20sgml.%20 http://info.cern.ch/hypertext/WWW/Proposal.html#:~:text=it%2...
- tinglymintyfrsh 4y agoGET / HTTP/1.0\r\n\r\n Still works with many websites.
- mannyv 4y agoWhen doing http by hand, it's better to do http/1.0 because that tells the server you (and it) can't do anything exciting.
- mannyv 4y agoAs far as i can tell the host header is pointless, because if it's ssl/tls you won't be able to read it and route it. That's what sni is for. If you aren't tls then you don't need it, unless you hit the server as an ip. But then why would you do that?
- Too 4y agoProxies doing TLS termination, with multiple servers behind.
- LukeShu 4y agoIt's for one server/IP serving multiple hostnames. For instance, the same physical server at 45.76.26.79 serves both www.lukeshu.com and git.lukeshu.com with the same instance of Nginx. Once Nginx decrypts the request, it needs to know which `server { … }` block to use to generate the reply. With TLS+SNI, this is redundant to the name from SNI. But we had TLS long before we had SNI, and we had HTTP long before we had TLS, and both of those scenarios need the `Host` header.
- mannyv 4y agoAlso, never trust the content length. It's been that way since before http was finalized. Use it as guidance, but don't treat it as canonical.