4 ms·

I'd add to this list: Chunk extensions. Most people know HTTP/1.1 can return a "chunked" response body: it breaks the body up into chunks, so that we can send

I'd add to this list: Chunk extensions. Most people know HTTP/1.1 can return a "chunked" response body: it breaks the body up into chunks, so that we can send a response whose length we don't know in advance, but also it allows us to keep the connection open after we're done. What most people don't know is that chunks can carry key-value metadata. The spec technically requires an implementation to at least parse them, though I think it is permitted to ignore them. I've never seen anything ever use this, and I hope that never changes. They're gone in HTTP/2. (So, also, if you thought HTTP/2 was backwards compatible: not technically!) The "Authorization" header: like "Referer", this header is misspelled. (It should be spelled, "Authentication".) Same applies to "401 Unauthorized", which really ought to be "401 Unauthenticated". ("Unauthorized" is "403 Forbidden", or sometimes "404 Not Found".) Also, header values. They're basically require implementing a custom string type to handle correctly; they're a baroque mix of characters & "opaque octets".

anticristi 6y ago
I used this tons for MJPEG streams.
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
jsmith45 6y ago
Those key value pairs in chunked encoding ("chunk extensions") are spec'ed to only be hop-by-hop, which makes them more or less completely unsuitable for actually using by end applications. Any proxy or reverse proxy are allowed to strip them. Indeed it can be argued that a conformant proxy is required to strip them, due to MUST ignore unknown extensions value requirement. (I suspect most do not strip them, and there is an argument to be made that blindly passing them through if not changing encoding could be considered ignoring them, but I'm not certain that is actually a conforming interpretation). Plus surely there are many crusty middleboxes that will break if anybody tried to use that feature. Remember all the hoops websockets had to jump through to have much of a chance working for most people because of those? Many break badly if anything they were not programmed to handle tries to pass through.
- deathanatos 6y ago
  > Those key value pairs in chunked encoding ("chunk extensions") are spec'ed to only be hop-by-hop, which makes them more or less completely unsuitable for actually using by end applications. Oof, I hadn't mentally connected those dots, but you're completely right. (As Transfer-Encoding is hop-by-hop, not end-to-end…)
varajelle 6y ago
Also the ridiculous User-Agent header which everyone spoofs.
- inopinatus 6y ago
  I heartily endorse surfing as Googlebot. It’s often a whole different web.
  - lanstin 6y ago
    Oh I have to try this now. Thanks.
  - pdimitar 6y ago
    How exactly? What's the difference?
deleted 6y ago
[deleted]
userbinator 6y ago
Also, header values. They're basically require implementing a custom string type to handle correctly; they're a baroque mix of characters & "opaque octets". You are supposed to treat all of them as "opaque octets"... or something like this might happen: https://news.ycombinator.com/item?id=25857729 https://news.ycombinator.com/item?id=25857729
- deathanatos 6y ago
  You can't. At some point, you have to actually make use of the headers, and some of those uses require decoding to a string. There is some wiggle room here, such as doing things like, header_as_raw_bytes == b"chunked" which I would argue is still decoding the header: your language of choice had to encode that string into bytes in some encoding in the first place, so even though you're comparing the encoded forms, there's still a character encoding at work. But, some of the headers are case-insensitive. E.g., Content-Type, Accept, Expect, etc. That golang bug is precisely not treating the non-characters (the "opaque octets", as defined by the standard, that is, the octets that form obs-text) as if they were characters. You won't hit that bug in the safe subset, presuming you're implementing other parts of the standard correctly. (Which is… a huge assumption, given HTTP's complexity, but that's sort of the point here.)
cookiengineer 6y ago
Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them. Lots of servers I've encountered with my browser stealth actually violate the spec and send "lengths" that do not match the sent payload lengths afterwards. Some reverse proxies also mess up the last chunk, so they're violating the spec there, too, and send a chunk with a negative length...and the spec doesn't even define how to handle this. I've also seen servers send random lengths in between, but without a payload that follows. I would also like to add range requests (206 partial content) here. In practice, it's totally unpredictable how a server behaves when requesting multiple content ranges. Some reply with no range at all, even with correct headers. Some reply with more or less ranges than requested. Some even reply with out of boundary ranges that are larger than the content length header of the same response because they seem to use a faulty regexp on the server side. It's a total shitshow.
- 1vuio0pswjnm7 6y ago
  "Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them." Always wondered if developers found that easy. I just strip out the chunk lengths with a filter, suitable for use in UNIX pipes. It's like three lines in flex. I have always been aware of the different things that servers could "legally" do with chunking from reading the HTTP/1.1 spec but as the parent says no ever does anything beyond the basic chunk lengths. For example, how many servers support chunked uploads. With the filter I wrote, as crude as it is, I have never had any problems. Works great with HTTP/1.1-pipelined DoH responses.
- deathanatos 6y ago
  > send a chunk with a negative length...and the spec doesn't even define how to handle this Hmm. I suppose it isn't explicitly called out, but I think it's fair to say that such a request is a 400 Bad Request, as it doesn't match the grammar. (There's no possibility for a negative chunk length, as there's no way to indicate it.)
nialv7 6y ago
I want to argue for the use of "Authorization". What you pass in the "Authorization" header is an user identity, which is established through authentication. And the server uses this identity to decide if you are authorized.

anticristi 6y ago

I used this tons for MJPEG streams.

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

jsmith45 6y ago

Those key value pairs in chunked encoding ("chunk extensions") are spec'ed to only be hop-by-hop, which makes them more or less completely unsuitable for actually using by end applications. Any proxy or reverse proxy are allowed to strip them. Indeed it can be argued that a conformant proxy is required to strip them, due to MUST ignore unknown extensions value requirement. (I suspect most do not strip them, and there is an argument to be made that blindly passing them through if not changing encoding could be considered ignoring them, but I'm not certain that is actually a conforming interpretation). Plus surely there are many crusty middleboxes that will break if anybody tried to use that feature. Remember all the hoops websockets had to jump through to have much of a chance working for most people because of those? Many break badly if anything they were not programmed to handle tries to pass through.

deathanatos 6y ago
> Those key value pairs in chunked encoding ("chunk extensions") are spec'ed to only be hop-by-hop, which makes them more or less completely unsuitable for actually using by end applications. Oof, I hadn't mentally connected those dots, but you're completely right. (As Transfer-Encoding is hop-by-hop, not end-to-end…)

varajelle 6y ago

Also the ridiculous User-Agent header which everyone spoofs.

inopinatus 6y ago
I heartily endorse surfing as Googlebot. It’s often a whole different web.
- lanstin 6y ago
  Oh I have to try this now. Thanks.
- pdimitar 6y ago
  How exactly? What's the difference?

deleted 6y ago

[deleted]

userbinator 6y ago

Also, header values. They're basically require implementing a custom string type to handle correctly; they're a baroque mix of characters & "opaque octets". You are supposed to treat all of them as "opaque octets"... or something like this might happen: https://news.ycombinator.com/item?id=25857729 https://news.ycombinator.com/item?id=25857729

deathanatos 6y ago
You can't. At some point, you have to actually make use of the headers, and some of those uses require decoding to a string. There is some wiggle room here, such as doing things like, header_as_raw_bytes == b"chunked" which I would argue is still decoding the header: your language of choice had to encode that string into bytes in some encoding in the first place, so even though you're comparing the encoded forms, there's still a character encoding at work. But, some of the headers are case-insensitive. E.g., Content-Type, Accept, Expect, etc. That golang bug is precisely not treating the non-characters (the "opaque octets", as defined by the standard, that is, the octets that form obs-text) as if they were characters. You won't hit that bug in the safe subset, presuming you're implementing other parts of the standard correctly. (Which is… a huge assumption, given HTTP's complexity, but that's sort of the point here.)

cookiengineer 6y ago

Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them. Lots of servers I've encountered with my browser stealth actually violate the spec and send "lengths" that do not match the sent payload lengths afterwards. Some reverse proxies also mess up the last chunk, so they're violating the spec there, too, and send a chunk with a negative length...and the spec doesn't even define how to handle this. I've also seen servers send random lengths in between, but without a payload that follows. I would also like to add range requests (206 partial content) here. In practice, it's totally unpredictable how a server behaves when requesting multiple content ranges. Some reply with no range at all, even with correct headers. Some reply with more or less ranges than requested. Some even reply with out of boundary ranges that are larger than the content length header of the same response because they seem to use a faulty regexp on the server side. It's a total shitshow.

1vuio0pswjnm7 6y ago
"Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them." Always wondered if developers found that easy. I just strip out the chunk lengths with a filter, suitable for use in UNIX pipes. It's like three lines in flex. I have always been aware of the different things that servers could "legally" do with chunking from reading the HTTP/1.1 spec but as the parent says no ever does anything beyond the basic chunk lengths. For example, how many servers support chunked uploads. With the filter I wrote, as crude as it is, I have never had any problems. Works great with HTTP/1.1-pipelined DoH responses.
deathanatos 6y ago
> send a chunk with a negative length...and the spec doesn't even define how to handle this Hmm. I suppose it isn't explicitly called out, but I think it's fair to say that such a request is a 400 Bad Request, as it doesn't match the grammar. (There's no possibility for a negative chunk length, as there's no way to indicate it.)

nialv7 6y ago

I want to argue for the use of "Authorization". What you pass in the "Authorization" header is an user identity, which is established through authentication. And the server uses this identity to decide if you are authorized.