3 ms·

Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them. Lots of servers I've encountered with my browser stealth actually

Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them. Lots of servers I've encountered with my browser stealth actually violate the spec and send "lengths" that do not match the sent payload lengths afterwards. Some reverse proxies also mess up the last chunk, so they're violating the spec there, too, and send a chunk with a negative length...and the spec doesn't even define how to handle this. I've also seen servers send random lengths in between, but without a payload that follows. I would also like to add range requests (206 partial content) here. In practice, it's totally unpredictable how a server behaves when requesting multiple content ranges. Some reply with no range at all, even with correct headers. Some reply with more or less ranges than requested. Some even reply with out of boundary ranges that are larger than the content length header of the same response because they seem to use a faulty regexp on the server side. It's a total shitshow.

1vuio0pswjnm7 6y ago
"Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them." Always wondered if developers found that easy. I just strip out the chunk lengths with a filter, suitable for use in UNIX pipes. It's like three lines in flex. I have always been aware of the different things that servers could "legally" do with chunking from reading the HTTP/1.1 spec but as the parent says no ever does anything beyond the basic chunk lengths. For example, how many servers support chunked uploads. With the filter I wrote, as crude as it is, I have never had any problems. Works great with HTTP/1.1-pipelined DoH responses.
deathanatos 6y ago
> send a chunk with a negative length...and the spec doesn't even define how to handle this Hmm. I suppose it isn't explicitly called out, but I think it's fair to say that such a request is a 400 Bad Request, as it doesn't match the grammar. (There's no possibility for a negative chunk length, as there's no way to indicate it.)

1vuio0pswjnm7 6y ago

"Chunked encoding is used a lot in DNS over HTTPS servers, and it's a real pain to parse them." Always wondered if developers found that easy. I just strip out the chunk lengths with a filter, suitable for use in UNIX pipes. It's like three lines in flex. I have always been aware of the different things that servers could "legally" do with chunking from reading the HTTP/1.1 spec but as the parent says no ever does anything beyond the basic chunk lengths. For example, how many servers support chunked uploads. With the filter I wrote, as crude as it is, I have never had any problems. Works great with HTTP/1.1-pipelined DoH responses.

deathanatos 6y ago

> send a chunk with a negative length...and the spec doesn't even define how to handle this Hmm. I suppose it isn't explicitly called out, but I think it's fair to say that such a request is a 400 Bad Request, as it doesn't match the grammar. (There's no possibility for a negative chunk length, as there's no way to indicate it.)