11 ms·

How do HTTP servers figure out Content-Length?

271 points by misonic 2y ago

pkulak 2y ago
And if you set your own content length header, most http servers will respect it and not chunk. That way, you can stream a 4-gig file that you know the size of per the metadata. This makes downloading nicer because browsers and such will then show a progress bar and time estimate. However, you better be right! I just found a bug in some really old code that was gzipping every response when it was appropriate (ie, asked for, textual, etc). But it was ignoring the content-length header! So, if it was set manually, it would then be wrong after compression. That caused insidious bugs for years. The fix, obviously, was to just delete that manual header if the stream was going to be compressed.
- bugtodiffer 2y ago
  Did you see if you could turn this into HTTP Request Smuggling? Or something else with security impact? Sounds like a powerful bug you have, potentially.
  - knallfrosch 2y ago
    To me it sounds like the server handled the request just fine and reused the header (which was wrong.) The client then had the problem of a wrong response.
    - guappa 2y ago
      If you say to read 1000 bytes from memory and then pass a 900 bytes long array, that's a security bug that can cause crash, corrupt data, and leaking stuff that shouldn't have leaked.
      - jpc0 2y ago
        The size of the buffer and how many bytes are written have nothing intrinsically linked to what the header says. It's a bug sure but does not mean there's any security issue on the server.
        guappa 2y ago
        It will likely generate corrupt files on the client as well.
        Aachen 2y ago
        Not very. The system might allocate that length ahead of time (I've seen that option in torrent clients and iirc ftp systems) but, latest by the time a FIN comes in, it'll know the file is finished and can truncate it. If finished-early downloads are not implemented despite it doing preallocation, that's still not a security bug
        guappa 2y ago
        if a FIN comes, the client will mark the file as partially downloaded. but it might not come, since decades http sends more than one file per connection, so it might just get the beginning of the next reply, write that and the next reply will be corrupt as well.
      - michaelmior 2y ago
        It's a bug for sure, but I think whether it's a security issue could depend on the language. If the callee is able to determine the length of the array, it can just return an error instead of a potential buffer overrun.
      - stickfigure 2y ago
        In this case, the 1000 bytes aren't being read from memory, they're being read from a socket. If you try to over-read from a socket the worst you'll get is a blocking call or an error (depending what mode you're in).
    - bugtodiffer 2y ago
      maybe response based trickery then? :D What happens to the response after that one, are the first 100 bytes cut of, or what? I'm pretty sure something like this can cause some form of HTTP desync in a loadbalancer/proxy setup.
- dotancohen 2y ago
  > That caused insidious bugs for years. A lot of people here could probably benefit professionally from hearing about what the bugs were. Knowing what to identify in the future could be really helpful. Thanks.
  - pkulak 2y ago
    Well, for years our logs would fill up with these nasty warnings of "connection dropped", something like that. Naturally, you think that's just some badly configured client, mobile connection, something. But then why would it be configured to log that as a warning (or it may have even triggered an error). I think that was because when payloads are small the compression overhead makes them larger, which means the Content-Length is too small and clients would be terminating the connection early. And getting garbage or truncated responses. Ouch!
    - dotancohen 2y ago
      Insidious was the right word. Thank you!
hobofan 2y ago
I think the article should be called "How do Go standard library HTTP servers figure out Content-Length?". In most HTTP server implementations from other languages I've worked with I recall having to either: - explicitly define the Content-Length up-front (clients then usually don't like it if you send too little and servers don't like it if you send too much) - have a single "write" operation with an object where the Content-Length can be figured out quite easily - turn on chunking myself and handle the chunk writing myself I don't recall having seen the kind of automatic chunking described in the article before (and I'm not too sure whether I'm a fan of it).
- lifthrasiir 2y ago
  I believe the closest prior art would be PHP. It buffers a response by default until the buffer is full or `flush()` gets called, and will automatically set `Content-Encoding: chunked` if `Content-Length` wasn't explicitly set. Any subsequent writes will be automatically chunked. This approach makes sense from the API standpoint because the caller generally has no idea whether the chunked encoding is necessary, or even its very existence. Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`.
  - nolok 2y ago
    Fun thing about that : core PHP used to and still is very very close to HTTP, to the point where I would say your average decent PHP programmer used to knows more about how HTTP work that your average other similar language where the web library abstract stuff. Eg a PHP dev knows a form has to be multipart/form-data if you send files etc ... But one of the if not THE major exception is this : buffering and flushing works automagically and a lot of PHP dev end up massively blindsinded by it at some point PS: with the rise of modern PHP and it's high quality object based framework, this become less and less true PS2: I am not in ANY way saying anything good or bad or superior or inferior about any dev here, just a difference in approach
aragilar 2y ago
Note that there can be trailer fields (the phrase "trailing header" is both an oxymoron and a good description of it): https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Tr...
- eastbound 2y ago
  Oh yeah, put the Content-Length after the content, when you know its size! /j
- askvictor 2y ago
  Seems as though "Footer" might have been a better term to use given that it's the opposite of a header in typographical terms, and that trailer, in movie-talk, is something that comes before a movie
_ache_ 2y ago
> Anyone who has implemented a simple HTTP server can tell you that it is a really simple protocol It's not. Like, hell no. That is so complex. Multiplexing, underlying TCP specifications, Server Push, Stream prioritization (vs priorization !), encryption (ALPN or NPN ?), extension like HSTS, CORS, WebDav or HLS, ... It's a great protocol, nowhere near simple. > Basically, it’s a text file that has some specific rules to make parsing it easier. Nope, since HTTP/2 that is just a textual representation, not the real "on the wire" protocol. HTTP/2 is 10 now.
- TickleSteve 2y ago
  He was referring to HTTP 1.0 & 1.1
  - lionkor 2y ago
    Which is reasonably simple that you can build a complete 1.0 server in C in an afternoon, and add some 1.1 stuff like keep-alives, content-length, etc. I did that for fun once, on github.com/lionkor/http
    - secondcoming 2y ago
      That’s the easy part. The hard part is working around non-compliant third parties. HTTP is a real mess.
      - lionkor 2y ago
        true
      - ozim 2y ago
        That's pretty much every protocol and working around non-compliant third parties is not specific protocol problem ;)
  - _ache_ 2y ago
    Should have say that so (Nowhere does the article say ‘up to HTTP/1.1’ even talking about HTTP/2 and HTTP/3). HTTP/1.0 is simple. HTTP/1.1 is undoubtedly more complex but manageable. The statement that HTTP is simple is just not true. Even if Go makes it look easy.
deleted 2y ago
[deleted]
lloeki 2y ago
Chunked progress is fun, not many know it supports more than just sending chunk size but can synchronously multiplex information! e.g I drafted this a long time ago, because if you generate something live and send it in a streaming fashion, well you can't have progress reporting since you don't know the final size in bytes, even though server side you know how far you're into generating. This was used for multiple things like generating CSV exports from a bunch of RDBM records, or compressed tarballs from a set of files, or a bunch of other silly things like generating sequences (Fibonacci, random integers, whatever...), that could take "a while" (as in, enough to be friendly and report progress). https://github.com/lloeki/http-chunked-progress/blob/master/draft-lnageleisen-http-chunked-progress-00 https://github.com/lloeki/http-chunked-progress/blob/master/...
- oefrha 2y ago
  I once wrote a terminal-style interactive web app as a single chunked HTML page, because I couldn’t be bothered to implement a websocket endpoint. HTML and inline JS are interweaved and sent in reaction to user actions on the page. The only problem is browsers think the page never finishes loading.
  - eastbound 2y ago
    Couldn’t you put the initial page in its own separate html, and load the rest as a long-running JS file using AJAX?
    - oefrha 2y ago
      Sure, but browsers don't do streaming execution of JavaScript (need to download the entire script), so you then need to manually stream the response and do HTML patching / eval chunked JS, as opposed to the browser trivially loading HTML / executing inline JS as the HTML page is streamed in all by itself. It does solve the loading spinner problem.
      - lloeki 2y ago
        Ha I'm curious now, did useragents eventually time out due to lack of interaction? If so, were you sending NOOP keepalives?
pknerd 2y ago
Why would someone implement the chunk logic when websockets are here? Am I missing something? What are the use cases?
- blueflow 2y ago
  chunked-encoding is a method of encoding an HTTP response body. The semantics for HTTP responses still apply, caching, compression, etc. Websocket is a different protocol that is started up via HTTP.
- rsynnott 2y ago
  HTTP/1.1 came out in 1997. It’s extremely well supported. Websockets were only standardised in 2011, and still have proxy traversal issues. You can absolutely assume that http 1.1 will work on basically anything; websockets are more finicky even now, and certainly were back in the day.
  - masklinn 2y ago
    Websockets are also on the far side of useless when it comes to streaming content the user is downloading. Javascript-filled pages are not the only clients of http.
    - dylan604 2y ago
      > Javascript-filled pages are not the only clients of http. Whaaaaa??? We should eliminate these non-JS filled nonsense immediately!
    - badmintonbaseba 2y ago
      It wouldn't be so bad, if web and application APIs made stream processing of messages possible. The protocol itself could handle streaming content just fine, or at least not worse than HTTP.
- mannyv 2y ago
  Web sockets have their own issues that can be/are implementation dependent. For example, some websocket servers don't pass back errors to the client (AWS). That makes it quite difficult to, say, retry on the client side. Chunked encoding is used by video players - so you can request X bytes of a video file. That means you don't have to download the whole file, and if the user closes the video you didn't waste bandwidth. There are likely more uses of it.
flohofwoe 2y ago
Unfortunately the article doesn't mention compression, because this is where it gets really ugly (especially with range requests), because IIRC the content-size reported in http responses and the range defined in range requests are on the compressed data, but at least in browsers you only get the uncompressed data back and don't even have access to the compressed data.
- meindnoch 2y ago
  +1 This is why in 2024 you still must use XmlHttpRequest instead of fetch() when progress reporting is needed. fetch() cannot do progress reporting on compressed streams.
  - shakna 2y ago
    Once the header is read, you can iterate over the ReadableStream, though, can't you?
    - meindnoch 2y ago
      1. You know the size of the compressed data from the Content-Length header. 2. You can iterate through the uncompressed response bytes with a ReadableStream. Please explain how would you produce a progress percentage from these?
      - lmz 2y ago
        If you had control of both ends you could embed a header in the uncompressed data with the number of uncompressed bytes.
        mananaysiempre 2y ago
        Or put that length in the response headers.
        2y ago
jaffathecake 2y ago
The results might be totally different now, but back in 2014 I looked at how browsers behave if the resource is different to the content-length https://github.com/w3c/ServiceWorker/issues/362#issuecomment-49011736 https://github.com/w3c/ServiceWorker/issues/362#issuecomment... Also in 2018, some fun where when downloading a file, browsers report bytes written to disk vs content-length, which is wildly out when you factor in gzip https://x.com/jaffathecake/status/996720156905820160 https://x.com/jaffathecake/status/996720156905820160
Am4TIfIsER0ppos 2y ago
stat()?
TZubiri 2y ago
len(response)
simonjgreen 2y ago
Along this theme of knowledge, there is the lost art of tuning your page and content sizes such that they fit in as few packets as possible to speed up transmission. The front page of Google for example famously fitted in a single packet (I don't know if that's still the case). There is a brilliant book that used to be a bit of a bible in the world of web sysadmin from the Yahoo Exceptional Performance Team which is less relevant these days but interesting to understand the era. https://www.oreilly.com/library/view/high-performance-web/9780596529307/ https://www.oreilly.com/library/view/high-performance-web/97...
- NelsonMinar 2y ago
  See also the 14KB website article: https://endtimes.dev/why-your-website-should-be-under-14kb-in-size/ https://endtimes.dev/why-your-website-should-be-under-14kb-i... Optimizing per-packet really improves things but has gotten very difficult with SSL and now QUIC. I'm not sure Google ever got the front page down to a single packet (would love a reference!) but it definitely paid very close attention to every byte and details of TCP performance.
  - ryantownsend 2y ago
    iirc, most content delivery networks have now configured initcwnd to be around 40, meaning ~58kb gets sent within the TCP slow start window and therefore 14kb is no longer relevant to most commercial websites (at least with H1/H2, as you mentioned QUIC/H3 uses UDP so it's different)
    - divbzero 2y ago
      When and where you heard that initcwnd is typically 40 for most CDNs? I was curious but the most recent data I could find was from 2017 when there was a mix of CDNs at initcwnd=10 and initcwnd>10: https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/ https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-p... Currently Linux still follows RFC6928 and defaults to initcwnd=10: https://github.com/torvalds/linux/blob/v6.11/include/net/tcp.h#L242-L243 https://github.com/torvalds/linux/blob/v6.11/include/net/tcp...
remon 2y ago
Totally worth an article.
skrebbel 2y ago
I thought I knew basic HTTP 1(.1), but I didn't know about trailers! Nice one, thanks.
- marcosdumay 2y ago
  Half of the middleboxes on the way in the internet don't know about them either.
  - klempner 2y ago
    And browsers don't support them either, at least in any useful manner.
jillesvangurp 2y ago
It's a nice exercise in any web framework to figure out how you would serve a big response without buffering it in memory. This can be surprisingly hard with some frameworks that just assume that you are buffering the entire response in memory. Usually, if you look hard there is a way around this. Buffering can be appropriate for small responses; or at least convenient. But for bigger responses this can be error prone. If you do this right, you serve the first byte of the response to the user before you read the last byte from wherever you are reading (database, file system, S3, etc.). If you do it wrong, you might run out of memory. Or your user's request times out before you are ready to respond. This is a thing that's gotten harder with non-blocking frameworks. Spring Boot in particular can be a PITA on this front if you use it with non-blocking IO. I had some fun figuring that out some years ago. Using Kotlin makes it slightly easier to deal with low level Spring internals (fluxes and what not). Sometimes the right answer is that it's too expensive to figure out the content length, or a content hash. Whatever you do, you need to send the headers with that information before you send anything else. And if you need to read everything before you can calculate that information and send it, your choices are buffering or omitting that information.
- jerf 2y ago
  "This can be surprisingly hard with some frameworks that just assume that you are buffering the entire response in memory. Usually, if you look hard there is a way around this." This is the #1 most common mistake made by a "web framework". Before $YOU jump up with a list of exceptions, it slowly gets better over time, and it has been getting better for a while, and there are many frameworks in the world, so the list that get it right is quite long. But there's still a lot of frameworks out there that assume this, that consider streaming to be the "exception" rather than non-streaming being a special case of streaming, and I still see new people make this mistake with some frequency, so the list of frameworks that still incorporate this mistake into their very core is also quite long. My favorite is when I see a new framework sit on top of something like Go that properly streams, and it actively wrecks the underlying streaming capability to turn an HTTP response into a string. Streaming properly is harder in the short term, but writing a framework where all responses are strings becomes harder in the long term. You eventually hit the wall where that is no longer feasible, but then, fixing it becomes very difficult. Simply not sending a content-length is often the right answer. In an API situation, whatever negative consequences there are are fairly muted. The real problem I encounter a lot is when I'm streaming out some response from some DB query and I encounter a situation that I would have yielded a 500-type response for after I've already streamed out some content. It can be helpful to specify in your API that you may both emit content and an error and users need to check both. For instance, in the common case of dumping JSON, you can spec a top-level {"results": [...], "error": ...} as your return type, stream out a "results", but if a later error occurs, still return an "error" later. Arguably suboptimal, but requiring all errors to be known up front in a streaming situation is impossible, so... suboptimal wins over impossible.
nraynaud 2y ago
I have done crazy stuff to compute the content length of some payloads. For context one of my client works in cloud stuff and I worked in converting hdd format on the fly in a UI VM. The webserver that accepts the files doesn’t do chunked encoding. And there is no space to store the file. So I had to resort to passing over the input file once to transform it, compute its allocation table and transformed size, then throw away everything but the file and the table, restart the scan with the correct header and re-do the transformation.
dicroce 2y ago
At least in the implementation I wrote the default way to provide the body was a string... which has a length. For binary data I believe the API could accept either a std::vector<uint8_t> (which has a size) or a pointer and a size. If you needed chunked transfer encoding you had to ask for it and then make repeated calls to write chunks (that each have a fixed length). To me the more interesting question is how web server receive an incoming request. You want to be able to read the whole thing into a single buffer, but you don't know how long its going to be until you actually read some of it. I learned recently that libc has a way to "peek" at some data without removing it from the recv buffer..... I'm curious if this is ever used to optimize the receive process?
- mikepurvis 2y ago
  Not sure about Linux, but for LwIP on embedded, the buffer isn’t continuous; it’s a linked list of preallocated pbuf objects. So you can either read directly from those if you play by their rules or if you really do need it in contiguous memory you call a function to copy from LwIP’s buffers to one you supply.
AndrewStephens 2y ago
When I worked on a commercial HTTP proxy in the early 2000s, it was very common for servers to return off-by-one values for Content-Length - so much so that we had to implement heuristics to ignore and fix such errors. It may be better now but a huge number of libraries and frameworks would either include the terminating NULL byte in the count but not send it, or not include the terminator in the count but include it in the stream.
matthewaveryusa 2y ago
Next up is how forms with (multiple) attachments are uploaded with Content-Type=multipart/form-data; boundary=$something_unique https://notes.benheater.com/books/web/page/multipart-forms-and-boundary-parameters https://notes.benheater.com/books/web/page/multipart-forms-a...
Sytten 2y ago
There is a whole class of attacks called HTTP Desync Attacks that target just that problem since it is hard to get that right, especially accross multiple different http stacks. And if you dont get it right the result.is that bytes are left on the TCP connections and read as the next request in case of a reuse.
6383353950 2y ago
My account not open plz help me
6383353950 2y ago
Help me sir

271 points by misonic 2y ago

And if you set your own content length header, most http servers will respect it and not chunk. That way, you can stream a 4-gig file that you know the size of per the metadata. This makes downloading nicer because browsers and such will then show a progress bar and time estimate. However, you better be right! I just found a bug in some really old code that was gzipping every response when it was appropriate (ie, asked for, textual, etc). But it was ignoring the content-length header! So, if it was set manually, it would then be wrong after compression. That caused insidious bugs for years. The fix, obviously, was to just delete that manual header if the stream was going to be compressed.

bugtodiffer 2y ago
Did you see if you could turn this into HTTP Request Smuggling? Or something else with security impact? Sounds like a powerful bug you have, potentially.
- knallfrosch 2y ago
  To me it sounds like the server handled the request just fine and reused the header (which was wrong.) The client then had the problem of a wrong response.
  - guappa 2y ago
    If you say to read 1000 bytes from memory and then pass a 900 bytes long array, that's a security bug that can cause crash, corrupt data, and leaking stuff that shouldn't have leaked.
    - jpc0 2y ago
      The size of the buffer and how many bytes are written have nothing intrinsically linked to what the header says. It's a bug sure but does not mean there's any security issue on the server.
      - guappa 2y ago
        It will likely generate corrupt files on the client as well.
        Aachen 2y ago
        Not very. The system might allocate that length ahead of time (I've seen that option in torrent clients and iirc ftp systems) but, latest by the time a FIN comes in, it'll know the file is finished and can truncate it. If finished-early downloads are not implemented despite it doing preallocation, that's still not a security bug
        guappa 2y ago
        if a FIN comes, the client will mark the file as partially downloaded. but it might not come, since decades http sends more than one file per connection, so it might just get the beginning of the next reply, write that and the next reply will be corrupt as well.
    - michaelmior 2y ago
      It's a bug for sure, but I think whether it's a security issue could depend on the language. If the callee is able to determine the length of the array, it can just return an error instead of a potential buffer overrun.
    - stickfigure 2y ago
      In this case, the 1000 bytes aren't being read from memory, they're being read from a socket. If you try to over-read from a socket the worst you'll get is a blocking call or an error (depending what mode you're in).
  - bugtodiffer 2y ago
    maybe response based trickery then? :D What happens to the response after that one, are the first 100 bytes cut of, or what? I'm pretty sure something like this can cause some form of HTTP desync in a loadbalancer/proxy setup.
dotancohen 2y ago
> That caused insidious bugs for years. A lot of people here could probably benefit professionally from hearing about what the bugs were. Knowing what to identify in the future could be really helpful. Thanks.
- pkulak 2y ago
  Well, for years our logs would fill up with these nasty warnings of "connection dropped", something like that. Naturally, you think that's just some badly configured client, mobile connection, something. But then why would it be configured to log that as a warning (or it may have even triggered an error). I think that was because when payloads are small the compression overhead makes them larger, which means the Content-Length is too small and clients would be terminating the connection early. And getting garbage or truncated responses. Ouch!
  - dotancohen 2y ago
    Insidious was the right word. Thank you!

hobofan 2y ago

I think the article should be called "How do Go standard library HTTP servers figure out Content-Length?". In most HTTP server implementations from other languages I've worked with I recall having to either: - explicitly define the Content-Length up-front (clients then usually don't like it if you send too little and servers don't like it if you send too much) - have a single "write" operation with an object where the Content-Length can be figured out quite easily - turn on chunking myself and handle the chunk writing myself I don't recall having seen the kind of automatic chunking described in the article before (and I'm not too sure whether I'm a fan of it).

lifthrasiir 2y ago
I believe the closest prior art would be PHP. It buffers a response by default until the buffer is full or `flush()` gets called, and will automatically set `Content-Encoding: chunked` if `Content-Length` wasn't explicitly set. Any subsequent writes will be automatically chunked. This approach makes sense from the API standpoint because the caller generally has no idea whether the chunked encoding is necessary, or even its very existence. Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`.
- nolok 2y ago
  Fun thing about that : core PHP used to and still is very very close to HTTP, to the point where I would say your average decent PHP programmer used to knows more about how HTTP work that your average other similar language where the web library abstract stuff. Eg a PHP dev knows a form has to be multipart/form-data if you send files etc ... But one of the if not THE major exception is this : buffering and flushing works automagically and a lot of PHP dev end up massively blindsinded by it at some point PS: with the rise of modern PHP and it's high quality object based framework, this become less and less true PS2: I am not in ANY way saying anything good or bad or superior or inferior about any dev here, just a difference in approach

aragilar 2y ago

Note that there can be trailer fields (the phrase "trailing header" is both an oxymoron and a good description of it): https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Tr...

eastbound 2y ago
Oh yeah, put the Content-Length after the content, when you know its size! /j
askvictor 2y ago
Seems as though "Footer" might have been a better term to use given that it's the opposite of a header in typographical terms, and that trailer, in movie-talk, is something that comes before a movie

_ache_ 2y ago

> Anyone who has implemented a simple HTTP server can tell you that it is a really simple protocol It's not. Like, hell no. That is so complex. Multiplexing, underlying TCP specifications, Server Push, Stream prioritization (vs priorization !), encryption (ALPN or NPN ?), extension like HSTS, CORS, WebDav or HLS, ... It's a great protocol, nowhere near simple. > Basically, it’s a text file that has some specific rules to make parsing it easier. Nope, since HTTP/2 that is just a textual representation, not the real "on the wire" protocol. HTTP/2 is 10 now.

TickleSteve 2y ago
He was referring to HTTP 1.0 & 1.1
- lionkor 2y ago
  Which is reasonably simple that you can build a complete 1.0 server in C in an afternoon, and add some 1.1 stuff like keep-alives, content-length, etc. I did that for fun once, on github.com/lionkor/http
  - secondcoming 2y ago
    That’s the easy part. The hard part is working around non-compliant third parties. HTTP is a real mess.
    - lionkor 2y ago
      true
    - ozim 2y ago
      That's pretty much every protocol and working around non-compliant third parties is not specific protocol problem ;)
- _ache_ 2y ago
  Should have say that so (Nowhere does the article say ‘up to HTTP/1.1’ even talking about HTTP/2 and HTTP/3). HTTP/1.0 is simple. HTTP/1.1 is undoubtedly more complex but manageable. The statement that HTTP is simple is just not true. Even if Go makes it look easy.

deleted 2y ago

[deleted]

lloeki 2y ago

Chunked progress is fun, not many know it supports more than just sending chunk size but can synchronously multiplex information! e.g I drafted this a long time ago, because if you generate something live and send it in a streaming fashion, well you can't have progress reporting since you don't know the final size in bytes, even though server side you know how far you're into generating. This was used for multiple things like generating CSV exports from a bunch of RDBM records, or compressed tarballs from a set of files, or a bunch of other silly things like generating sequences (Fibonacci, random integers, whatever...), that could take "a while" (as in, enough to be friendly and report progress). https://github.com/lloeki/http-chunked-progress/blob/master/draft-lnageleisen-http-chunked-progress-00 https://github.com/lloeki/http-chunked-progress/blob/master/...

oefrha 2y ago
I once wrote a terminal-style interactive web app as a single chunked HTML page, because I couldn’t be bothered to implement a websocket endpoint. HTML and inline JS are interweaved and sent in reaction to user actions on the page. The only problem is browsers think the page never finishes loading.
- eastbound 2y ago
  Couldn’t you put the initial page in its own separate html, and load the rest as a long-running JS file using AJAX?
  - oefrha 2y ago
    Sure, but browsers don't do streaming execution of JavaScript (need to download the entire script), so you then need to manually stream the response and do HTML patching / eval chunked JS, as opposed to the browser trivially loading HTML / executing inline JS as the HTML page is streamed in all by itself. It does solve the loading spinner problem.
    - lloeki 2y ago
      Ha I'm curious now, did useragents eventually time out due to lack of interaction? If so, were you sending NOOP keepalives?

pknerd 2y ago

Why would someone implement the chunk logic when websockets are here? Am I missing something? What are the use cases?

blueflow 2y ago
chunked-encoding is a method of encoding an HTTP response body. The semantics for HTTP responses still apply, caching, compression, etc. Websocket is a different protocol that is started up via HTTP.
rsynnott 2y ago
HTTP/1.1 came out in 1997. It’s extremely well supported. Websockets were only standardised in 2011, and still have proxy traversal issues. You can absolutely assume that http 1.1 will work on basically anything; websockets are more finicky even now, and certainly were back in the day.
- masklinn 2y ago
  Websockets are also on the far side of useless when it comes to streaming content the user is downloading. Javascript-filled pages are not the only clients of http.
  - dylan604 2y ago
    > Javascript-filled pages are not the only clients of http. Whaaaaa??? We should eliminate these non-JS filled nonsense immediately!
  - badmintonbaseba 2y ago
    It wouldn't be so bad, if web and application APIs made stream processing of messages possible. The protocol itself could handle streaming content just fine, or at least not worse than HTTP.
mannyv 2y ago
Web sockets have their own issues that can be/are implementation dependent. For example, some websocket servers don't pass back errors to the client (AWS). That makes it quite difficult to, say, retry on the client side. Chunked encoding is used by video players - so you can request X bytes of a video file. That means you don't have to download the whole file, and if the user closes the video you didn't waste bandwidth. There are likely more uses of it.

flohofwoe 2y ago

Unfortunately the article doesn't mention compression, because this is where it gets really ugly (especially with range requests), because IIRC the content-size reported in http responses and the range defined in range requests are on the compressed data, but at least in browsers you only get the uncompressed data back and don't even have access to the compressed data.

meindnoch 2y ago
+1 This is why in 2024 you still must use XmlHttpRequest instead of fetch() when progress reporting is needed. fetch() cannot do progress reporting on compressed streams.
- shakna 2y ago
  Once the header is read, you can iterate over the ReadableStream, though, can't you?
  - meindnoch 2y ago
    1. You know the size of the compressed data from the Content-Length header. 2. You can iterate through the uncompressed response bytes with a ReadableStream. Please explain how would you produce a progress percentage from these?
    - lmz 2y ago
      If you had control of both ends you could embed a header in the uncompressed data with the number of uncompressed bytes.
      - mananaysiempre 2y ago
        Or put that length in the response headers.
        2y ago

jaffathecake 2y ago

The results might be totally different now, but back in 2014 I looked at how browsers behave if the resource is different to the content-length https://github.com/w3c/ServiceWorker/issues/362#issuecomment-49011736 https://github.com/w3c/ServiceWorker/issues/362#issuecomment... Also in 2018, some fun where when downloading a file, browsers report bytes written to disk vs content-length, which is wildly out when you factor in gzip https://x.com/jaffathecake/status/996720156905820160 https://x.com/jaffathecake/status/996720156905820160

Am4TIfIsER0ppos 2y ago

stat()?

TZubiri 2y ago

len(response)

simonjgreen 2y ago

Along this theme of knowledge, there is the lost art of tuning your page and content sizes such that they fit in as few packets as possible to speed up transmission. The front page of Google for example famously fitted in a single packet (I don't know if that's still the case). There is a brilliant book that used to be a bit of a bible in the world of web sysadmin from the Yahoo Exceptional Performance Team which is less relevant these days but interesting to understand the era. https://www.oreilly.com/library/view/high-performance-web/9780596529307/ https://www.oreilly.com/library/view/high-performance-web/97...

NelsonMinar 2y ago
See also the 14KB website article: https://endtimes.dev/why-your-website-should-be-under-14kb-in-size/ https://endtimes.dev/why-your-website-should-be-under-14kb-i... Optimizing per-packet really improves things but has gotten very difficult with SSL and now QUIC. I'm not sure Google ever got the front page down to a single packet (would love a reference!) but it definitely paid very close attention to every byte and details of TCP performance.
- ryantownsend 2y ago
  iirc, most content delivery networks have now configured initcwnd to be around 40, meaning ~58kb gets sent within the TCP slow start window and therefore 14kb is no longer relevant to most commercial websites (at least with H1/H2, as you mentioned QUIC/H3 uses UDP so it's different)
  - divbzero 2y ago
    When and where you heard that initcwnd is typically 40 for most CDNs? I was curious but the most recent data I could find was from 2017 when there was a mix of CDNs at initcwnd=10 and initcwnd>10: https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/ https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-p... Currently Linux still follows RFC6928 and defaults to initcwnd=10: https://github.com/torvalds/linux/blob/v6.11/include/net/tcp.h#L242-L243 https://github.com/torvalds/linux/blob/v6.11/include/net/tcp...

remon 2y ago

Totally worth an article.

skrebbel 2y ago

I thought I knew basic HTTP 1(.1), but I didn't know about trailers! Nice one, thanks.

marcosdumay 2y ago
Half of the middleboxes on the way in the internet don't know about them either.
- klempner 2y ago
  And browsers don't support them either, at least in any useful manner.

jillesvangurp 2y ago

It's a nice exercise in any web framework to figure out how you would serve a big response without buffering it in memory. This can be surprisingly hard with some frameworks that just assume that you are buffering the entire response in memory. Usually, if you look hard there is a way around this. Buffering can be appropriate for small responses; or at least convenient. But for bigger responses this can be error prone. If you do this right, you serve the first byte of the response to the user before you read the last byte from wherever you are reading (database, file system, S3, etc.). If you do it wrong, you might run out of memory. Or your user's request times out before you are ready to respond. This is a thing that's gotten harder with non-blocking frameworks. Spring Boot in particular can be a PITA on this front if you use it with non-blocking IO. I had some fun figuring that out some years ago. Using Kotlin makes it slightly easier to deal with low level Spring internals (fluxes and what not). Sometimes the right answer is that it's too expensive to figure out the content length, or a content hash. Whatever you do, you need to send the headers with that information before you send anything else. And if you need to read everything before you can calculate that information and send it, your choices are buffering or omitting that information.

jerf 2y ago
"This can be surprisingly hard with some frameworks that just assume that you are buffering the entire response in memory. Usually, if you look hard there is a way around this." This is the #1 most common mistake made by a "web framework". Before $YOU jump up with a list of exceptions, it slowly gets better over time, and it has been getting better for a while, and there are many frameworks in the world, so the list that get it right is quite long. But there's still a lot of frameworks out there that assume this, that consider streaming to be the "exception" rather than non-streaming being a special case of streaming, and I still see new people make this mistake with some frequency, so the list of frameworks that still incorporate this mistake into their very core is also quite long. My favorite is when I see a new framework sit on top of something like Go that properly streams, and it actively wrecks the underlying streaming capability to turn an HTTP response into a string. Streaming properly is harder in the short term, but writing a framework where all responses are strings becomes harder in the long term. You eventually hit the wall where that is no longer feasible, but then, fixing it becomes very difficult. Simply not sending a content-length is often the right answer. In an API situation, whatever negative consequences there are are fairly muted. The real problem I encounter a lot is when I'm streaming out some response from some DB query and I encounter a situation that I would have yielded a 500-type response for after I've already streamed out some content. It can be helpful to specify in your API that you may both emit content and an error and users need to check both. For instance, in the common case of dumping JSON, you can spec a top-level {"results": [...], "error": ...} as your return type, stream out a "results", but if a later error occurs, still return an "error" later. Arguably suboptimal, but requiring all errors to be known up front in a streaming situation is impossible, so... suboptimal wins over impossible.

nraynaud 2y ago

I have done crazy stuff to compute the content length of some payloads. For context one of my client works in cloud stuff and I worked in converting hdd format on the fly in a UI VM. The webserver that accepts the files doesn’t do chunked encoding. And there is no space to store the file. So I had to resort to passing over the input file once to transform it, compute its allocation table and transformed size, then throw away everything but the file and the table, restart the scan with the correct header and re-do the transformation.

dicroce 2y ago

At least in the implementation I wrote the default way to provide the body was a string... which has a length. For binary data I believe the API could accept either a std::vector<uint8_t> (which has a size) or a pointer and a size. If you needed chunked transfer encoding you had to ask for it and then make repeated calls to write chunks (that each have a fixed length). To me the more interesting question is how web server receive an incoming request. You want to be able to read the whole thing into a single buffer, but you don't know how long its going to be until you actually read some of it. I learned recently that libc has a way to "peek" at some data without removing it from the recv buffer..... I'm curious if this is ever used to optimize the receive process?

mikepurvis 2y ago
Not sure about Linux, but for LwIP on embedded, the buffer isn’t continuous; it’s a linked list of preallocated pbuf objects. So you can either read directly from those if you play by their rules or if you really do need it in contiguous memory you call a function to copy from LwIP’s buffers to one you supply.

AndrewStephens 2y ago

When I worked on a commercial HTTP proxy in the early 2000s, it was very common for servers to return off-by-one values for Content-Length - so much so that we had to implement heuristics to ignore and fix such errors. It may be better now but a huge number of libraries and frameworks would either include the terminating NULL byte in the count but not send it, or not include the terminator in the count but include it in the stream.

matthewaveryusa 2y ago

Next up is how forms with (multiple) attachments are uploaded with Content-Type=multipart/form-data; boundary=$something_unique https://notes.benheater.com/books/web/page/multipart-forms-and-boundary-parameters https://notes.benheater.com/books/web/page/multipart-forms-a...

Sytten 2y ago

There is a whole class of attacks called HTTP Desync Attacks that target just that problem since it is hard to get that right, especially accross multiple different http stacks. And if you dont get it right the result.is that bytes are left on the TCP connections and read as the next request in case of a reuse.

6383353950 2y ago

My account not open plz help me

6383353950 2y ago

Help me sir