4 ms·

I think the article should be called "How do Go standard library HTTP servers figure out Content-Length?". In most HTTP server implementations from other langu

I think the article should be called "How do Go standard library HTTP servers figure out Content-Length?". In most HTTP server implementations from other languages I've worked with I recall having to either: - explicitly define the Content-Length up-front (clients then usually don't like it if you send too little and servers don't like it if you send too much) - have a single "write" operation with an object where the Content-Length can be figured out quite easily - turn on chunking myself and handle the chunk writing myself I don't recall having seen the kind of automatic chunking described in the article before (and I'm not too sure whether I'm a fan of it).

lifthrasiir 2y ago
I believe the closest prior art would be PHP. It buffers a response by default until the buffer is full or `flush()` gets called, and will automatically set `Content-Encoding: chunked` if `Content-Length` wasn't explicitly set. Any subsequent writes will be automatically chunked. This approach makes sense from the API standpoint because the caller generally has no idea whether the chunked encoding is necessary, or even its very existence. Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`.
- nolok 2y ago
  Fun thing about that : core PHP used to and still is very very close to HTTP, to the point where I would say your average decent PHP programmer used to knows more about how HTTP work that your average other similar language where the web library abstract stuff. Eg a PHP dev knows a form has to be multipart/form-data if you send files etc ... But one of the if not THE major exception is this : buffering and flushing works automagically and a lot of PHP dev end up massively blindsinded by it at some point PS: with the rise of modern PHP and it's high quality object based framework, this become less and less true PS2: I am not in ANY way saying anything good or bad or superior or inferior about any dev here, just a difference in approach
  - JodieBenitez 2y ago
    Well... not sure it was that magic. We used to play a lot with the ob_* functions.
    - nolok 2y ago
      Oh I didn't mean it figured stuff out in a smart way, only that it did auto buffering/chuncking/flushing for you in a way to abstract that whole idea from the dev, while other plateform had you care about it (cf above messages). But yeah the moment you ended up wanting to do anything advanced, you were doing your own buffer on top of that anyway, or disabling it and going raw.
      - JodieBenitez 2y ago
        and the joy of "pure php scripts" with closing ?> tags messing with the output when you didn't want it... all in libraries you had no say... I can't say I miss those days ! Or this platform for that matter.
        lifthrasiir 2y ago
        Eventually people started omitting ?> at the very end, which is correct but also unsettling.
        gary_0 2y ago
        It's even recommended in the PHP docs: https://www.php.net/manual/en/language.basic-syntax.phptags.php https://www.php.net/manual/en/language.basic-syntax.phptags....
        earthboundkid 2y ago
        When I started my first job, I was like "Hey, these files don't have a closing ?>!!" and then my boss sighed.
        stefs 2y ago
- lolinder 2y ago
  > Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`. This feels like a completely random swipe at an unrelated feature of a JavaScript framework, and I'm not even sure that it's an accurate swipe. The entire point of Function.length (slight nit: Function.prototype.length is different and is always zero) is to check the arity of the function [0]. There's no "tries to": if your middleware function accepts three arguments then it will have a length of 3. Aside from that, I've also done a bunch of digging and can't find any evidence that they're doing [1]. Do you have a source for the claim that this is what they're doing? [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/length https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... [1] https://github.com/search?q=repo%3Aexpressjs%2Fexpress%20%22.length%22&type=code https://github.com/search?q=repo%3Aexpressjs%2Fexpress%20%22...
  - lifthrasiir 2y ago
    Because we were talking about HTTP server frameworks, it seemed not that problematic to mention one of the most surprising things I've seen in this space. Not necessarily JS bashing, but sorry for that anyway. I'm surprised to see that it's now gone too! The exact commit is [1], which happened before Express.js 4.7, and you can search for the variable name `arity` in any previous versions to see what I was talking. It seems that my memory was slightly off as well, my bad. The correct description would be that older versions of Express.js used to distinguish "error" callbacks from normal router callbacks by their arities, so `(req, res)` and `(req, res, next)` would have been thankfully okay, while any extra argument added by an accident will effectively disable that callback without any indication. It was a very good reason for me to be surprised and annoyed at that time. [1] https://github.com/expressjs/express/commit/76e8bfa1dcb7b29389c55d55d2072491edbccd91 https://github.com/expressjs/express/commit/76e8bfa1dcb7b293...
    - LegionMammal978 2y ago
      Actually, it still uses ≤ 3 vs. = 4 arguments to distinguish between request callbacks and error callbacks. Check out the added lines to lib/router/layer.js in the commit you mention, or the equivalent functions in the current router v2.0.0 package [0]. [0] https://github.com/pillarjs/router/blob/2e7fb67ad1b0c1cd2d9eb35c2244439c5c57891a/lib/layer.js#L103 https://github.com/pillarjs/router/blob/2e7fb67ad1b0c1cd2d9e...
- everforward 2y ago
  I may be alone here, but I don't it find it that absurd (though the implementation may be if JS doesn't support this well, no idea). This would be crazy in languages that actually enforce typing, but function type signature overloading to alter behavior seems semi-common in languages like Python and JS. Entirely unrelated, but the older I get, the more it seems like exposing the things under ".prototype" as parts of the object was probably a mistake. If I'm not mistaken, that is reflection, and it feels like JS reaches for reflection much more often than other languages. I think in part because it's a native part of the object rather than a reflection library, so it feels like less of an anti-pattern.
  - lifthrasiir 2y ago
    To be clear, distinguishing different types based on arity would have been okay if JS was statically typed or `Function` exposed more thorough information about its signature. `Function.prototype.length` is very primitive (it doesn't count any spread argument, partly because it dates back to the first edition of ECMAScript) and there is even no easy way to override it like Python's `@functools.wraps`. JS functions also don't check the number of given arguments at all, which is already much worse compared to Python but anyway, JS programmers like me would have reasonably expected excess arguments to be simply ignored.
    - everforward 2y ago
      > JS functions also don't check the number of given arguments at all I never really thought about this, but it does explain how optional arguments without a default value work in Typescript. How very strange of a language decision. > To be clear, distinguishing different types based on arity would have been okay if JS was statically typed or `Function` exposed more thorough information about its signature. I actually like this less in a system with better typing. I don't personally think it's a good tradeoff to dramatically increase the complexity of the types just to avoid having a separate method to register a chunked handler. It would make more sense to me to have "app.get()" and "app.getChunked()", or some kind of closure that converts a chunked handler to something app.get() will allow, like "app.get(chunked((req, res, next) => {}))". The typing effectively becomes part of the control flow of the application, which is something I tend to prefer avoiding. Data modelling should model the domain, code should implement business logic. Having data modelling impact business logic feels like some kind of recursive anti-pattern, but I'm not quite clever enough to figure out why it makes me feel that way.
deleted 2y ago
[deleted]
ahoka 2y ago
Some frameworks do automatic chunking when you pass a stream as the response body.
dbrueck 2y ago
Chunked transfer encoding can be a pain, but it's a reasonable solution to several problems: when the response is too big to fit into memory, when the response size is unknown by the HTTP library, when the response size is unknown by the caller of the HTTP library, or when the response doesn't have a total size at all (never-ending data stream).
danudey 2y ago
> explicitly define the Content-Length up-front (clients then usually don't like it if you send too little and servers don't like it if you send too much) We had a small router/firewall thing at a previous company that had a web interface, but for some reason its Content-Length header had an off-by-one error. IIRC Chrome handled this okay (once the connection was closed it would display the content) while Firefox would hang waiting for that one extra byte that never came.

lifthrasiir 2y ago

I believe the closest prior art would be PHP. It buffers a response by default until the buffer is full or `flush()` gets called, and will automatically set `Content-Encoding: chunked` if `Content-Length` wasn't explicitly set. Any subsequent writes will be automatically chunked. This approach makes sense from the API standpoint because the caller generally has no idea whether the chunked encoding is necessary, or even its very existence. Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`.

nolok 2y ago
Fun thing about that : core PHP used to and still is very very close to HTTP, to the point where I would say your average decent PHP programmer used to knows more about how HTTP work that your average other similar language where the web library abstract stuff. Eg a PHP dev knows a form has to be multipart/form-data if you send files etc ... But one of the if not THE major exception is this : buffering and flushing works automagically and a lot of PHP dev end up massively blindsinded by it at some point PS: with the rise of modern PHP and it's high quality object based framework, this become less and less true PS2: I am not in ANY way saying anything good or bad or superior or inferior about any dev here, just a difference in approach
- JodieBenitez 2y ago
  Well... not sure it was that magic. We used to play a lot with the ob_* functions.
  - nolok 2y ago
    Oh I didn't mean it figured stuff out in a smart way, only that it did auto buffering/chuncking/flushing for you in a way to abstract that whole idea from the dev, while other plateform had you care about it (cf above messages). But yeah the moment you ended up wanting to do anything advanced, you were doing your own buffer on top of that anyway, or disabling it and going raw.
    - JodieBenitez 2y ago
      and the joy of "pure php scripts" with closing ?> tags messing with the output when you didn't want it... all in libraries you had no say... I can't say I miss those days ! Or this platform for that matter.
      - lifthrasiir 2y ago
        Eventually people started omitting ?> at the very end, which is correct but also unsettling.
        gary_0 2y ago
        It's even recommended in the PHP docs: https://www.php.net/manual/en/language.basic-syntax.phptags.php https://www.php.net/manual/en/language.basic-syntax.phptags....
        earthboundkid 2y ago
        When I started my first job, I was like "Hey, these files don't have a closing ?>!!" and then my boss sighed.
        stefs 2y ago
lolinder 2y ago
> Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`. This feels like a completely random swipe at an unrelated feature of a JavaScript framework, and I'm not even sure that it's an accurate swipe. The entire point of Function.length (slight nit: Function.prototype.length is different and is always zero) is to check the arity of the function [0]. There's no "tries to": if your middleware function accepts three arguments then it will have a length of 3. Aside from that, I've also done a bunch of digging and can't find any evidence that they're doing [1]. Do you have a source for the claim that this is what they're doing? [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/length https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... [1] https://github.com/search?q=repo%3Aexpressjs%2Fexpress%20%22.length%22&type=code https://github.com/search?q=repo%3Aexpressjs%2Fexpress%20%22...
- lifthrasiir 2y ago
  Because we were talking about HTTP server frameworks, it seemed not that problematic to mention one of the most surprising things I've seen in this space. Not necessarily JS bashing, but sorry for that anyway. I'm surprised to see that it's now gone too! The exact commit is [1], which happened before Express.js 4.7, and you can search for the variable name `arity` in any previous versions to see what I was talking. It seems that my memory was slightly off as well, my bad. The correct description would be that older versions of Express.js used to distinguish "error" callbacks from normal router callbacks by their arities, so `(req, res)` and `(req, res, next)` would have been thankfully okay, while any extra argument added by an accident will effectively disable that callback without any indication. It was a very good reason for me to be surprised and annoyed at that time. [1] https://github.com/expressjs/express/commit/76e8bfa1dcb7b29389c55d55d2072491edbccd91 https://github.com/expressjs/express/commit/76e8bfa1dcb7b293...
  - LegionMammal978 2y ago
    Actually, it still uses ≤ 3 vs. = 4 arguments to distinguish between request callbacks and error callbacks. Check out the added lines to lib/router/layer.js in the commit you mention, or the equivalent functions in the current router v2.0.0 package [0]. [0] https://github.com/pillarjs/router/blob/2e7fb67ad1b0c1cd2d9eb35c2244439c5c57891a/lib/layer.js#L103 https://github.com/pillarjs/router/blob/2e7fb67ad1b0c1cd2d9e...
everforward 2y ago
I may be alone here, but I don't it find it that absurd (though the implementation may be if JS doesn't support this well, no idea). This would be crazy in languages that actually enforce typing, but function type signature overloading to alter behavior seems semi-common in languages like Python and JS. Entirely unrelated, but the older I get, the more it seems like exposing the things under ".prototype" as parts of the object was probably a mistake. If I'm not mistaken, that is reflection, and it feels like JS reaches for reflection much more often than other languages. I think in part because it's a native part of the object rather than a reflection library, so it feels like less of an anti-pattern.
- lifthrasiir 2y ago
  To be clear, distinguishing different types based on arity would have been okay if JS was statically typed or `Function` exposed more thorough information about its signature. `Function.prototype.length` is very primitive (it doesn't count any spread argument, partly because it dates back to the first edition of ECMAScript) and there is even no easy way to override it like Python's `@functools.wraps`. JS functions also don't check the number of given arguments at all, which is already much worse compared to Python but anyway, JS programmers like me would have reasonably expected excess arguments to be simply ignored.
  - everforward 2y ago
    > JS functions also don't check the number of given arguments at all I never really thought about this, but it does explain how optional arguments without a default value work in Typescript. How very strange of a language decision. > To be clear, distinguishing different types based on arity would have been okay if JS was statically typed or `Function` exposed more thorough information about its signature. I actually like this less in a system with better typing. I don't personally think it's a good tradeoff to dramatically increase the complexity of the types just to avoid having a separate method to register a chunked handler. It would make more sense to me to have "app.get()" and "app.getChunked()", or some kind of closure that converts a chunked handler to something app.get() will allow, like "app.get(chunked((req, res, next) => {}))". The typing effectively becomes part of the control flow of the application, which is something I tend to prefer avoiding. Data modelling should model the domain, code should implement business logic. Having data modelling impact business logic feels like some kind of recursive anti-pattern, but I'm not quite clever enough to figure out why it makes me feel that way.

deleted 2y ago

[deleted]

ahoka 2y ago

Some frameworks do automatic chunking when you pass a stream as the response body.

dbrueck 2y ago

Chunked transfer encoding can be a pain, but it's a reasonable solution to several problems: when the response is too big to fit into memory, when the response size is unknown by the HTTP library, when the response size is unknown by the caller of the HTTP library, or when the response doesn't have a total size at all (never-ending data stream).

danudey 2y ago

> explicitly define the Content-Length up-front (clients then usually don't like it if you send too little and servers don't like it if you send too much) We had a small router/firewall thing at a previous company that had a web interface, but for some reason its Content-Length header had an off-by-one error. IIRC Chrome handled this okay (once the connection was closed it would display the content) while Firefox would hang waiting for that one extra byte that never came.