7 ms·

HTTP for Servers

97 points by dedalus 14y ago

jvehent 14y ago
And this is why we use actual web servers, that are HTTP compliant (it takes years), to handle incoming HTTP requests from the Internet. 90% of the "web framework" I've played with only have limited HTTP support and must not be directly on port 80.
- vhf 14y ago
  What "actual web server" HTTP/1.1 compliant would you recommend, since this article tends to show that no "actual web server" handle it correctly ?
  - pjscott 14y ago
    Leaving aside the pesky little issue of what the spec actually says, in practice Apache or nginx will serve nicely.
luriel 14y ago
HTTP is a truly messy protocol, and from what i have seen the proposals for HTTP2 aren't going to make it much simpler, quite the contrary (as the author of Varnish explained here: http://news.ycombinator.com/item?id=4253538 http://news.ycombinator.com/item?id=4253538 ) and don't address some of its fundamental flaws. I have been thinking for a while about writing down a simple and small subset of HTTP plus some conventions and call it HTTP 0.2, I even got a pre-draft (hah) of what the goals for such a spec would be, maybe i should start to work on it again: http://http02.cat-v.org http://http02.cat-v.org
- jallmann 14y ago
  HTTP has its problems, but implementing it is a walk in the park compared to many other telecom protocols. Try SIP and its dozens of add-on RFCs, or tracing a H.323 call flow, or even making sense of something like Megaco from the ITU/IETF spec.
- ilikeit 14y ago
  Cool idea. Less is more. People have done cool things, e.g., with C by retsricting themselves to a subset of it. It seems counterintuitive but I think reducing the number of features in a spec, a language, etc., actually gives way to _more_ creativity. Indeed, it seems "too much freedom/lack of limitations" in the HTTP spec appears to be the underlying condition that fuels Antill's rant. (And, funnily enough, an article about the "tyranny of choice" serendipitously appears on the HN simulatneous with this one. I didn't notice it until _after_ I typed this comment.)
  - aut0mat0n1c 14y ago
    It's not counter-intuitive at all. What is architecture if not a set of constraints?
    - ilikeit 14y ago
      So how do we convince programmers (architects?) to use languages and programs that are small, which due to lack of features appear to have lots of restraints? (really they don't; but one needs to be creative) How do we produce specs that are not the result of committees and packed with everyone's favorite feature? (and thereby masochistic for any mortal to try to implement) See the 1999 interview with Ken Thompson that luriel posted earlier. It has some great comments about these dynamics. When I say "counterintuitive", I am referring to most programmers, judging by what's said in forums like this one, will _not_ be receptive to perceived restraints. They will believe it limits what they can do. Instead, they will see a langauge where there are multiple ways to do the same thing with the same effectiveness as somehow more flexible, making them more productive. They want hundreds of libraries. They want IDE's. They do not want to write things in Scheme. Some even hate the command line. I say these things about "most" programmers. Not all. (thankfully) Again, Thompson's comments are insightful here. Many programmers are cogs in a corporate wheel. They have little opportunity to be creative. They want to save time and effort in any way possible. But some people, the increasingly rare few, will see things differently. I side with Thompson's preference in that I cannot understand large languages where I have to work from the top down and read a 500 page manual to try to understand all the components. I cannot manage that degree of complexity as easily as I can I can work with something simple and build things from the bottom up. (Alas, the things I can build do not amount to a UNIX kernel! :) So, yeah, it's not counterintuitive to me. It makes perfect sense. But I don't find myself agreeing with most comments about these issues I see in forums. I can't even read StackOverflow. The herd mentality is overbearing.
mbell 14y ago
The best idea from the article is the recommendation of an official unit test suite. I'd love it if every group that builds an API or protocol spec were required to include a unit test suite for the defined behavior prior to finalization of the spec. It would not only make implementations easier but more importantly result in the spec writers running into some of the design problems they tend to miss.
- motter 14y ago
  I think this is similar to what happens in the Java Community Process with TCKs. "One of the three required pieces for each JSR is the Technology Compatibility Kit (TCK). A TCK is a suite of tests, tools and documentation that determines whether or not a product complies with a particular Java™ technology specification. A TCK must be included in a final Java technology release under the direction of its Expert Group." Quoted from http://jcp.org/en/resources/tdk http://jcp.org/en/resources/tdk
jtchang 14y ago
The reality is though that even with how screwed up the spec is HTTP is an enormous success. It got more complex as time went on but in the early days it was fairly simple. Now contrast that with SAML, OAuth2, WS-Security (anyone care to add any others?) and you can see how much of a trainwreck we avoided.
- jmgao 14y ago
  CORBA.
- patio11 14y ago
  Just imagine what would have happened if 301/302 redirects had been replaced with delegation, OpenID style. We might not have an Internet yet.
est 14y ago
This trick is handy to break DPI systems or firewalls (Like China's). I remember few years ago I connect to youtube directly in China using a python http proxy by replacing spaces to \t in http request headers. The GFW let the request pass.
- peterwwillis 14y ago
  Another neat trick: linefeed instead of CRLF. Some access control software that major mobile carriers use to captive-portal their data traffic only match on HTTP requests that use CRLF. If the same HTTP traffic uses linefeeds, it's not matched by the ACLs. A simple HTTP proxy that does s/\r\n$/\n/g would allow one to surf for free, even if their bill hasn't been paid in months. Boy, that HTTP standard sure is handy :D
hiccup 14y ago
This rant is from 2007.
- andrewthornton 14y ago
  2007 is still pretty recent considering the lifespan of 1.1. http://www.w3.org/Protocols/History.html http://www.w3.org/Protocols/History.html
the_mitsuhiko 14y ago
The whitespace rule as stated there is as far as my understanding goes incorrect. Whitespace is allowed between quoted strings and word tokens, not between individual production rules (if that is the term). By that I mean "q = 0 . 1" is invalid, "q = 0.1" however is valid. That simplifies things a lot. If you lex the headers in two phases (handle line continuations and header combination first, feed joined lines to parser) HTTP is reasonably simple to parse.
- d0mine 14y ago
  yes. "Implied LWS" rule says it can be included between any tokens, but tokens MUST be separated by a delimiter therefore if 0.000 is allowed (no delimiters) then it means it is a single token and "implied LWS" rule is not applicable for the 0.000 part in the quality values: * implied LWS The grammar described by this specification is word-based. Except where noted otherwise, linear white space (LWS) can be included between any two adjacent words (token or quoted-string), and between adjacent words and separators, without changing the interpretation of a field. At least one delimiter (LWS and/or separators) MUST exist between any two tokens (for the definition of "token" below), since they would otherwise be interpreted as a single token.* </quote> http://www.ietf.org/rfc/rfc2616.txt http://www.ietf.org/rfc/rfc2616.txt Quality values: qvalue = ( "0" [ "." 0*3DIGIT ] ) | ( "1" [ "." 0*3("0") ] ) It seems the author is mistaken in allowing: 0 . 0 0 0
saurik 14y ago
> This is hidden at the bottom of section 2.1 "Augmented BNF" ... which you're almost guaranteed to skip. I have no sympathy for someone who reads the descriptive parts of the specification, trying to build their implementation from examples, and then ignores the machine-parse-able, normative grammar at the bottom: if you aren't reading the specification, you are not qualified to build a server for it. Seriously: this guy seems to believe that the correct way to implement something is with a massive set of examples and test cases, where you massage your implementation until all of the examples work right and all the tests pass. > Then any sane person can have a quick look through the rfc, impliment what they think is required and when they're finished run the tests to see. No: you don't just "implement what you think is required", you implement the specification. It really isn't that hard to build a parser: starting from scratch, I wrote a parser combinator library followed by an IMAP parser (very intricate grammar) in a few days. In truth, the people at the IETF went through a lot of work to build that grammar, and if you implement that grammar and prove your implementation of the grammar, you have implemented this specification and can move on with your life. To quote Mark Crispin, the guy who designed and is still in charge of the IMAP specification (whose "combined works", in the form of mailing list posts and IETF documents, I've recently been studying to level up my "historical perspective") said just last year: > First and foremost, the Formal Syntax section of RFC 3501 should be your holy book. If any part of RFC 3501 distracts you from the Formal Syntax, ignore it in favor of the Formal Syntax. ... > Whatever you do, DO NOT ATTEMPT TO IMPLEMENT ANY COMMAND OR RESPONSE BY LOOKING AT THE EXAMPLES! You are guarantee to screw up if you use the examples as a model (that means YOU, Microsoft!). Use only the Formal Syntax. The result should indeed look like the examples; and if it doesn't then study the Formal Syntax to understand why. ... > Fifth, keep in mind that pretty much all of RFC 3501 is mandatory to implement. ... http://mailman2.u.washington.edu/pipermail/imap-protocol/2011-June/001471.html http://mailman2.u.washington.edu/pipermail/imap-protocol/201... (edit: I had a ton more complaints here, but I believe that they are irrelevant and overly long.)
- masklinn 14y ago
  > Seriously: this guy seems to believe that the correct way to implement something is with a massive set of examples and test cases, where you massage your implementation until all of the examples work right and all the tests pass. You have no idea how common that utterly bonkers way of thinking is. I have a colleague according to whom every protocol is trivial to implement. Why? "Trace what an existing implementation does, and replace the values. It works"
buro9 14y ago
The quality range between 0 and 1 is because it's a percentage. Think of an image, the HTTP request being for an image. The client can say (in an Accept header), "Give me a lossless .tiff if you have it, but if you don't then I'd like an image in .jpg format that is 80% quality. If you can't do that then I'd prefer a .gif that is equivalent to 50% quality. If you've failed on that... well, screw it, I'll take ASCII art, it's better than nothing.". The quality represents a loss in quality of the supplied resource, and a preference list of the order of acceptable types (in the case of an Accept header) for each step down in quality. It's a bit of an abuse to read quality as "order by" and to treat the percent quality as "range 0-1000". I think that example is actually given in the specs, but I'm fairly sure audio was the resource type used.
SageRaven 14y ago
Has anyone found the software yet? All the source links appear to be dead-ends.
rdrr3 14y ago
Why oh why do people dig up crap from years ago and post it on HN

97 points by dedalus 14y ago

And this is why we use actual web servers, that are HTTP compliant (it takes years), to handle incoming HTTP requests from the Internet. 90% of the "web framework" I've played with only have limited HTTP support and must not be directly on port 80.

vhf 14y ago
What "actual web server" HTTP/1.1 compliant would you recommend, since this article tends to show that no "actual web server" handle it correctly ?
- pjscott 14y ago
  Leaving aside the pesky little issue of what the spec actually says, in practice Apache or nginx will serve nicely.

luriel 14y ago

HTTP is a truly messy protocol, and from what i have seen the proposals for HTTP2 aren't going to make it much simpler, quite the contrary (as the author of Varnish explained here: http://news.ycombinator.com/item?id=4253538 http://news.ycombinator.com/item?id=4253538 ) and don't address some of its fundamental flaws. I have been thinking for a while about writing down a simple and small subset of HTTP plus some conventions and call it HTTP 0.2, I even got a pre-draft (hah) of what the goals for such a spec would be, maybe i should start to work on it again: http://http02.cat-v.org http://http02.cat-v.org

jallmann 14y ago
HTTP has its problems, but implementing it is a walk in the park compared to many other telecom protocols. Try SIP and its dozens of add-on RFCs, or tracing a H.323 call flow, or even making sense of something like Megaco from the ITU/IETF spec.
ilikeit 14y ago
Cool idea. Less is more. People have done cool things, e.g., with C by retsricting themselves to a subset of it. It seems counterintuitive but I think reducing the number of features in a spec, a language, etc., actually gives way to _more_ creativity. Indeed, it seems "too much freedom/lack of limitations" in the HTTP spec appears to be the underlying condition that fuels Antill's rant. (And, funnily enough, an article about the "tyranny of choice" serendipitously appears on the HN simulatneous with this one. I didn't notice it until _after_ I typed this comment.)
- aut0mat0n1c 14y ago
  It's not counter-intuitive at all. What is architecture if not a set of constraints?
  - ilikeit 14y ago
    So how do we convince programmers (architects?) to use languages and programs that are small, which due to lack of features appear to have lots of restraints? (really they don't; but one needs to be creative) How do we produce specs that are not the result of committees and packed with everyone's favorite feature? (and thereby masochistic for any mortal to try to implement) See the 1999 interview with Ken Thompson that luriel posted earlier. It has some great comments about these dynamics. When I say "counterintuitive", I am referring to most programmers, judging by what's said in forums like this one, will _not_ be receptive to perceived restraints. They will believe it limits what they can do. Instead, they will see a langauge where there are multiple ways to do the same thing with the same effectiveness as somehow more flexible, making them more productive. They want hundreds of libraries. They want IDE's. They do not want to write things in Scheme. Some even hate the command line. I say these things about "most" programmers. Not all. (thankfully) Again, Thompson's comments are insightful here. Many programmers are cogs in a corporate wheel. They have little opportunity to be creative. They want to save time and effort in any way possible. But some people, the increasingly rare few, will see things differently. I side with Thompson's preference in that I cannot understand large languages where I have to work from the top down and read a 500 page manual to try to understand all the components. I cannot manage that degree of complexity as easily as I can I can work with something simple and build things from the bottom up. (Alas, the things I can build do not amount to a UNIX kernel! :) So, yeah, it's not counterintuitive to me. It makes perfect sense. But I don't find myself agreeing with most comments about these issues I see in forums. I can't even read StackOverflow. The herd mentality is overbearing.

mbell 14y ago

The best idea from the article is the recommendation of an official unit test suite. I'd love it if every group that builds an API or protocol spec were required to include a unit test suite for the defined behavior prior to finalization of the spec. It would not only make implementations easier but more importantly result in the spec writers running into some of the design problems they tend to miss.

motter 14y ago
I think this is similar to what happens in the Java Community Process with TCKs. "One of the three required pieces for each JSR is the Technology Compatibility Kit (TCK). A TCK is a suite of tests, tools and documentation that determines whether or not a product complies with a particular Java™ technology specification. A TCK must be included in a final Java technology release under the direction of its Expert Group." Quoted from http://jcp.org/en/resources/tdk http://jcp.org/en/resources/tdk

jtchang 14y ago

The reality is though that even with how screwed up the spec is HTTP is an enormous success. It got more complex as time went on but in the early days it was fairly simple. Now contrast that with SAML, OAuth2, WS-Security (anyone care to add any others?) and you can see how much of a trainwreck we avoided.

jmgao 14y ago
CORBA.
patio11 14y ago
Just imagine what would have happened if 301/302 redirects had been replaced with delegation, OpenID style. We might not have an Internet yet.

est 14y ago

This trick is handy to break DPI systems or firewalls (Like China's). I remember few years ago I connect to youtube directly in China using a python http proxy by replacing spaces to \t in http request headers. The GFW let the request pass.

peterwwillis 14y ago
Another neat trick: linefeed instead of CRLF. Some access control software that major mobile carriers use to captive-portal their data traffic only match on HTTP requests that use CRLF. If the same HTTP traffic uses linefeeds, it's not matched by the ACLs. A simple HTTP proxy that does s/\r\n$/\n/g would allow one to surf for free, even if their bill hasn't been paid in months. Boy, that HTTP standard sure is handy :D

hiccup 14y ago

This rant is from 2007.

andrewthornton 14y ago
2007 is still pretty recent considering the lifespan of 1.1. http://www.w3.org/Protocols/History.html http://www.w3.org/Protocols/History.html

the_mitsuhiko 14y ago

The whitespace rule as stated there is as far as my understanding goes incorrect. Whitespace is allowed between quoted strings and word tokens, not between individual production rules (if that is the term). By that I mean "q = 0 . 1" is invalid, "q = 0.1" however is valid. That simplifies things a lot. If you lex the headers in two phases (handle line continuations and header combination first, feed joined lines to parser) HTTP is reasonably simple to parse.

d0mine 14y ago
yes. "Implied LWS" rule says it can be included between any tokens, but tokens MUST be separated by a delimiter therefore if 0.000 is allowed (no delimiters) then it means it is a single token and "implied LWS" rule is not applicable for the 0.000 part in the quality values: * implied LWS The grammar described by this specification is word-based. Except where noted otherwise, linear white space (LWS) can be included between any two adjacent words (token or quoted-string), and between adjacent words and separators, without changing the interpretation of a field. At least one delimiter (LWS and/or separators) MUST exist between any two tokens (for the definition of "token" below), since they would otherwise be interpreted as a single token.* </quote> http://www.ietf.org/rfc/rfc2616.txt http://www.ietf.org/rfc/rfc2616.txt Quality values: qvalue = ( "0" [ "." 0*3DIGIT ] ) | ( "1" [ "." 0*3("0") ] ) It seems the author is mistaken in allowing: 0 . 0 0 0

saurik 14y ago

> This is hidden at the bottom of section 2.1 "Augmented BNF" ... which you're almost guaranteed to skip. I have no sympathy for someone who reads the descriptive parts of the specification, trying to build their implementation from examples, and then ignores the machine-parse-able, normative grammar at the bottom: if you aren't reading the specification, you are not qualified to build a server for it. Seriously: this guy seems to believe that the correct way to implement something is with a massive set of examples and test cases, where you massage your implementation until all of the examples work right and all the tests pass. > Then any sane person can have a quick look through the rfc, impliment what they think is required and when they're finished run the tests to see. No: you don't just "implement what you think is required", you implement the specification. It really isn't that hard to build a parser: starting from scratch, I wrote a parser combinator library followed by an IMAP parser (very intricate grammar) in a few days. In truth, the people at the IETF went through a lot of work to build that grammar, and if you implement that grammar and prove your implementation of the grammar, you have implemented this specification and can move on with your life. To quote Mark Crispin, the guy who designed and is still in charge of the IMAP specification (whose "combined works", in the form of mailing list posts and IETF documents, I've recently been studying to level up my "historical perspective") said just last year: > First and foremost, the Formal Syntax section of RFC 3501 should be your holy book. If any part of RFC 3501 distracts you from the Formal Syntax, ignore it in favor of the Formal Syntax. ... > Whatever you do, DO NOT ATTEMPT TO IMPLEMENT ANY COMMAND OR RESPONSE BY LOOKING AT THE EXAMPLES! You are guarantee to screw up if you use the examples as a model (that means YOU, Microsoft!). Use only the Formal Syntax. The result should indeed look like the examples; and if it doesn't then study the Formal Syntax to understand why. ... > Fifth, keep in mind that pretty much all of RFC 3501 is mandatory to implement. ... http://mailman2.u.washington.edu/pipermail/imap-protocol/2011-June/001471.html http://mailman2.u.washington.edu/pipermail/imap-protocol/201... (edit: I had a ton more complaints here, but I believe that they are irrelevant and overly long.)

masklinn 14y ago
> Seriously: this guy seems to believe that the correct way to implement something is with a massive set of examples and test cases, where you massage your implementation until all of the examples work right and all the tests pass. You have no idea how common that utterly bonkers way of thinking is. I have a colleague according to whom every protocol is trivial to implement. Why? "Trace what an existing implementation does, and replace the values. It works"

buro9 14y ago

The quality range between 0 and 1 is because it's a percentage. Think of an image, the HTTP request being for an image. The client can say (in an Accept header), "Give me a lossless .tiff if you have it, but if you don't then I'd like an image in .jpg format that is 80% quality. If you can't do that then I'd prefer a .gif that is equivalent to 50% quality. If you've failed on that... well, screw it, I'll take ASCII art, it's better than nothing.". The quality represents a loss in quality of the supplied resource, and a preference list of the order of acceptable types (in the case of an Accept header) for each step down in quality. It's a bit of an abuse to read quality as "order by" and to treat the percent quality as "range 0-1000". I think that example is actually given in the specs, but I'm fairly sure audio was the resource type used.

SageRaven 14y ago

Has anyone found the software yet? All the source links appear to be dead-ends.

rdrr3 14y ago

Why oh why do people dig up crap from years ago and post it on HN