6 ms·

Tell HN: Please Stop Breaking HTTP Clients

124 points by tdavis 17y ago

For whatever ungodly reason, the news.yc web server doesn't terminate header lines properly when sending a response, which happens to break any HTTP client that actually follows the RFC. Please see RFC 2616 section 3.7.1 and use \r\n as God intended. This flexibility regarding line breaks applies only to text media in the entity-body; a bare CR or LF MUST NOT be substituted for CRLF within any of the HTTP control structures (such as header fields and multipart boundaries). Thank you!

aristus 17y ago
Voted up. But a lot of homebrew webservers don't adhere and you'll just have to deal. In this case "God" is W3C, and not every decision they made was good, eg the "Referer" [sic] header. :) [deleted grumble about writing production HTTP clients in 2009]
- tdavis 17y ago
  Neither, I was reading through a Twisted ticket and it was mentioned that this server was breaking the new HTTP client (and why). I then confirmed this ridiculousness myself, and posted it. I wrote one in 2008, though. Okay, most of one. But in my defense, no comparable [python] client existed at the time and I needed the functionality (synchronous requests were out of the question; a couple threaded clients were available but not robust or fast enough). As a general programmer rule, I tend to avoid writing code wherever possible :)
- paulgb 17y ago
  I wonder if anyone has tried to calculate how many bytes of data transfer have been saved from w3c's spelling of the word "referer".
  - aristus 17y ago
    Fewer than that wasted by redundant CRLFs.
- deleted 17y ago
  [deleted]
deleted 17y ago
[deleted]
nir 17y ago
Voted up, as a good example why it's better to build on top of existing software rather than reinvent the wheel (or in this case, Apache)
- blasdel 17y ago
  Apache is probably the worst example you could have possibly picked, it's such a dog.
  - davidw 17y ago
    Eh? Apache is a fine web server which has done a great job over the last 10+ years for all kinds of people and businesses.
  - kaens 17y ago
    Could you expand on why you don't like Apache? I'll grant you that it can be confusing to get used to configuring, especially if you're not really familiar with how the web works, but as far as I can tell it's a pretty damn good server. What do you prefer over apache?
    - davidw 17y ago
      "select" based web servers and their modern (poll/kevent/whatever) brethren, for serving purely static content, are faster and smaller. But that's always been the case.
    - ErrantX 17y ago
      I find lighttpd definitely superior :) (not disagreeing that Apache is a good choice too - just answering your question :D)
    - blasdel 17y ago
      My fundamental beef with Apache is that people wrap way too much functionality into it, to the point where their entire end-to-end web stack is two sets of processes: Apache and a database server. On top of that it's configuration is awful, the modules are all awful compared to their domain-specific alternatives, the architecture is awful (especially wrt concurrency), the development process is molasses, and the Apache Software Foundation has all but abandoned httpd for IBM-focused all-Java astronaut architecture with as much bureaucracy as they can possibly fit into a public process. I hate Apache because I'm intimately familiar with how the web works, and the ASF is responsible for so much web-hostile WS-* garbage. For a while (~4 years ago) I thought lighttpd was worthwhile as a total replacement, and did a project that hacked it's WebDAV implementation for userspace filesystems (pre-MacFUSE), but fundamentally its architecture is still Apache just with sensible concurrency. I've come to be really fond of the reverse-proxy model, and really like Nginx running in front of independent app processes using whatever HTTP abstraction is native to the language (WSGI, Rack, Servlets, etc.) along with a nice native high-level spec-focused HTTP server (twisted.web, mongrel, ???). The last web application I wrote from scratch was on Google AppEngine, and I really like their version of WSGI.
pufuwozu 17y ago
I had the same problem a while ago and ended up writing a patch to make PHP's HTTP library more flexible: http://pecl.php.net/bugs/bug.php?id=15223 http://pecl.php.net/bugs/bug.php?id=15223
coderrr 17y ago
I had to make changes to ruby's em-http-request for this http://github.com/coderrr/em-http-request/commit/8e6444fe4727f05a7c4e1efc2cc39555a2ae2338 http://github.com/coderrr/em-http-request/commit/8e6444fe472...
ivank 17y ago
This was (probably) fixed in Anarki a while ago: http://github.com/nex3/arc/commit/6cb43b3a5977950a61bfd6ce5a0af6f18f1df558 http://github.com/nex3/arc/commit/6cb43b3a5977950a61bfd6ce5a...
visitor4rmindia 17y ago
Maybe this would help? Section 19.3 The line terminator for message-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR.
- tdavis 17y ago
  A recommendation to applications is hardly as noteworthy as a written spec. Most clients (rightly) go by the spec, as you can see in this thread (PHP and Ruby have been mentioned; Python's urllib.urlopen also fails to properly parse the CR out of headers). So, no, that doesn't help the fact that the current production server violates the spec and breaks most mainstream http clients in the process, and does so with any oversight that is trivial to fix (which should never have made it into production and indeed already has been fixed in the repo).
- blasdel 17y ago
  That guidance is for how best to "be liberal in what you accept" news.yc's silly problem is in the area of "be conservative in what you emit"
herf 17y ago
The rather good Fiddler debugging proxy (http://www.fiddler2.com/fiddler2/ http://www.fiddler2.com/fiddler2/) always flags HN for this. In a popup, no less. So it drives me nuts too, and if it could be fixed, I could click on fewer popups.
davidw 17y ago
I wonder if this is the reason I get a proxy error when I try and browse HN from my mobile phone.
jacquesm 17y ago
So, you're crawling HN and you expect the target of your crawl to fix something so you have an easier time of it :) ?
- kaens 17y ago
  I would think that HN wouldn't really care about someone crawling them as long as they were respectful about it. Also, as far as writing crawlers/scrapers goes, a header-handling error is one of the least annoying things you can run into.
- tdavis 17y ago
  I'm not doing anything with HN. I expect the server to emit the proper line breaks because that's what it should do, especially if it's going to be made freely available and used by others. There's no reason that such a simple bug should have made it into or remained in the production source for so long. It's laziness for laziness' sake.
stefano 17y ago
It broke my http client too (written in Arc, btw) some time ago: http://arclanguage.org/item?id=8283 http://arclanguage.org/item?id=8283

124 points by tdavis 17y ago

aristus 17y ago

Voted up. But a lot of homebrew webservers don't adhere and you'll just have to deal. In this case "God" is W3C, and not every decision they made was good, eg the "Referer" [sic] header. :) [deleted grumble about writing production HTTP clients in 2009]

tdavis 17y ago
Neither, I was reading through a Twisted ticket and it was mentioned that this server was breaking the new HTTP client (and why). I then confirmed this ridiculousness myself, and posted it. I wrote one in 2008, though. Okay, most of one. But in my defense, no comparable [python] client existed at the time and I needed the functionality (synchronous requests were out of the question; a couple threaded clients were available but not robust or fast enough). As a general programmer rule, I tend to avoid writing code wherever possible :)
paulgb 17y ago
I wonder if anyone has tried to calculate how many bytes of data transfer have been saved from w3c's spelling of the word "referer".
- aristus 17y ago
  Fewer than that wasted by redundant CRLFs.
deleted 17y ago
[deleted]

deleted 17y ago

[deleted]

nir 17y ago

Voted up, as a good example why it's better to build on top of existing software rather than reinvent the wheel (or in this case, Apache)

blasdel 17y ago
Apache is probably the worst example you could have possibly picked, it's such a dog.
- davidw 17y ago
  Eh? Apache is a fine web server which has done a great job over the last 10+ years for all kinds of people and businesses.
- kaens 17y ago
  Could you expand on why you don't like Apache? I'll grant you that it can be confusing to get used to configuring, especially if you're not really familiar with how the web works, but as far as I can tell it's a pretty damn good server. What do you prefer over apache?
  - davidw 17y ago
    "select" based web servers and their modern (poll/kevent/whatever) brethren, for serving purely static content, are faster and smaller. But that's always been the case.
  - ErrantX 17y ago
    I find lighttpd definitely superior :) (not disagreeing that Apache is a good choice too - just answering your question :D)
  - blasdel 17y ago
    My fundamental beef with Apache is that people wrap way too much functionality into it, to the point where their entire end-to-end web stack is two sets of processes: Apache and a database server. On top of that it's configuration is awful, the modules are all awful compared to their domain-specific alternatives, the architecture is awful (especially wrt concurrency), the development process is molasses, and the Apache Software Foundation has all but abandoned httpd for IBM-focused all-Java astronaut architecture with as much bureaucracy as they can possibly fit into a public process. I hate Apache because I'm intimately familiar with how the web works, and the ASF is responsible for so much web-hostile WS-* garbage. For a while (~4 years ago) I thought lighttpd was worthwhile as a total replacement, and did a project that hacked it's WebDAV implementation for userspace filesystems (pre-MacFUSE), but fundamentally its architecture is still Apache just with sensible concurrency. I've come to be really fond of the reverse-proxy model, and really like Nginx running in front of independent app processes using whatever HTTP abstraction is native to the language (WSGI, Rack, Servlets, etc.) along with a nice native high-level spec-focused HTTP server (twisted.web, mongrel, ???). The last web application I wrote from scratch was on Google AppEngine, and I really like their version of WSGI.

pufuwozu 17y ago

I had the same problem a while ago and ended up writing a patch to make PHP's HTTP library more flexible: http://pecl.php.net/bugs/bug.php?id=15223 http://pecl.php.net/bugs/bug.php?id=15223

coderrr 17y ago

I had to make changes to ruby's em-http-request for this http://github.com/coderrr/em-http-request/commit/8e6444fe4727f05a7c4e1efc2cc39555a2ae2338 http://github.com/coderrr/em-http-request/commit/8e6444fe472...

ivank 17y ago

This was (probably) fixed in Anarki a while ago: http://github.com/nex3/arc/commit/6cb43b3a5977950a61bfd6ce5a0af6f18f1df558 http://github.com/nex3/arc/commit/6cb43b3a5977950a61bfd6ce5a...

visitor4rmindia 17y ago

Maybe this would help? Section 19.3 The line terminator for message-header fields is the sequence CRLF. However, we recommend that applications, when parsing such headers, recognize a single LF as a line terminator and ignore the leading CR.

tdavis 17y ago
A recommendation to applications is hardly as noteworthy as a written spec. Most clients (rightly) go by the spec, as you can see in this thread (PHP and Ruby have been mentioned; Python's urllib.urlopen also fails to properly parse the CR out of headers). So, no, that doesn't help the fact that the current production server violates the spec and breaks most mainstream http clients in the process, and does so with any oversight that is trivial to fix (which should never have made it into production and indeed already has been fixed in the repo).
blasdel 17y ago
That guidance is for how best to "be liberal in what you accept" news.yc's silly problem is in the area of "be conservative in what you emit"

herf 17y ago

The rather good Fiddler debugging proxy (http://www.fiddler2.com/fiddler2/ http://www.fiddler2.com/fiddler2/) always flags HN for this. In a popup, no less. So it drives me nuts too, and if it could be fixed, I could click on fewer popups.

davidw 17y ago

I wonder if this is the reason I get a proxy error when I try and browse HN from my mobile phone.

jacquesm 17y ago

So, you're crawling HN and you expect the target of your crawl to fix something so you have an easier time of it :) ?

kaens 17y ago
I would think that HN wouldn't really care about someone crawling them as long as they were respectful about it. Also, as far as writing crawlers/scrapers goes, a header-handling error is one of the least annoying things you can run into.
tdavis 17y ago
I'm not doing anything with HN. I expect the server to emit the proper line breaks because that's what it should do, especially if it's going to be made freely available and used by others. There's no reason that such a simple bug should have made it into or remained in the production source for so long. It's laziness for laziness' sake.

stefano 17y ago

It broke my http client too (written in Arc, btw) some time ago: http://arclanguage.org/item?id=8283 http://arclanguage.org/item?id=8283