9 ms·

HTTP/1.1 just got a major update

248 points by treve 12y ago

purephase 12y ago
Forwarded and a permanent re-direct are nice changes. This is a welcome change. It will likely be awhile before widespread adoption, but to see a standard move forward in such a seemingly small, but considerable way is great. Kudos to those involved. I can't imagine it was an easy feat.
gamegoblin 12y ago
And I just finished implementing 2616 in a toy server of mine. Sigh. C'est la vie. I'm glad this new spec apparently resolves a lot of ambiguities. I hated reading 2616 and some specs it depended on (email, URI, etc).
- treve 12y ago
  These specs clarify 2616. 'Major' may have been a poor choice of words, but if you were 2616 compliant, you should be largely compliant with these specs as well.
- profil 12y ago
  Is your toy server open source? I would love to see the source.
  - perbu 12y ago
    Are you lacking OSS HTTP/1.1 servers to study? There are tons.
BorisMelnik 12y ago
I wonder what Googlebot will make of this. There is a lot of debate already amongst search engine developers on which type of redirect to use.
- treve 12y ago
  A crawler like that should typically only do GET requests. The 308 is really mainly useful for HTTP clients doing for example a PUT or POST request on some url, and the server wants the client to repeat that exact request on for example a different server.
- dsl 12y ago
  Search engines largely handle all redirects the same, because they know nobody uses them correctly. If you aren't seeing the behavior you want, you can use Webmaster Tools or metadata to fix it.
__david__ 12y ago
Finally, the spec is crystal clear that message bodies on GET requests are not illegal.
- colanderman 12y ago
  Pardon my ignorance but how are GET payloads useful? Does that not violate REST principles?
  - colanderman 12y ago
    From the horse's mouth; they are not useful: https://groups.yahoo.com/neo/groups/rest-discuss/conversations/topics/9962 https://groups.yahoo.com/neo/groups/rest-discuss/conversatio...
  - wtetzner 12y ago
    I'm not sure how. As long as a GET request doesn't change your resources, it should be fine. The reason I can think of for sending a payload in a GET is if the data you wanted to send in a query string is too large.
    - treve 12y ago
      It's still a bad idea. It has no defined semantics, meaning that servers, clients, proxies and anything else are free to ignore or drop it. It also defeats caching and any other reasons why you would want to use GET over some other http requests. I would argue that the only reason left why you would use GET for that, is because it's aesthetically pleasing.
      - xxs 12y ago
        My thoughts exactly - GET with payload virtually means no caching.
        masklinn 12y ago
        If you've got to use body payloads, requests are probably way too big and unique for the involvement of caches browser or intermediate caches to be a good idea.
derengel 12y ago
As someone new to HTTP, what would be the most pragmatic way to read through this RFCs in the context of building web applications or HTTP APIs but not to the level of wanting to implement a http server or http client? for example: order of reading, what can be avoided, what is not widely use or implemented, the basics of the protocol, etc...
- treve 12y ago
  On a high level: You can certainly skip most of "Message Syntax and Routing". That's the stuff that concerns server and client implementers that just have tcp sockets to work with. I would absolutely read "Semantics and Content". It's a really good idea to be aware of "Conditional Requests", and you only really have to read "Caching", "Range requests" and "Authentication" if you need to know about those features.
- muaddirac 12y ago
  HTTP methods, status codes, and headers are all you need to understand for developing at the level of HTTP APIs.
  - jalfresi 12y ago
    And media-types and rel types if you're developing HTTP APIs
  - masklinn 12y ago
    The problem being that it's been spread all over the specs. The essentials would be: * 7231 which covers core methods, statuses and basic headers. It looks like the spec authors have also added security considerations sections * 7232 is probably a good read as it covers conditional requests (304 and 412 statuses) * 7234 covers caching and cache controls, don't skip it. Even if you don't want your response to be cached, you need to know how caching works, which actors are involved and how to disable it * 7238 is the 308 redirection, understanding it and the background for its introduction is a good idea and will help with understanding other redirection statuses (301, 302 and 307) "Various others" would be * 7233 is Range requests, can probably be skipped unless you have big media payloads. On one hand it's underused, on the other hand it has limited general applicability * 7235 is Authentication, can be useful for API (the user experience being terrible for browsers) but can probably be skipped unless absolutely necessary * 7239 is forwarding, to understand what happens when your HTTP endpoint is behind a proxy. Although I'd guess proxies don't implement it yet the ideas already existed as non-standard extensions and reading this is a good idea for "real-world" concerns. Not completely necessary, but useful * 7240 is the Prefer header. It's a fairly recent and quite advanced addition, probably useful but not utterly necessary You can ignore * 7230 is about req/resp format. The only interesting parts are the URI and Host parts which your HTTP library probably handles for you * 7236 complements 7235 with auth scheme registration for standard auth types. Only read it if you've read 7235 * 7237 is a registry of additional (wrt 7231) methods, mostly from WebDAV
userbinator 12y ago
I don't think splitting it up like that is such a good idea; now, instead of searching through one file, I have to remember that there are several and look through them all, just for one conceptual protocol. (TCP has a similar issue, although most of it is still in 793.) As for the extra verbosity, I'm not sure what to think; while some things may be specified more precisely, standards should also attempt to be concise and to-the-point. Some of the sentences in the new RFCs seem almost parenthetical (e.g. look at the description of GET.)
- masklinn 12y ago
  OTOH, that means I don't have to dive through the minutiae of response message format when I'm just looking for the basic header stuff. All the important concerns (core, caching, conditional requests, auth and forwarding) get their own RFC and are thus easier to skim and search through. Although 308 and Range (and Prefer) also getting their RFCs is a bit weird. Likewise, syntax and routing get RFC 3270 so if you're implementing a client or server the reading experience should be much tighter,
- colanderman 12y ago
  Copy+paste them into one text file?
lloeki 12y ago
The clarifications are very welcome but I wish it included embedded unit-less progress information on chunk encoding without having to rely on a side channel [0] (shameless plug, but any progress — ha ha — on this front would be fine) [0]: https://github.com/lloeki/http-chunked-progress/blob/master/draft-lnageleisen-http-chunked-progress-00 https://github.com/lloeki/http-chunked-progress/blob/master/...
fafner 12y ago
RFC7238 is still marked as experimental.
cyphunk 12y ago
Great writeup. One issue... It's now suggested to use the about:blank uri in the Referer header when no referer exists, to distinguish between "there was no referrer" and "I don't want to send a referrer". For the sake of privacy would it not be better if there was no such distinction. Basically now any privacy conscious services need to add 'about:blank' as the referrer when users do not want to have their behaviour categorised and fingerprinted?
- robryk 12y ago
  If a user doesn't want to send the referrer when there is no referrer, no referrer should be sent. This then allows sites to distinguish between direct traffic from users that don't block referrers and traffic with blocked referrers. I wouldn't expect this to be a significant concern, because the volume of actual direct traffic is not very large.
  - gorhill 12y ago
    > This then allows sites to distinguish between direct traffic from users that don't block referrers and traffic with blocked referrers Any example of benefits for servers to distinguish direct traffic vs. blocked referrers?
    - sergiosgc 12y ago
      When analyzing traffic sources for your site, you could use this to remove noise created by privacy conscious users. For example, if you wish to evaluate the efficiency of a magazine add, today you can't distinguish between ad conversions and privacy conscious users. It'll take a while for clients to be compliant, if they'll ever be, though.
      - gorhill 12y ago
        > if you wish to evaluate the efficiency of a magazine ad Sorry I still don't get it. No referrer or about:blank are both "noise" in such case, I still don't see how the distinction is useful to the server to evaluate efficiency of a particular ad.
0x0 12y ago
Why is it not called HTTP/1.2? Or, how will clients (or servers) tell the difference between a peer implementing RFC 2616 (HTTP/1.1 old version) and RFC 723x (HTTP/1.1 new version)? Could it be that there is so much software hardcoded to look for "HTTP/1.1" that a "HTTP/1.2" string would break them all?
- Lukasa 12y ago
  The argument is that the two should be able to transparently interop together, and specifically that RFC 723[0-5] simply codifies the way HTTP/1.1 already works in the real world. We'll see how well that goes.
  - 0x0 12y ago
    Shouldn't minor revisions of HTTP interop anyways? I.e. a HTTP/1.2 server should be able to talk with a HTTP/1.1 client even if they use different minor versions in their protocol string? So, it "SHOULDN'T" hurt to bump the version to 1.2, plus it would make it easier to identify up to date spec compliance?
- masklinn 12y ago
  Because it does not really change the protocol. It clarifies details and implements spec fixes (e.g. matches the spec better to actual real-world use).
  - zurn 12y ago
    There are significant changes of semantics according to TFA, for example: > Default charset of ISO-8859-1 has been removed > The 204, 404, 405, 414 and 501 status codes are now cachable. > The Location header can now contain relative uri's as well as fragment identifiers
    - gsnedders 12y ago
      All three are things pretty much all HTTP/1.1 clients have done for over a decade. It's incompatible with RFC2616, yes — but not with implementations of "HTTP/1.1".
mjs 12y ago
The thing I'm most surprised by is the change in the default cacheability of 404 responses from uncacheable to cacheable. Though I guess since defaulting to cacheable doesn't mean that responses must be cached, so you can still be compliant with RFC7231 by never caching 404s. RFC2616: http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.4 http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13... "A response received with [a status code other than 200, 203, 206, 300, 301 or 410] MUST NOT be returned in a reply to a subsequent request unless there are cache-control directives or another header(s) that explicitly allow it." RFC7231: http://tools.ietf.org/html/rfc7231#section-6.5.4 http://tools.ietf.org/html/rfc7231#section-6.5.4 "A 404 response is cacheable by default"
mrottenkolber 12y ago
I am still happily speaking HTTP/1.0 [1] :) [1] http://mr.gy/software/httpd0/ http://mr.gy/software/httpd0/

248 points by treve 12y ago

Forwarded and a permanent re-direct are nice changes. This is a welcome change. It will likely be awhile before widespread adoption, but to see a standard move forward in such a seemingly small, but considerable way is great. Kudos to those involved. I can't imagine it was an easy feat.

gamegoblin 12y ago

And I just finished implementing 2616 in a toy server of mine. Sigh. C'est la vie. I'm glad this new spec apparently resolves a lot of ambiguities. I hated reading 2616 and some specs it depended on (email, URI, etc).

treve 12y ago
These specs clarify 2616. 'Major' may have been a poor choice of words, but if you were 2616 compliant, you should be largely compliant with these specs as well.
profil 12y ago
Is your toy server open source? I would love to see the source.
- perbu 12y ago
  Are you lacking OSS HTTP/1.1 servers to study? There are tons.

BorisMelnik 12y ago

I wonder what Googlebot will make of this. There is a lot of debate already amongst search engine developers on which type of redirect to use.

treve 12y ago
A crawler like that should typically only do GET requests. The 308 is really mainly useful for HTTP clients doing for example a PUT or POST request on some url, and the server wants the client to repeat that exact request on for example a different server.
dsl 12y ago
Search engines largely handle all redirects the same, because they know nobody uses them correctly. If you aren't seeing the behavior you want, you can use Webmaster Tools or metadata to fix it.

__david__ 12y ago

Finally, the spec is crystal clear that message bodies on GET requests are not illegal.

colanderman 12y ago
Pardon my ignorance but how are GET payloads useful? Does that not violate REST principles?
- colanderman 12y ago
  From the horse's mouth; they are not useful: https://groups.yahoo.com/neo/groups/rest-discuss/conversations/topics/9962 https://groups.yahoo.com/neo/groups/rest-discuss/conversatio...
- wtetzner 12y ago
  I'm not sure how. As long as a GET request doesn't change your resources, it should be fine. The reason I can think of for sending a payload in a GET is if the data you wanted to send in a query string is too large.
  - treve 12y ago
    It's still a bad idea. It has no defined semantics, meaning that servers, clients, proxies and anything else are free to ignore or drop it. It also defeats caching and any other reasons why you would want to use GET over some other http requests. I would argue that the only reason left why you would use GET for that, is because it's aesthetically pleasing.
    - xxs 12y ago
      My thoughts exactly - GET with payload virtually means no caching.
      - masklinn 12y ago
        If you've got to use body payloads, requests are probably way too big and unique for the involvement of caches browser or intermediate caches to be a good idea.

derengel 12y ago

As someone new to HTTP, what would be the most pragmatic way to read through this RFCs in the context of building web applications or HTTP APIs but not to the level of wanting to implement a http server or http client? for example: order of reading, what can be avoided, what is not widely use or implemented, the basics of the protocol, etc...

treve 12y ago
On a high level: You can certainly skip most of "Message Syntax and Routing". That's the stuff that concerns server and client implementers that just have tcp sockets to work with. I would absolutely read "Semantics and Content". It's a really good idea to be aware of "Conditional Requests", and you only really have to read "Caching", "Range requests" and "Authentication" if you need to know about those features.
muaddirac 12y ago
HTTP methods, status codes, and headers are all you need to understand for developing at the level of HTTP APIs.
- jalfresi 12y ago
  And media-types and rel types if you're developing HTTP APIs
- masklinn 12y ago
  The problem being that it's been spread all over the specs. The essentials would be: * 7231 which covers core methods, statuses and basic headers. It looks like the spec authors have also added security considerations sections * 7232 is probably a good read as it covers conditional requests (304 and 412 statuses) * 7234 covers caching and cache controls, don't skip it. Even if you don't want your response to be cached, you need to know how caching works, which actors are involved and how to disable it * 7238 is the 308 redirection, understanding it and the background for its introduction is a good idea and will help with understanding other redirection statuses (301, 302 and 307) "Various others" would be * 7233 is Range requests, can probably be skipped unless you have big media payloads. On one hand it's underused, on the other hand it has limited general applicability * 7235 is Authentication, can be useful for API (the user experience being terrible for browsers) but can probably be skipped unless absolutely necessary * 7239 is forwarding, to understand what happens when your HTTP endpoint is behind a proxy. Although I'd guess proxies don't implement it yet the ideas already existed as non-standard extensions and reading this is a good idea for "real-world" concerns. Not completely necessary, but useful * 7240 is the Prefer header. It's a fairly recent and quite advanced addition, probably useful but not utterly necessary You can ignore * 7230 is about req/resp format. The only interesting parts are the URI and Host parts which your HTTP library probably handles for you * 7236 complements 7235 with auth scheme registration for standard auth types. Only read it if you've read 7235 * 7237 is a registry of additional (wrt 7231) methods, mostly from WebDAV

userbinator 12y ago

I don't think splitting it up like that is such a good idea; now, instead of searching through one file, I have to remember that there are several and look through them all, just for one conceptual protocol. (TCP has a similar issue, although most of it is still in 793.) As for the extra verbosity, I'm not sure what to think; while some things may be specified more precisely, standards should also attempt to be concise and to-the-point. Some of the sentences in the new RFCs seem almost parenthetical (e.g. look at the description of GET.)

masklinn 12y ago
OTOH, that means I don't have to dive through the minutiae of response message format when I'm just looking for the basic header stuff. All the important concerns (core, caching, conditional requests, auth and forwarding) get their own RFC and are thus easier to skim and search through. Although 308 and Range (and Prefer) also getting their RFCs is a bit weird. Likewise, syntax and routing get RFC 3270 so if you're implementing a client or server the reading experience should be much tighter,
colanderman 12y ago
Copy+paste them into one text file?

lloeki 12y ago

The clarifications are very welcome but I wish it included embedded unit-less progress information on chunk encoding without having to rely on a side channel [0] (shameless plug, but any progress — ha ha — on this front would be fine) [0]: https://github.com/lloeki/http-chunked-progress/blob/master/draft-lnageleisen-http-chunked-progress-00 https://github.com/lloeki/http-chunked-progress/blob/master/...

fafner 12y ago

RFC7238 is still marked as experimental.

cyphunk 12y ago

Great writeup. One issue... It's now suggested to use the about:blank uri in the Referer header when no referer exists, to distinguish between "there was no referrer" and "I don't want to send a referrer". For the sake of privacy would it not be better if there was no such distinction. Basically now any privacy conscious services need to add 'about:blank' as the referrer when users do not want to have their behaviour categorised and fingerprinted?

robryk 12y ago
If a user doesn't want to send the referrer when there is no referrer, no referrer should be sent. This then allows sites to distinguish between direct traffic from users that don't block referrers and traffic with blocked referrers. I wouldn't expect this to be a significant concern, because the volume of actual direct traffic is not very large.
- gorhill 12y ago
  > This then allows sites to distinguish between direct traffic from users that don't block referrers and traffic with blocked referrers Any example of benefits for servers to distinguish direct traffic vs. blocked referrers?
  - sergiosgc 12y ago
    When analyzing traffic sources for your site, you could use this to remove noise created by privacy conscious users. For example, if you wish to evaluate the efficiency of a magazine add, today you can't distinguish between ad conversions and privacy conscious users. It'll take a while for clients to be compliant, if they'll ever be, though.
    - gorhill 12y ago
      > if you wish to evaluate the efficiency of a magazine ad Sorry I still don't get it. No referrer or about:blank are both "noise" in such case, I still don't see how the distinction is useful to the server to evaluate efficiency of a particular ad.

0x0 12y ago

Why is it not called HTTP/1.2? Or, how will clients (or servers) tell the difference between a peer implementing RFC 2616 (HTTP/1.1 old version) and RFC 723x (HTTP/1.1 new version)? Could it be that there is so much software hardcoded to look for "HTTP/1.1" that a "HTTP/1.2" string would break them all?

Lukasa 12y ago
The argument is that the two should be able to transparently interop together, and specifically that RFC 723[0-5] simply codifies the way HTTP/1.1 already works in the real world. We'll see how well that goes.
- 0x0 12y ago
  Shouldn't minor revisions of HTTP interop anyways? I.e. a HTTP/1.2 server should be able to talk with a HTTP/1.1 client even if they use different minor versions in their protocol string? So, it "SHOULDN'T" hurt to bump the version to 1.2, plus it would make it easier to identify up to date spec compliance?
masklinn 12y ago
Because it does not really change the protocol. It clarifies details and implements spec fixes (e.g. matches the spec better to actual real-world use).
- zurn 12y ago
  There are significant changes of semantics according to TFA, for example: > Default charset of ISO-8859-1 has been removed > The 204, 404, 405, 414 and 501 status codes are now cachable. > The Location header can now contain relative uri's as well as fragment identifiers
  - gsnedders 12y ago
    All three are things pretty much all HTTP/1.1 clients have done for over a decade. It's incompatible with RFC2616, yes — but not with implementations of "HTTP/1.1".

mjs 12y ago

The thing I'm most surprised by is the change in the default cacheability of 404 responses from uncacheable to cacheable. Though I guess since defaulting to cacheable doesn't mean that responses must be cached, so you can still be compliant with RFC7231 by never caching 404s. RFC2616: http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.4 http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13... "A response received with [a status code other than 200, 203, 206, 300, 301 or 410] MUST NOT be returned in a reply to a subsequent request unless there are cache-control directives or another header(s) that explicitly allow it." RFC7231: http://tools.ietf.org/html/rfc7231#section-6.5.4 http://tools.ietf.org/html/rfc7231#section-6.5.4 "A 404 response is cacheable by default"

mrottenkolber 12y ago

I am still happily speaking HTTP/1.0 [1] :) [1] http://mr.gy/software/httpd0/ http://mr.gy/software/httpd0/