9 ms·

HTTP Immutable Responses

150 points by okket 9y ago

lgierth 9y ago
Nice to see this has made it through the standards process -- the `immutable` keyword is tremendously useful for systems that store and provide actual immutable data, e.g. content-addressed distributed systems.
- deleted 9y ago
  [deleted]
cpburns2009 9y ago
I like the idea of this. This would be very useful for versioned resources (e.g., images, JS, and CSS files) because it would eliminate unnecessary requests for them after they're cached. The example of a 1 year cache time seems a little extreme, though. I think a month would be better.
- lgas 9y ago
  Why? What matters when deciding how long to cache immutable content?
  - cpburns2009 9y ago
    Instructing the browser to cache a resource for a whole year just seems wasteful to me. I know it's completely arbitrary but if your browser hasn't used something cached for over a month, you likely don't need it cached anymore. Now, I do realize browsers have their own rules for maintaining their caches and may not even cache something for a year. That just seems like a very long time. For example: Say many websites start adopting immutable, version all of their resources, and ~25% percent of their resources are updated monthly. I could easily see the browser cache growing to an excessive size over a year before things begin being purged from the cache. Excessively long cache times are sloppy. That's why I think it matters.
    - eridius 9y ago
      Just because the server gives it a year timeout doesn't mean browsers must hold onto it for a year! The max-age is just a maximum age, not an exact age. Browsers can of course discard any cached responses that they don't think they're going to need again.
    - bzbarsky 9y ago
      Browser caches can hold on to things longer than the freshness lifetime, and do. They can also evict things before the freshness lifetime is up, and do. The freshness lifetime is one of the things a browser can use to decide how long to keep something in its cache, but not the only one.
- fooey 9y ago
  A 1 year cache time doesn't force the browser to hang on to it for a year, it just lets the browser know that if it's seen that resource in the last year it shouldn't bother getting it again. The browser is still free to dump whatever it wants to reduce its footprint.
  - jrochkind1 9y ago
    Of course the browser/agent still can with a response tagged immutable too. There will never be a standard that _forces_ agents to hold on to cached content regardless of their disk space availability and needs. But the immutable keyword gives an additional clue to the agent about semantics, to inform caching.
meandmycode 9y ago
What does immutable really mean when you only rent a domain name? I think about this occasionally with domains and email addresses, they've become a trusted piece of information, but over time the ownership of that thing changes, does make me wonder about fraud in the future when sizeable companies die off and their domains free up.
- deleted 9y ago
  [deleted]
- __jal 9y ago
  It is no different than, say, incorporating. If you fail to perform certain rituals each year, you can lose your legal status. The odds of this happening any time soon with Coke are slim, similarly with ownership of coke.com, but it could. Nothing is 'immutable' if you look at it hard enough. The Bible's teachings have changed over time. On the right scale, the Earth is a temporary novelty and questions about proton decay become relevant. Tl;dr: set your expiration dates appropriately and you won't have a problem.
  - AgentME 9y ago
    The issue is that an attacker could buy a domain, serve content from it with the cache-control: immutable response, and then later someone else buys the domain, puts their own content on it, but a lot of their users (who had accessed the domain when it was owned by the attacker) have a lot of the attacker's immutable content permanently cached. Even if the second owner follows the advice of setting expiration dates appropriately, nothing helps them.
    - taeric 9y ago
      That ignores attacks on dns and certificate authorities. I could readily see a fun novel about a nation state or isp using this trick to seed malicious code to a lot of public sites. Imagine setting it for index.html on any site with a source that merely bootstraps to the intended site. Along with the malicious code.
carussell 9y ago
It would be nice to have strong resource pinning. I've been in contract situations where coordinating with in-house IT for a new server deployment would have been a massive, go-nowhere headache, while throwing a webapp together on my own and self-hosting would've been easy but a big problem wrt company policy on data export. Resource pinning solves this by bringing us into the realm of "auditable". Strong resource pinning would look something like this RFC, except the browser would refuse to accept new resource deployments without deliberate consent of the client. (Something that can be bypassed with a hard refresh, as in this RFC, is not strong enough.) Other situations I imagine would benefit from this are web crypto and HIPAA compliance.
- pg_is_a_butt 9y ago
  uh... how is "just trust the resources already on the client" more compliant than forcing known good resources to be included with every response? cache poisoning is specifically mentioned in the RFC as a major issue. this is terrible for web crypto and HIPAA compliance. you're all idiots
masterleep 9y ago
It's sad that there's still no good way to do deployment based expiration of assets without horrible hacks like sticking the asset checksum in the URL. I know that none of my assets will ever change unless a deployment occurs, and even then, most of them won't change. HTTP doesn't seem to support this use case well at all.
- toomim 9y ago
  What do you mean by "deployment-based"? You want things to expire each time you "git pull" on the server?
  - jeremiep 9y ago
    Wait you're actually running git on production boxes? Doesn't that mean your entire build toolchain also lives on production? The last company I worked for did that and everything was much slower and fragile than it would've been had we deployed packaged artifacts instead.
    - jrochkind1 9y ago
      Eh, heroku seems to handle it fine.
      - krallja 9y ago
        No, Heroku has a separate build step before it deploys to your dynos.
    - octalmage 9y ago
      Yeah this just doesn't scale, especially if you're pulling from GitHub.
  - masterleep 9y ago
    I'd like to have a way to force If-Modified type checks on cached assets after a deployment, but not if no deployment occurred.
manigandham 9y ago
It would be far better to have user agents handle caching headers correctly instead of creating another configuration option (which will likely suffer from the same implementation problems). cache-control: private with either sliding or concrete expiration time already handles this.
- deleted 9y ago
  [deleted]
- foota 9y ago
  Doesn't that indicate though that the cached resource can't be shared?
  - manigandham 9y ago
    Then use cache-control: public with a sliding or concrete expiration time. We can even set different expirations for middle proxies and end user clients.
- prolurker 9y ago
  cache-control: private doesn't seem to imply that a resource won't ever change and on page refresh the browsers have to check if the resource has been updated, immutable would avoid the 304's responses cascade.
  - manigandham 9y ago
    That's the entire point of the expiration time. Use a 2 year range and it's effectively immutable. No content will stay on device forever anyway and headers can easily be set to a smaller time-frame or must-revalidate if the content owner wants it. Browsers mistakenly continue checking for new copies when they shouldn't within the expiration time. Fixing poor implementations with more standards never works well.
    - prolurker 9y ago
      The problem is that servers are allowed to update their resources at any time without waiting for any specific expiration time. So when a user instructs it's browser to refresh the page, usually expecting to get the most up to date version, the browser has to choose between giving the still valid, but maybe not completely updated, cached version or actually checking if the resource has been updated. Immutable makes it clear that the server won't update the resource in place and will handle updates by generating a new one so the browser can happily avoid checking those resources on page refresh.
glacials 9y ago
This is also known as key-based cache expiration, detailed by DHH here: https://signalvnoise.com/posts/3113-how-key-based-cache-expiration-works https://signalvnoise.com/posts/3113-how-key-based-cache-expi... You never worry about when to expire your cache entries if the key changes every time the item does. It's nice to finally see cache-busting coming out of the woods.
- jrochkind1 9y ago
  You can already use that kind of cache expiration with HTTP of course, and many of us do -- but you can't _tell the user-agent_ or other client that you are using it. The best you can do is set a far-future expires date. Some agents/clients will still do HEAD/If-Modified-Since requests to check if it _really_ changed before your expire date. So this is a new thing, properly called immutable responses, to tell the client that they really can treat this (in various ways) as a completely immutable response.
edmorley 9y ago
For more on the rationale behind this feature, see: https://www.ietf.org/mail-archive/web/httpbisa/current/msg25463.html https://www.ietf.org/mail-archive/web/httpbisa/current/msg25... https://bitsup.blogspot.co.uk/2016/05/cache-control-immutable.html https://bitsup.blogspot.co.uk/2016/05/cache-control-immutabl... Rough summary: > At Facebook, ... we've noticed that despite our nearly infinite expiration dates we see 10-20% of requests (depending on browser) for static resource being conditional revalidation. We believe this happens because UAs perform revalidation of requests if a user refreshes the page. > A user who refreshes their Facebook page isn't looking for new versions of our _javascript_. Really they want updated content from our site. However UAs refresh all subresoruces of a page when the user refreshes a web page. This is designed to serve cases such as a weather site that says <img src="" ... > Without an additional header, web sites are unable to control UA's behavior when the user uses the refresh button. UA's are rightfully hesitant in any solution that alters the long standing semantics of the refresh button (for example, not refreshing subresources).
- simcop2387 9y ago
  I understand their rationale for it, but I don't think it should be implemented. Having something immutable like this allows it to end up being used for tracking purposes. Just add a <script src="trackingcookie.js" /> that calls a function with the cookie and all the sudden there's yet another place to covertly store an ID for tracking a user.
  - joosters 9y ago
    But you're removing HTTP requests, so how can it make tracking easier? Any ID that a company puts in their immutable content can also be put in their normal content. The change doesn't make tracking any easier than it already is.
    - simcop2387 9y ago
      It makes it another place that they can store the tracking cookie and have your browser give it back out to them. Deleting the cookies or other such things wouldn't remove the tracking ID as long as the browser continues to use the cached immutable resource. Similar to how this[1] works by using multiple storage methods. https://github.com/samyk/evercookie https://github.com/samyk/evercookie
DanWaterworth 9y ago
I think a better way to handle this would be to use subresource integrity [1]. Then, if the browser's cached version matches the hash, it can be sure that it doesn't need to do any requests. [1] https://developer.mozilla.org/en-US/docs/Web/Security/ https://developer.mozilla.org/en-US/docs/Web/Security/
Lxr 9y ago
What happens when wrongly configured servers or frameworks with default settings implement this too aggressively? Browser vendors want their product to work as expected when the user hits refresh, so will be forced to either voliolate the standard or show stale content.
- deleted 9y ago
  [deleted]
- jlgaddis 9y ago
  Per foota [0]: Clients SHOULD NOT issue a conditional request during the response's freshness lifetime (e.g., upon a reload) unless explicitly overridden by the user (e.g., a force reload). The server is still going to serve up the resource when requested. This behavior is for the client-side of things (browser). [0]: https://news.ycombinator.com/item?id=15262108 https://news.ycombinator.com/item?id=15262108
IncRnd 9y ago
Section 3. Security Considerations "Clients SHOULD ignore the immutable extension from resources that are not part of an authenticated context such as HTTPS. Authenticated resources are less vulnerable to cache poisoning." This must NOT read SHOULD. It must read MUST! Otherwise, your computer will be subject to an executable planting vulnerability. I'm surprised they don't have a much larger list of security considerations. There are many other issues that can happen.
- jlgaddis 9y ago
  If "an authenticated context such as HTTPS" isn't being used -- regardless of the presence of this extension -- isn't your computer already "subject to an executable planting vulnerability"?
  - IncRnd 9y ago
    Maybe. That isn't exactly true. This adds the ability to persist a new threat. There is a temporal difference. An attacker may wish to plant something today, in a coffee shop, that would execute in a protected environment, tomorrow. Immutability of caching can only help an attacker. Yes, there are other ways for an attacker do this, but there is no reason to add to more ways! That's why the web is in its current state. I see how you put "executable planting vulnerability" in quotes. Sometimes (in the current marketing), these are called APTs, but they have been around forever. e.g. Think of dll planting in Windows and the millions of attacks and three or four new API sets from Microsoft that resulted from that single ability to plant a dll in the search path. This type of persistence can also be called incubation.
jlgaddis 9y ago
I have a feeling this will end up like HSTS. It sounds really great at first, then a bunch of folks will get burned by it (just wait until somebody accidentally sets it for /index.html or whatever), and finally the general recommendation will be to stop using it altogether ("more harm than good"). Forever is such a long time. Besides, aren't there already ways to say "cache this resource for <acceptable timeframe>"?
mjs 9y ago
Chrome is not going to get this: https://bugs.chromium.org/p/chromium/issues/detail?id=611416#c46 https://bugs.chromium.org/p/chromium/issues/detail?id=611416... Instead, when the user attempts a full-page reload, Chrome will revalidate the resource in the URL bar, but not subresources (they will come from the browser cache, if the cache-control header checks out): https://blog.chromium.org/2017/01/reload-reloaded-faster-and-leaner-page_26.html https://blog.chromium.org/2017/01/reload-reloaded-faster-and...
- jbverschoor 9y ago
  Yeah I thought that was the current behaviour, and shift-cmd-r actually invalidates all
deleted 9y ago
[deleted]

150 points by okket 9y ago

Nice to see this has made it through the standards process -- the `immutable` keyword is tremendously useful for systems that store and provide actual immutable data, e.g. content-addressed distributed systems.

deleted 9y ago
[deleted]

cpburns2009 9y ago

I like the idea of this. This would be very useful for versioned resources (e.g., images, JS, and CSS files) because it would eliminate unnecessary requests for them after they're cached. The example of a 1 year cache time seems a little extreme, though. I think a month would be better.

lgas 9y ago
Why? What matters when deciding how long to cache immutable content?
- cpburns2009 9y ago
  Instructing the browser to cache a resource for a whole year just seems wasteful to me. I know it's completely arbitrary but if your browser hasn't used something cached for over a month, you likely don't need it cached anymore. Now, I do realize browsers have their own rules for maintaining their caches and may not even cache something for a year. That just seems like a very long time. For example: Say many websites start adopting immutable, version all of their resources, and ~25% percent of their resources are updated monthly. I could easily see the browser cache growing to an excessive size over a year before things begin being purged from the cache. Excessively long cache times are sloppy. That's why I think it matters.
  - eridius 9y ago
    Just because the server gives it a year timeout doesn't mean browsers must hold onto it for a year! The max-age is just a maximum age, not an exact age. Browsers can of course discard any cached responses that they don't think they're going to need again.
  - bzbarsky 9y ago
    Browser caches can hold on to things longer than the freshness lifetime, and do. They can also evict things before the freshness lifetime is up, and do. The freshness lifetime is one of the things a browser can use to decide how long to keep something in its cache, but not the only one.
fooey 9y ago
A 1 year cache time doesn't force the browser to hang on to it for a year, it just lets the browser know that if it's seen that resource in the last year it shouldn't bother getting it again. The browser is still free to dump whatever it wants to reduce its footprint.
- jrochkind1 9y ago
  Of course the browser/agent still can with a response tagged immutable too. There will never be a standard that _forces_ agents to hold on to cached content regardless of their disk space availability and needs. But the immutable keyword gives an additional clue to the agent about semantics, to inform caching.

meandmycode 9y ago

What does immutable really mean when you only rent a domain name? I think about this occasionally with domains and email addresses, they've become a trusted piece of information, but over time the ownership of that thing changes, does make me wonder about fraud in the future when sizeable companies die off and their domains free up.

deleted 9y ago
[deleted]
__jal 9y ago
It is no different than, say, incorporating. If you fail to perform certain rituals each year, you can lose your legal status. The odds of this happening any time soon with Coke are slim, similarly with ownership of coke.com, but it could. Nothing is 'immutable' if you look at it hard enough. The Bible's teachings have changed over time. On the right scale, the Earth is a temporary novelty and questions about proton decay become relevant. Tl;dr: set your expiration dates appropriately and you won't have a problem.
- AgentME 9y ago
  The issue is that an attacker could buy a domain, serve content from it with the cache-control: immutable response, and then later someone else buys the domain, puts their own content on it, but a lot of their users (who had accessed the domain when it was owned by the attacker) have a lot of the attacker's immutable content permanently cached. Even if the second owner follows the advice of setting expiration dates appropriately, nothing helps them.
  - taeric 9y ago
    That ignores attacks on dns and certificate authorities. I could readily see a fun novel about a nation state or isp using this trick to seed malicious code to a lot of public sites. Imagine setting it for index.html on any site with a source that merely bootstraps to the intended site. Along with the malicious code.

carussell 9y ago

It would be nice to have strong resource pinning. I've been in contract situations where coordinating with in-house IT for a new server deployment would have been a massive, go-nowhere headache, while throwing a webapp together on my own and self-hosting would've been easy but a big problem wrt company policy on data export. Resource pinning solves this by bringing us into the realm of "auditable". Strong resource pinning would look something like this RFC, except the browser would refuse to accept new resource deployments without deliberate consent of the client. (Something that can be bypassed with a hard refresh, as in this RFC, is not strong enough.) Other situations I imagine would benefit from this are web crypto and HIPAA compliance.

pg_is_a_butt 9y ago
uh... how is "just trust the resources already on the client" more compliant than forcing known good resources to be included with every response? cache poisoning is specifically mentioned in the RFC as a major issue. this is terrible for web crypto and HIPAA compliance. you're all idiots

masterleep 9y ago

It's sad that there's still no good way to do deployment based expiration of assets without horrible hacks like sticking the asset checksum in the URL. I know that none of my assets will ever change unless a deployment occurs, and even then, most of them won't change. HTTP doesn't seem to support this use case well at all.

toomim 9y ago
What do you mean by "deployment-based"? You want things to expire each time you "git pull" on the server?
- jeremiep 9y ago
  Wait you're actually running git on production boxes? Doesn't that mean your entire build toolchain also lives on production? The last company I worked for did that and everything was much slower and fragile than it would've been had we deployed packaged artifacts instead.
  - jrochkind1 9y ago
    Eh, heroku seems to handle it fine.
    - krallja 9y ago
      No, Heroku has a separate build step before it deploys to your dynos.
  - octalmage 9y ago
    Yeah this just doesn't scale, especially if you're pulling from GitHub.
- masterleep 9y ago
  I'd like to have a way to force If-Modified type checks on cached assets after a deployment, but not if no deployment occurred.

manigandham 9y ago

It would be far better to have user agents handle caching headers correctly instead of creating another configuration option (which will likely suffer from the same implementation problems). cache-control: private with either sliding or concrete expiration time already handles this.

deleted 9y ago
[deleted]
foota 9y ago
Doesn't that indicate though that the cached resource can't be shared?
- manigandham 9y ago
  Then use cache-control: public with a sliding or concrete expiration time. We can even set different expirations for middle proxies and end user clients.
prolurker 9y ago
cache-control: private doesn't seem to imply that a resource won't ever change and on page refresh the browsers have to check if the resource has been updated, immutable would avoid the 304's responses cascade.
- manigandham 9y ago
  That's the entire point of the expiration time. Use a 2 year range and it's effectively immutable. No content will stay on device forever anyway and headers can easily be set to a smaller time-frame or must-revalidate if the content owner wants it. Browsers mistakenly continue checking for new copies when they shouldn't within the expiration time. Fixing poor implementations with more standards never works well.
  - prolurker 9y ago
    The problem is that servers are allowed to update their resources at any time without waiting for any specific expiration time. So when a user instructs it's browser to refresh the page, usually expecting to get the most up to date version, the browser has to choose between giving the still valid, but maybe not completely updated, cached version or actually checking if the resource has been updated. Immutable makes it clear that the server won't update the resource in place and will handle updates by generating a new one so the browser can happily avoid checking those resources on page refresh.

glacials 9y ago

This is also known as key-based cache expiration, detailed by DHH here: https://signalvnoise.com/posts/3113-how-key-based-cache-expiration-works https://signalvnoise.com/posts/3113-how-key-based-cache-expi... You never worry about when to expire your cache entries if the key changes every time the item does. It's nice to finally see cache-busting coming out of the woods.

jrochkind1 9y ago
You can already use that kind of cache expiration with HTTP of course, and many of us do -- but you can't _tell the user-agent_ or other client that you are using it. The best you can do is set a far-future expires date. Some agents/clients will still do HEAD/If-Modified-Since requests to check if it _really_ changed before your expire date. So this is a new thing, properly called immutable responses, to tell the client that they really can treat this (in various ways) as a completely immutable response.

edmorley 9y ago

For more on the rationale behind this feature, see: https://www.ietf.org/mail-archive/web/httpbisa/current/msg25463.html https://www.ietf.org/mail-archive/web/httpbisa/current/msg25... https://bitsup.blogspot.co.uk/2016/05/cache-control-immutable.html https://bitsup.blogspot.co.uk/2016/05/cache-control-immutabl... Rough summary: > At Facebook, ... we've noticed that despite our nearly infinite expiration dates we see 10-20% of requests (depending on browser) for static resource being conditional revalidation. We believe this happens because UAs perform revalidation of requests if a user refreshes the page. > A user who refreshes their Facebook page isn't looking for new versions of our _javascript_. Really they want updated content from our site. However UAs refresh all subresoruces of a page when the user refreshes a web page. This is designed to serve cases such as a weather site that says <img src="" ... > Without an additional header, web sites are unable to control UA's behavior when the user uses the refresh button. UA's are rightfully hesitant in any solution that alters the long standing semantics of the refresh button (for example, not refreshing subresources).

simcop2387 9y ago
I understand their rationale for it, but I don't think it should be implemented. Having something immutable like this allows it to end up being used for tracking purposes. Just add a <script src="trackingcookie.js" /> that calls a function with the cookie and all the sudden there's yet another place to covertly store an ID for tracking a user.
- joosters 9y ago
  But you're removing HTTP requests, so how can it make tracking easier? Any ID that a company puts in their immutable content can also be put in their normal content. The change doesn't make tracking any easier than it already is.
  - simcop2387 9y ago
    It makes it another place that they can store the tracking cookie and have your browser give it back out to them. Deleting the cookies or other such things wouldn't remove the tracking ID as long as the browser continues to use the cached immutable resource. Similar to how this[1] works by using multiple storage methods. https://github.com/samyk/evercookie https://github.com/samyk/evercookie

DanWaterworth 9y ago

I think a better way to handle this would be to use subresource integrity [1]. Then, if the browser's cached version matches the hash, it can be sure that it doesn't need to do any requests. [1] https://developer.mozilla.org/en-US/docs/Web/Security/ https://developer.mozilla.org/en-US/docs/Web/Security/

Lxr 9y ago

What happens when wrongly configured servers or frameworks with default settings implement this too aggressively? Browser vendors want their product to work as expected when the user hits refresh, so will be forced to either voliolate the standard or show stale content.

deleted 9y ago
[deleted]
jlgaddis 9y ago
Per foota [0]: Clients SHOULD NOT issue a conditional request during the response's freshness lifetime (e.g., upon a reload) unless explicitly overridden by the user (e.g., a force reload). The server is still going to serve up the resource when requested. This behavior is for the client-side of things (browser). [0]: https://news.ycombinator.com/item?id=15262108 https://news.ycombinator.com/item?id=15262108

IncRnd 9y ago

Section 3. Security Considerations "Clients SHOULD ignore the immutable extension from resources that are not part of an authenticated context such as HTTPS. Authenticated resources are less vulnerable to cache poisoning." This must NOT read SHOULD. It must read MUST! Otherwise, your computer will be subject to an executable planting vulnerability. I'm surprised they don't have a much larger list of security considerations. There are many other issues that can happen.

jlgaddis 9y ago
If "an authenticated context such as HTTPS" isn't being used -- regardless of the presence of this extension -- isn't your computer already "subject to an executable planting vulnerability"?
- IncRnd 9y ago
  Maybe. That isn't exactly true. This adds the ability to persist a new threat. There is a temporal difference. An attacker may wish to plant something today, in a coffee shop, that would execute in a protected environment, tomorrow. Immutability of caching can only help an attacker. Yes, there are other ways for an attacker do this, but there is no reason to add to more ways! That's why the web is in its current state. I see how you put "executable planting vulnerability" in quotes. Sometimes (in the current marketing), these are called APTs, but they have been around forever. e.g. Think of dll planting in Windows and the millions of attacks and three or four new API sets from Microsoft that resulted from that single ability to plant a dll in the search path. This type of persistence can also be called incubation.

jlgaddis 9y ago

I have a feeling this will end up like HSTS. It sounds really great at first, then a bunch of folks will get burned by it (just wait until somebody accidentally sets it for /index.html or whatever), and finally the general recommendation will be to stop using it altogether ("more harm than good"). Forever is such a long time. Besides, aren't there already ways to say "cache this resource for <acceptable timeframe>"?

mjs 9y ago

Chrome is not going to get this: https://bugs.chromium.org/p/chromium/issues/detail?id=611416#c46 https://bugs.chromium.org/p/chromium/issues/detail?id=611416... Instead, when the user attempts a full-page reload, Chrome will revalidate the resource in the URL bar, but not subresources (they will come from the browser cache, if the cache-control header checks out): https://blog.chromium.org/2017/01/reload-reloaded-faster-and-leaner-page_26.html https://blog.chromium.org/2017/01/reload-reloaded-faster-and...

jbverschoor 9y ago
Yeah I thought that was the current behaviour, and shift-cmd-r actually invalidates all

deleted 9y ago

[deleted]