5 ms·

You know, the other day I was lamenting the sorry state that is caching on the web. I decided a pretty good protocol would be similar to this: make a standard h

You know, the other day I was lamenting the sorry state that is caching on the web. I decided a pretty good protocol would be similar to this: make a standard http header that includes a SHA of the content for that url. That way if the content changes the SHA changes, and you get really good cache-ability without the worries of content timeout etc[1]. Further, cache control pragmas/headers/etc don't accidentally get misconfigured in a bad way. Caches just send a header request upstream and if the header hasn't changed, just blast the content on the presumably faster/cheaper local link. [1] They could still exist for hints or what not, but I presume reasonably advanced caches will be able to heuristically figure out how long the content lasts and do things like start sending results while waiting on the header to return... with some sort of reset if it turns out to be a bad cached result.

prodigal_erik 16y ago
Are you describing http://en.wikipedia.org/wiki/HTTP_ETag http://en.wikipedia.org/wiki/HTTP_ETag or something different?
- sophacles 16y ago
  Apparently ETag, but you know, with people actually using it, and a standardized hashing mechanism (to reduce duplication of common elements across sites). (honestly I had never even heard of it before just now).
  - psadauskas 16y ago
    All browsers take advantage of it. Not very many library http clients do, though. (I had to write my own, https://github.com/paul/resourceful https://github.com/paul/resourceful )
  - riffraff 16y ago
    ommon elements are shared by using a single source (e.g. google's CDN for javascripts) but standardizing makes little sense imvho. For one, you can use an ETag without generating the page and hashing it if it only depends on a db-stored resource, and that is mightily application dependent.
    - sophacles 16y ago
      1) What does a CDN have to do with the simple real-world fact that content gets duplicated around sites all the time, particularly stupid crap like funny picture memes? A good global hash of this would certainly help optimize web-caching. 2) A well designed app will hopefully not have every page rendered dynamically at the server anymore, but just have an api to send data back to the page for rendering client side. This benefits from content caching (provided the content is big enough to benefit from caching rather than retransmission). 3. Using something like varnish in front of your site will allow you to dynamically invalidate the hash when needed, and in the interim, a generated result to a url is no different than a static one.
      - riffraff 16y ago
        1) that common stuff reused as-is (such as jquery) can be cached and reused by many, funny picture memes are usually resized, cropped, modified and I frankly do not see many of them, but I load jquery 200 times a day, it seems to me you want to optimize a non-bottleneck. 2) how do you send such content? suppose it's in the db, you need to convert it to a wire format (json or whatever) and then generate an hash for it, but you do not need to, you could just send a revision id as ETag without even reading the full data from the db 3) whatever you want to do with an hash, you can do using an hash as the etag. The latter is simply more generic
        sophacles 16y ago
        1) You are doing browser level cache optimization and bike-shedding, I am looking at it from an ISP level. Once you have a big pipe and many users, things like this actually start to have a noticeable effect on bandwidth usage. Further, no matter what your use case, it has almost no bearing on the normal (statistically normal no digressions please) use case. 2) So I see what you are saying, but you are wrong -- the ETag still requires a lookup of "did content change since then, based on this tag". My system does not preclude a similar mapping of (tag value, last change) in the server. 3) Yeah, I know, my point is that a standardized hashing method for the ETag provides benefits on top of the ETag. Sometimes everyone playing nicely together actually works out better than lots of flexibility.
grimlck 16y ago
GWT's perfect caching is pretty similar to this - Use the MD5 of the file contents as the filename, and send out 'cache forever' headers with the file. Thus, if the file hasn't changed, the browser doesn't even need to make a HTTP request/wait for a 304 response. The MD5 filenames are stored in a 'never cache' bootstrap. http://code.google.com/webtoolkit/doc/latest/DevGuideCompilingAndDebugging.html#perfect_caching http://code.google.com/webtoolkit/doc/latest/DevGuideCompili... I'm surprised i don't see this type of thing more often in other libraries.

prodigal_erik 16y ago

Are you describing http://en.wikipedia.org/wiki/HTTP_ETag http://en.wikipedia.org/wiki/HTTP_ETag or something different?

sophacles 16y ago
Apparently ETag, but you know, with people actually using it, and a standardized hashing mechanism (to reduce duplication of common elements across sites). (honestly I had never even heard of it before just now).
- psadauskas 16y ago
  All browsers take advantage of it. Not very many library http clients do, though. (I had to write my own, https://github.com/paul/resourceful https://github.com/paul/resourceful )
- riffraff 16y ago
  ommon elements are shared by using a single source (e.g. google's CDN for javascripts) but standardizing makes little sense imvho. For one, you can use an ETag without generating the page and hashing it if it only depends on a db-stored resource, and that is mightily application dependent.
  - sophacles 16y ago
    1) What does a CDN have to do with the simple real-world fact that content gets duplicated around sites all the time, particularly stupid crap like funny picture memes? A good global hash of this would certainly help optimize web-caching. 2) A well designed app will hopefully not have every page rendered dynamically at the server anymore, but just have an api to send data back to the page for rendering client side. This benefits from content caching (provided the content is big enough to benefit from caching rather than retransmission). 3. Using something like varnish in front of your site will allow you to dynamically invalidate the hash when needed, and in the interim, a generated result to a url is no different than a static one.
    - riffraff 16y ago
      1) that common stuff reused as-is (such as jquery) can be cached and reused by many, funny picture memes are usually resized, cropped, modified and I frankly do not see many of them, but I load jquery 200 times a day, it seems to me you want to optimize a non-bottleneck. 2) how do you send such content? suppose it's in the db, you need to convert it to a wire format (json or whatever) and then generate an hash for it, but you do not need to, you could just send a revision id as ETag without even reading the full data from the db 3) whatever you want to do with an hash, you can do using an hash as the etag. The latter is simply more generic
      - sophacles 16y ago
        1) You are doing browser level cache optimization and bike-shedding, I am looking at it from an ISP level. Once you have a big pipe and many users, things like this actually start to have a noticeable effect on bandwidth usage. Further, no matter what your use case, it has almost no bearing on the normal (statistically normal no digressions please) use case. 2) So I see what you are saying, but you are wrong -- the ETag still requires a lookup of "did content change since then, based on this tag". My system does not preclude a similar mapping of (tag value, last change) in the server. 3) Yeah, I know, my point is that a standardized hashing method for the ETag provides benefits on top of the ETag. Sometimes everyone playing nicely together actually works out better than lots of flexibility.

grimlck 16y ago

GWT's perfect caching is pretty similar to this - Use the MD5 of the file contents as the filename, and send out 'cache forever' headers with the file. Thus, if the file hasn't changed, the browser doesn't even need to make a HTTP request/wait for a 304 response. The MD5 filenames are stored in a 'never cache' bootstrap. http://code.google.com/webtoolkit/doc/latest/DevGuideCompilingAndDebugging.html#perfect_caching http://code.google.com/webtoolkit/doc/latest/DevGuideCompili... I'm surprised i don't see this type of thing more often in other libraries.