9 ms·

ETag and HTTP Caching

152 points by nalgeon 3y ago

JaggedJax 3y ago
How is the sample `calculateETag()` function generating a weak ETag? It looks like it will generate a different hash due to any JSON formatting changes. It seems like generating a weak ETag would take more effort since you'd need to either ensure consistent ordering and formatting of the JSON, or generate the Etag on the content before converting it to a JSON string.
- clessg 3y ago
  That's because it isn't really generating a weak ETag. From the article: > You could make the `calculateETag` function format-agnostic, so the hash stays the same if the JSON format changes but the content does not. The current `calculateETag` implementation is susceptible to format changes, and I kept it that way to keep the code shorter. They seem to agree, a true weak ETag implementation would probably be trickier and require more code :P I'd be fascinated to see how that might work in practice, though.
  - JaggedJax 3y ago
    Not sure how I missed that, thanks.
  - lstamour 3y ago
    I could probably write one pretty quick, under the assumption that we are storing only JS-compatible JSON with no encoding hiccups (JSON sadly isn’t as standard as it appears at first glance…) just hash(JSON.stringify(JSON.parse(fileText))) and you’re done. This assumes the same parse and serialize methods are expected to be used at both ends, that they only normalize formatting and that you don’t have to worry about number representation doing weird things. I wouldn’t actually sort keys as sorting is technically a change in behavior and good browsers today do not re-order object keys for you, though your code can do that, of course. I considered skipping the second JSON serialize, but it makes a buffer out of an object so it’s easy enough to use. One could imagine a more efficient approach would modify the hashing to occur against buffer chunks of the JSON, but intentionally skip the whitespace. This avoids unintentional data serialization but obviously the parsing routine would have to match the recipient exactly to work correctly. And it still assumes you’re receiving oddly formatted but valid JSON, which doesn’t sound like a safe assumption to make. If your JSON varies in format I wouldn’t ever want to assume I’d be able to parse it correctly. I mean, what if a return character slips in by mistake amongst all the newlines?
    - nuphonenuacc 3y ago
      Since the author wrote in go, an approximate equivalent would sort the keys by default (assuming you decode into a map and Marshall that).
    - JaggedJax 3y ago
      I'm kind of thinking now, why bother with generating a weak ETag at all? Unless your backend is doing things that would commonly cause differences in JSON formatting for the same data, this is probably a rare case and not worth the extra effort or processing. Figure it out when you're at a scale that it actually matters, and stick with strong ETags for now. It's good to know about this option for handling in the frontend if a system returns one though.
      - lstamour 3y ago
        Yeah, I’ve never really heard of “weak etags” before in any sort of common usage of the term. Honestly, most people tend to skip etags by embedding hashes in filenames directly, this way you can avoid any bad proxies serving up stale content or dropping headers. It’s rare these days to be an issue given the use of TLS end-to-end encryption, but I’m sure it still occasionally happens. And yes, the more serious approach to possibly poorly formatted JSON is to “normalize it” into the expected format. It’s less about caching and more about ensuring what you serve to your front end is consistent, even if you are liberal in what inputs you can handle. E.g. if someone gives you XML, rather than write a front end that can handle both XML and JSON, pull the data out of both and make your own JSON later.
wirelesspotat 3y ago
[flagged]
- MasterScrat 3y ago
  I’m not saying you’re a GPT4-based bot, but I want to point out this comment really triggered my LLM radar
  - userbinator 3y ago
    I was under the impression that LLMs don't generate grammatical errors often if at all, so "what they needs to know" caught my attention. It did trigger my"empty praise" vacuous spam detector though.
  - randomdata 3y ago
    Point out to who? The GPT4-based bot?
- dilyevsky 3y ago
  It really delves into the topic, would you say?
  - codetrotter 3y ago
    In the blog post that was shared in this Hacker News post they explore themes such as ETag, and leverage diagrams and examples to present a dynamic presentation that elevates understanding for the reader in a compelling manner.
    - Filligree 3y ago
      Thank you for sharing your insights and highlighting the innovative approaches discussed in the blog post, especially in relation to ETag. The utilization of diagrams and real-world examples undoubtedly enhances the reader’s comprehension in a substantive way. We firmly believe that leveraging such dynamic presentations not only facilitates a deeper understanding but also fosters an environment of learning that is both engaging and informative. It’s encouraging to see the community’s positive reception and the valuable discussions that emerge around these key technological concepts.
akira2501 3y ago
The ETag can be _anything_. I have an API that serves "files" from a backend storage system. Whenever files are written a revision number is incremented. This is perfect for a weak validator and so my ETags are also blisfully short and semantically useful, typically: ETag: W/"750" This also means the API can just check the revision number and avoid pulling out and decompressing some of the larger payloads that are stored there and the implementation is absolutely minimal. It's a great standard.
- kevincox 3y ago
  Can't it be a strong validator if the files don't change at all between revisions? Or does the revision number only increment on "significant" revisions?
  - akira2501 3y ago
    The data does not but certain metadata elements might. It probably could be a strong validator anyways for it's use cases, but I made the decision in a hurry.
- choppaface 3y ago
  A great standard but also provides an effective cookie-less mechanism for user tracking e.g. https://levelup.gitconnected.com/no-cookies-no-problem-using-etags-for-user-tracking-3e745544176b https://levelup.gitconnected.com/no-cookies-no-problem-using...
  - LtWorf 3y ago
    That's why tracking requires consent, regardless of the specific hack used to do it.
ajayvk 3y ago
An approach like https://github.com/benbjohnson/hashfs https://github.com/benbjohnson/hashfs allows file names to be updated at runtime to be content hashed. This removes the need for the extra "304 Not Modified" API calls from the client. This content hash based file renaming is usually done using a build step which renames files. For applications where the static file serving and HTTP request processing are done in the same application, this can be done in memory without a build step for file renames. I am using that approach in my project https://github.com/claceio/clace https://github.com/claceio/clace. It removes the need for a build step while making aggressive static file caching possible.
- codetrotter 3y ago
  I use content hashes for some of the images in parts of my site. And I use the IPFS scheme for it, and have the path be under /ipfs/ or some such. And so you could find the same file on IPFS if anyone served it there, as the content hash in the url tells you what to look for. Even though on my side I’ve done this all completely manually, so much so that it’s literally just me calculating the IPFS hash on my machine one time and then having symlinks with those content hashes so that /ipfs/ directory on my sites contains content that is served by their IPFS content hash, even though my server does not run an IPFS node or anything. A very interesting side effect of this is that one time I loaded one of my pages, the web browser actually picked up on the pattern and offered to load those files over actual IPFS!
  - EGreg 3y ago
    Wait what? Browsers now offer to load over IPFS? I was wondering how you can trust an IFPS gateway, does does browser verify the file is legit using some checksum? Maybw subresource integrity supports IPFS content hashing or something? How does it generate cid anyway ? https://docs.ipfs.tech/concepts/content-addressing/#cids-are-not-file-hashes https://docs.ipfs.tech/concepts/content-addressing/#cids-are... How would you use SRI here to verify the cid (and not an additional out-of-band hash) make sure the gateway isn’t returning some crap to, say, inject malicious JS?
    - 3y ago
deleted 3y ago
[deleted]
alganet 3y ago
This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna document it. It means the account is out of balance. — What is this garbage? Please read the spec. This is going to make our API gateways, CDNs, everything go crazy. Can't let you move on with this PR. — Look. I documented it, made an enum with the code, it's clean. I'm an experienced REST developer. — It... it doesn't work like that Bob. Please, read the spec. — Hey, got enough approvals, "412 Account Out Of Balance" it is! It passes the tests. For each dev that knows proper HTTP, there's 10.000 Bobs.
- esafak 3y ago
  Which article would you ask Bob to read to learn the right way to do it?
  - alganet 3y ago
    Bob has since moved on to crypto, leaving the cache invalidation eternally crippled. New Bob decided that everything is useless and wants to rewrite the whole backend using a faster language.
  - mozey 3y ago
    Have to admit I've never used this code, and didn't know what it was about. Quickly read up about it. So ETag is a hash of the resource. You must provide it with requests that modify the resource. If your hash doesn't match the server hash, then 412 Precondition Failed is returned? https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412 https://www.rfc-editor.org/rfc/rfc9110#status.412 https://www.rfc-editor.org/rfc/rfc9110#status.412
    - alganet 3y ago
      You can provide all sorts of conditions using HTTP headers such as "If-Match", "If-None-Match", "If-Modified-Since", "If-Range", etc. The server can chose an HTTP code to indicate some sort of cache invalidation signal. 304 means "you're good, your cached version satisfies the conditions" 412 means "you're not good, your cached version does not satisfy the conditions" 412 usually applies to modifications, but it could be for reading too, in the case of ranged requests (getting a specific range of bytes from a large representation). See the "Range" header. These are all interconected. The headers, the codes, etc. They are very useful for caching and can save a lot of bandwidth. Browsers and CDNs use them extensively. Server-to-server communcation could use them as well, but I haven't seen popular implementations (let's say, a web framework that provides abstraction over these mechanisms).
rollulus 3y ago
I recently implemented this, great write-up. Regarding the hashing function, I’m curious about opinions. In my implementation I went for a cheap but weak cryptographic hash at first. Then I got worried that some auditor would flag it and time would be wasted convincing them to change their mind. But then I stumbled upon FNV [1], a non-cryptographic hash and part of Go’s standard library and went for it. Any thoughts? [1]: https://en.m.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function https://en.m.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93...
- f_devd 3y ago
  I recommend checking out XXHash[0], FNV is simple but not really optimized and relatively low quality (often still good enough). From the readme page: Hash Name, Bandwidth, Small Velocity, Quality, Comment XXH3 (SSE2), 31.5 GB/s, 133.1, 10, FNV64, 1.2 GB/s, 62.7, 5, Poor avalanche properties [0]: https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash Also ETag is exactly the kind of thing non-cryptographic hashes are meant for, but if you can't convince them Blake3 is a very fast modern cryptographic hash function.
- rednafi 3y ago
  I feel like people tend to overthink in this regard. If SHA-256 hashing is good enough for GitHub's REST endpoints, it's good enough for me. If you're implementing weak validation, then you might need to preprocess the payload before running it through the hash function. For example, if your payload is JSON and you want to make it format-agnostic, then you'll need to normalize the payload and then compute the hash. In either case, the hashing algo probably doesn't matter as much.
ulrischa 3y ago
Oh this topic can cost you some years of your life
joosters 3y ago
I've not seen a very convincing use-case for ETags vs Last-Modified date caching. In the example request, the server still has to do all of the work generating the page, in order to calculate the ETag and then determine whether or not the page has changed. In most situations, it's simpler to have timestamps to compare against, because that gives the server a faster way to spot unmodified data. e.g. you get a HTTP request for some data that you know is sourced from a particular file, or a DB table. If the client sends a If-Modified-Since (or whatever the header name is), you have a good chance to be able to check the modified time of the data source before doing any complicated data processing, and are able to send back a not modified response sooner.
- johneth 3y ago
  When I use ETags, I'll generate the ETag value once (on startup), then cache and serve that value. When the resource changes, regenerate the ETag value. Obviously doesn't work if you're using ETags on dynamic resources, but works well for non-dynamic, but unpredictably frequently changing resources.
  - derkades 3y ago
    And couldn't you do the same thing with a timestamp instead of a hash?
- jmull 3y ago
  They are essentially the same thing. Sometimes you have a last modified timestamp handy and sometimes you have a revision number handy. I would agree that you shouldn't be doing signifiant calculations -- like checksumming the contents of a file -- to determine the current etag value. Also, the excessive semantics of timestamps is a negative. It adds complication which sometimes trips developers up but doesn't improve the caching.
nikolayasdf123 3y ago
if only browsers respected this. none of browsers use ETag and If-None-Match mechanism. instead they do their own wizzardy caching...
- demurgos 3y ago
  I was refactoring a project serving user uploaded files yesterday, and had the occasion to test caching. Both Firefox and Chrome used ETag and If-None-Match properly to check and cache queries. Which problems did you encounter? There was still one thing that surprised me a bit (but also makes sense). Images are fetched only once per page load in my testing. If an image with 60sec of cache is loaded, then removed by JS and added back after 2 minutes, then the browser will reuse the image from the initial load.
  - nikolayasdf123 2y ago
    the issue I got was: 1) I (AWS CloudFront) supply ETag and If-None-Match header; I can see that header in responses. 2) browsers sometimes do respect that (once) I see 304 in responses, but 99% of the time they don't include ETag/If-None-Match in requests and thus I never get 304 responses (albeit AWS CloudFront, resource, data — nothing changed) and instead they perform some other caching and reload whole resource again with TTL that does not seem to come from my headers, totally disregarding ETags/If-None-Match logic. for videos it is even worse. unless you set `preload=none` in html, Safari, Firefox, Chrome will have all different policies trying to preload all videos on screen ignoring lazyload html tags. worse of all, caching does not work well, and videos will be attempted to be loaded almost every time and ETags/If-None-Match totally ignored.
    - withinboredom 2y ago
      It sounds like you have caching disabled in your browser, either in dev-tools itself or through an extension. This is NOT normal behavior.
      - nikolayasdf123 2y ago

152 points by nalgeon 3y ago

How is the sample `calculateETag()` function generating a weak ETag? It looks like it will generate a different hash due to any JSON formatting changes. It seems like generating a weak ETag would take more effort since you'd need to either ensure consistent ordering and formatting of the JSON, or generate the Etag on the content before converting it to a JSON string.

clessg 3y ago
That's because it isn't really generating a weak ETag. From the article: > You could make the `calculateETag` function format-agnostic, so the hash stays the same if the JSON format changes but the content does not. The current `calculateETag` implementation is susceptible to format changes, and I kept it that way to keep the code shorter. They seem to agree, a true weak ETag implementation would probably be trickier and require more code :P I'd be fascinated to see how that might work in practice, though.
- JaggedJax 3y ago
  Not sure how I missed that, thanks.
- lstamour 3y ago
  I could probably write one pretty quick, under the assumption that we are storing only JS-compatible JSON with no encoding hiccups (JSON sadly isn’t as standard as it appears at first glance…) just hash(JSON.stringify(JSON.parse(fileText))) and you’re done. This assumes the same parse and serialize methods are expected to be used at both ends, that they only normalize formatting and that you don’t have to worry about number representation doing weird things. I wouldn’t actually sort keys as sorting is technically a change in behavior and good browsers today do not re-order object keys for you, though your code can do that, of course. I considered skipping the second JSON serialize, but it makes a buffer out of an object so it’s easy enough to use. One could imagine a more efficient approach would modify the hashing to occur against buffer chunks of the JSON, but intentionally skip the whitespace. This avoids unintentional data serialization but obviously the parsing routine would have to match the recipient exactly to work correctly. And it still assumes you’re receiving oddly formatted but valid JSON, which doesn’t sound like a safe assumption to make. If your JSON varies in format I wouldn’t ever want to assume I’d be able to parse it correctly. I mean, what if a return character slips in by mistake amongst all the newlines?
  - nuphonenuacc 3y ago
    Since the author wrote in go, an approximate equivalent would sort the keys by default (assuming you decode into a map and Marshall that).
  - JaggedJax 3y ago
    I'm kind of thinking now, why bother with generating a weak ETag at all? Unless your backend is doing things that would commonly cause differences in JSON formatting for the same data, this is probably a rare case and not worth the extra effort or processing. Figure it out when you're at a scale that it actually matters, and stick with strong ETags for now. It's good to know about this option for handling in the frontend if a system returns one though.
    - lstamour 3y ago
      Yeah, I’ve never really heard of “weak etags” before in any sort of common usage of the term. Honestly, most people tend to skip etags by embedding hashes in filenames directly, this way you can avoid any bad proxies serving up stale content or dropping headers. It’s rare these days to be an issue given the use of TLS end-to-end encryption, but I’m sure it still occasionally happens. And yes, the more serious approach to possibly poorly formatted JSON is to “normalize it” into the expected format. It’s less about caching and more about ensuring what you serve to your front end is consistent, even if you are liberal in what inputs you can handle. E.g. if someone gives you XML, rather than write a front end that can handle both XML and JSON, pull the data out of both and make your own JSON later.

wirelesspotat 3y ago

[flagged]

MasterScrat 3y ago
I’m not saying you’re a GPT4-based bot, but I want to point out this comment really triggered my LLM radar
- userbinator 3y ago
  I was under the impression that LLMs don't generate grammatical errors often if at all, so "what they needs to know" caught my attention. It did trigger my"empty praise" vacuous spam detector though.
- randomdata 3y ago
  Point out to who? The GPT4-based bot?
dilyevsky 3y ago
It really delves into the topic, would you say?
- codetrotter 3y ago
  In the blog post that was shared in this Hacker News post they explore themes such as ETag, and leverage diagrams and examples to present a dynamic presentation that elevates understanding for the reader in a compelling manner.
  - Filligree 3y ago
    Thank you for sharing your insights and highlighting the innovative approaches discussed in the blog post, especially in relation to ETag. The utilization of diagrams and real-world examples undoubtedly enhances the reader’s comprehension in a substantive way. We firmly believe that leveraging such dynamic presentations not only facilitates a deeper understanding but also fosters an environment of learning that is both engaging and informative. It’s encouraging to see the community’s positive reception and the valuable discussions that emerge around these key technological concepts.

akira2501 3y ago

The ETag can be _anything_. I have an API that serves "files" from a backend storage system. Whenever files are written a revision number is incremented. This is perfect for a weak validator and so my ETags are also blisfully short and semantically useful, typically: ETag: W/"750" This also means the API can just check the revision number and avoid pulling out and decompressing some of the larger payloads that are stored there and the implementation is absolutely minimal. It's a great standard.

kevincox 3y ago
Can't it be a strong validator if the files don't change at all between revisions? Or does the revision number only increment on "significant" revisions?
- akira2501 3y ago
  The data does not but certain metadata elements might. It probably could be a strong validator anyways for it's use cases, but I made the decision in a hurry.
choppaface 3y ago
A great standard but also provides an effective cookie-less mechanism for user tracking e.g. https://levelup.gitconnected.com/no-cookies-no-problem-using-etags-for-user-tracking-3e745544176b https://levelup.gitconnected.com/no-cookies-no-problem-using...
- LtWorf 3y ago
  That's why tracking requires consent, regardless of the specific hack used to do it.

ajayvk 3y ago

An approach like https://github.com/benbjohnson/hashfs https://github.com/benbjohnson/hashfs allows file names to be updated at runtime to be content hashed. This removes the need for the extra "304 Not Modified" API calls from the client. This content hash based file renaming is usually done using a build step which renames files. For applications where the static file serving and HTTP request processing are done in the same application, this can be done in memory without a build step for file renames. I am using that approach in my project https://github.com/claceio/clace https://github.com/claceio/clace. It removes the need for a build step while making aggressive static file caching possible.

codetrotter 3y ago
I use content hashes for some of the images in parts of my site. And I use the IPFS scheme for it, and have the path be under /ipfs/ or some such. And so you could find the same file on IPFS if anyone served it there, as the content hash in the url tells you what to look for. Even though on my side I’ve done this all completely manually, so much so that it’s literally just me calculating the IPFS hash on my machine one time and then having symlinks with those content hashes so that /ipfs/ directory on my sites contains content that is served by their IPFS content hash, even though my server does not run an IPFS node or anything. A very interesting side effect of this is that one time I loaded one of my pages, the web browser actually picked up on the pattern and offered to load those files over actual IPFS!
- EGreg 3y ago
  Wait what? Browsers now offer to load over IPFS? I was wondering how you can trust an IFPS gateway, does does browser verify the file is legit using some checksum? Maybw subresource integrity supports IPFS content hashing or something? How does it generate cid anyway ? https://docs.ipfs.tech/concepts/content-addressing/#cids-are-not-file-hashes https://docs.ipfs.tech/concepts/content-addressing/#cids-are... How would you use SRI here to verify the cid (and not an additional out-of-band hash) make sure the gateway isn’t returning some crap to, say, inject malicious JS?
  - 3y ago

deleted 3y ago

[deleted]

alganet 3y ago

This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna document it. It means the account is out of balance. — What is this garbage? Please read the spec. This is going to make our API gateways, CDNs, everything go crazy. Can't let you move on with this PR. — Look. I documented it, made an enum with the code, it's clean. I'm an experienced REST developer. — It... it doesn't work like that Bob. Please, read the spec. — Hey, got enough approvals, "412 Account Out Of Balance" it is! It passes the tests. For each dev that knows proper HTTP, there's 10.000 Bobs.

esafak 3y ago
Which article would you ask Bob to read to learn the right way to do it?
- alganet 3y ago
  Bob has since moved on to crypto, leaving the cache invalidation eternally crippled. New Bob decided that everything is useless and wants to rewrite the whole backend using a faster language.
- mozey 3y ago
  Have to admit I've never used this code, and didn't know what it was about. Quickly read up about it. So ETag is a hash of the resource. You must provide it with requests that modify the resource. If your hash doesn't match the server hash, then 412 Precondition Failed is returned? https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412 https://www.rfc-editor.org/rfc/rfc9110#status.412 https://www.rfc-editor.org/rfc/rfc9110#status.412
  - alganet 3y ago
    You can provide all sorts of conditions using HTTP headers such as "If-Match", "If-None-Match", "If-Modified-Since", "If-Range", etc. The server can chose an HTTP code to indicate some sort of cache invalidation signal. 304 means "you're good, your cached version satisfies the conditions" 412 means "you're not good, your cached version does not satisfy the conditions" 412 usually applies to modifications, but it could be for reading too, in the case of ranged requests (getting a specific range of bytes from a large representation). See the "Range" header. These are all interconected. The headers, the codes, etc. They are very useful for caching and can save a lot of bandwidth. Browsers and CDNs use them extensively. Server-to-server communcation could use them as well, but I haven't seen popular implementations (let's say, a web framework that provides abstraction over these mechanisms).

rollulus 3y ago

I recently implemented this, great write-up. Regarding the hashing function, I’m curious about opinions. In my implementation I went for a cheap but weak cryptographic hash at first. Then I got worried that some auditor would flag it and time would be wasted convincing them to change their mind. But then I stumbled upon FNV [1], a non-cryptographic hash and part of Go’s standard library and went for it. Any thoughts? [1]: https://en.m.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function https://en.m.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93...

f_devd 3y ago
I recommend checking out XXHash[0], FNV is simple but not really optimized and relatively low quality (often still good enough). From the readme page: Hash Name, Bandwidth, Small Velocity, Quality, Comment XXH3 (SSE2), 31.5 GB/s, 133.1, 10, FNV64, 1.2 GB/s, 62.7, 5, Poor avalanche properties [0]: https://github.com/Cyan4973/xxHash https://github.com/Cyan4973/xxHash Also ETag is exactly the kind of thing non-cryptographic hashes are meant for, but if you can't convince them Blake3 is a very fast modern cryptographic hash function.
rednafi 3y ago
I feel like people tend to overthink in this regard. If SHA-256 hashing is good enough for GitHub's REST endpoints, it's good enough for me. If you're implementing weak validation, then you might need to preprocess the payload before running it through the hash function. For example, if your payload is JSON and you want to make it format-agnostic, then you'll need to normalize the payload and then compute the hash. In either case, the hashing algo probably doesn't matter as much.

ulrischa 3y ago

Oh this topic can cost you some years of your life

joosters 3y ago

I've not seen a very convincing use-case for ETags vs Last-Modified date caching. In the example request, the server still has to do all of the work generating the page, in order to calculate the ETag and then determine whether or not the page has changed. In most situations, it's simpler to have timestamps to compare against, because that gives the server a faster way to spot unmodified data. e.g. you get a HTTP request for some data that you know is sourced from a particular file, or a DB table. If the client sends a If-Modified-Since (or whatever the header name is), you have a good chance to be able to check the modified time of the data source before doing any complicated data processing, and are able to send back a not modified response sooner.

johneth 3y ago
When I use ETags, I'll generate the ETag value once (on startup), then cache and serve that value. When the resource changes, regenerate the ETag value. Obviously doesn't work if you're using ETags on dynamic resources, but works well for non-dynamic, but unpredictably frequently changing resources.
- derkades 3y ago
  And couldn't you do the same thing with a timestamp instead of a hash?
jmull 3y ago
They are essentially the same thing. Sometimes you have a last modified timestamp handy and sometimes you have a revision number handy. I would agree that you shouldn't be doing signifiant calculations -- like checksumming the contents of a file -- to determine the current etag value. Also, the excessive semantics of timestamps is a negative. It adds complication which sometimes trips developers up but doesn't improve the caching.

nikolayasdf123 3y ago

if only browsers respected this. none of browsers use ETag and If-None-Match mechanism. instead they do their own wizzardy caching...

demurgos 3y ago
I was refactoring a project serving user uploaded files yesterday, and had the occasion to test caching. Both Firefox and Chrome used ETag and If-None-Match properly to check and cache queries. Which problems did you encounter? There was still one thing that surprised me a bit (but also makes sense). Images are fetched only once per page load in my testing. If an image with 60sec of cache is loaded, then removed by JS and added back after 2 minutes, then the browser will reuse the image from the initial load.
- nikolayasdf123 2y ago
  the issue I got was: 1) I (AWS CloudFront) supply ETag and If-None-Match header; I can see that header in responses. 2) browsers sometimes do respect that (once) I see 304 in responses, but 99% of the time they don't include ETag/If-None-Match in requests and thus I never get 304 responses (albeit AWS CloudFront, resource, data — nothing changed) and instead they perform some other caching and reload whole resource again with TTL that does not seem to come from my headers, totally disregarding ETags/If-None-Match logic. for videos it is even worse. unless you set `preload=none` in html, Safari, Firefox, Chrome will have all different policies trying to preload all videos on screen ignoring lazyload html tags. worse of all, caching does not work well, and videos will be attempted to be loaded almost every time and ETags/If-None-Match totally ignored.
  - withinboredom 2y ago
    It sounds like you have caching disabled in your browser, either in dev-tools itself or through an extension. This is NOT normal behavior.
    - nikolayasdf123 2y ago