4 ms·

Hey HN, When I built my last startup, Spectrum, we spent months building custom caching for our GraphQL API from scratch. It never worked well enough to allevi

by mxstbr 5y ago

Hey HN, When I built my last startup, Spectrum, we spent months building custom caching for our GraphQL API from scratch. It never worked well enough to alleviate our scaling troubles as we could only cache data for unauthenticated users since we had no invalidation. (since we open sourced it all before GitHub acquired us, you can even read through my terrible code[0]) When Tim told me he had built a prototype of a CDN specifically for caching GraphQL query results with proper invalidation, my first thought was: "Finally!" Not only had he made it possible to cache POST requests (which GraphQL requests usually are), he had made it possible to purge cached query results per specific GraphQL object. For example, when a user edits their name the API can call a purgeUser(id: $ID) mutation and any cached query result that contains that user's data is invalidated. GraphCDN is based on Fastly Compute@Edge under the hood, which is really the main reason we were able to spin this up so quickly. Huge shoutout to the folks building that! We'll be around all day to answer any questions you have about GraphCDN — ask us anything! [0]: https://github.com/withspectrum/spectrum/blob/alpha/api/apollo-server.js https://github.com/withspectrum/spectrum/blob/alpha/api/apol...

cpursley 5y ago
Really interesting. Can this work with Hasura?
- timsuchanek 5y ago
  Yes, it does! We have more and more people who want a more powerful cache setup than what Hasura offers or are just self-hosting Hasura. Both self-hosted and Hasura cloud work!
  - cpursley 5y ago
    Shut up and take my money! This will save me a bunch of time from my original plan of rolling a distributed Elixir proxy caching cluster to put in front of Hasura. I'd love to see a write up/blog post for working specifically with Hasura and their auth system. Would probably be helpful for seo purposes as well.
    - mxstbr 5y ago
      Absolutely, we'll work on that. Let us know if you have any questions or run into any trouble at support@graphcdn.io!
0xy 5y ago
This is a super interesting project and thanks for posting it. I'm just wondering why you decided on Fastly over Cloudflare Workers or AWS Lambda Edge for this?
- timsuchanek 5y ago
  Good question! These are the main reasons: Fastly has much faster cache purging - it can purge any content globally in about 150ms. Purging is one of THE crucial parts which make GraphCDN work. With such a fast purging, we can deliver Read-after-write consistency . In our tests with Cloudflare that took a few seconds. Additionally, in our tests, Fastly was in general just a bit faster. While we're also JavaScript and V8 fans, our edge layer on Fastly is written in Rust - which is a pleasure to use, especially for a product as ours. Fastly runs it as WASM on the edge - they can load the worker in about 40 microseconds. Fastly Compute@Edge is still in limited availability, but if you get a chance, I highly recommend checking it out!
  - the_mitsuhiko 5y ago
    > we can deliver Read-after-write consistency Based on the comment on max below that's not read-after-write consistency but it just becomes eventually consistent but ideally within ~150ms. What would be the consequences of bypassing the cache for read after write, does that break your model in any way? (I guess at the very least you will lose your statistics feature)
    - timsuchanek 5y ago
      In all mutations you run through GraphCDN, we wait until the purging is done (as you said, ideally ~150ms). So that means at the point the mutation result comes back to the client, all other queries will get the new content from then on. You can bypass the cache if you want to, but it's not necessary. As it would just be like a Cache MISS, that would totally work.
      - the_mitsuhiko 5y ago
pbowyer 5y ago
Good work! What was it about this problem that made you need Compute@Edge rather than the standard Fastly/Varnish VCL-backed caching? If you tried that route, how far were you able to get with Varnish before needing the new Compute@Edge product?
- gnz00 5y ago
  It's been awhile but IIRC you can't inspect the body on a POST request using Varnish. Either that or any real compute in Varnish gets super gnarly. I also think Fastly has a distributed K/V for state storage that you can access if you're using their new compute.
  - timsuchanek 5y ago
    That is correct, you need to first transform the POST request into something Fastly can cache. They don't have a K/V Store besides the edge dictionaries with max 8kb value size yet, but I've heard something is coming...
    - gnz00 5y ago
      Ah cool, thanks for the update. I haven't looked at their stuff in over a year. Congrats on the launch, looks really slick.
- timsuchanek 5y ago
  Good question pbowyer! We actually do a bunch of GraphQL parsing / calculations in the Compute@Edge layer. So without that, we would not be able to provide our current features. We also support a feature we call "scopes" - where we both support cookies and headers in the "Vary" header so you can cache user-specific data only for the user with a specific "Authorization" header for example. In order to support scopes with cookies, we needed Compute@Edge and could not make that work with VCL on its own. But believe me - we tried. Our first version was actually mostly built on top of VCL.
latch 5y ago
> We spent months building custom caching for our GraphQL API from scratch Could you not abandon GraphQL? Returning non-customized responses, while taking more bandwidth, is much more cache friendly for the client, proxies and particularly servers - where you can do simple yet powerful things (e.g. version-based invalidation and pre-generated payloads). Like, I went to https://spectrum.chat/explore https://spectrum.chat/explore and clicked on "Tech. It uses GraphQL to load the communities. Why not just hit /v1/communities?category=tech which would loosely translate into one of: -- if you want to serialize the object on each get select * from communities where tags @> array['tech'] -- if you can look up the communities in a cache by id select id from communities where tags @> array['tech'] -- if you have a high read / write and can just pre-serialize the payload and then glue together the response select summary from communities where tags @> array['tech']
- timsuchanek 5y ago
  I think it depends what you want as your cache invalidation policy. Simple TTL-based cache invalidation wouldn't necessarily need GraphCDN. However, no matter if REST, GraphQL or any other protocol - making sure, that when content changed, to invalidate it properly on the edge is not trivial. If I understand correctly, you're suggesting to cache on a data-layer level (below the app in the stack) instead of what we do - above the app in the stack - just in front of the client. That is also a valid approach. It'll need more custom code in your application and has the disadvantage, that your origin will still be hit on every request, while we fully cache on the edge - cached queries won't hit your server anymore and will take load off origin and have the minimum latency possible.
  - latch 5y ago
    I'm suggesting that you can cache both at the data layer and "above the app". I was just highlighting one particularly powerful patterns available if you're having trouble scaling (like my parent said). > making sure, that when content changed, to invalidate it properly on the edge is not trivial The trivial way to do this is simply not to invalidate. Include a version in the cache key. When content changes, the version increases, and clients get new versions. (I think this is sometimes called lazy cache invalidation). It works well with LRU caches. It's still a hit for the top-level keys, but you avoid all the heavy data load and rendering. Granted, that doesn't solve every case, but I'm not sure it's fair to say that it'll take more custom code when the initial approach took "months building custom caching" that never really worked. Also, I think you'll find your origin under less load due to having fewer variants.
nivertech 5y ago
How GraphQL subscriptions are handled? Are they supported? Do they count as a single or multiple requests for pricing purposes?
- timsuchanek 5y ago
  As of now, GraphCDN only caches queries. Subscriptions are just passed through to the origin - a normal WS connection is built. One WS connection counts as one request.
eurasiantiger 5y ago
How does the invalidation work if data changes independently in the backend, i.e. without using mutations?
- timsuchanek 5y ago
  We give you a purging API (GraphQL as well) where you can target specific queries (`allUsers`) or types (`User`) that you want to purge. You can read more here: https://graphcdn.io/docs/cache-purging https://graphcdn.io/docs/cache-purging
ivanvanderbyl 5y ago
Max & Tim, thanks for building this. This is something I recently spent 2 weeks trying to solve, then more pressing issues came up so we threw more CPUs at the problem. So from that I can appreciate how complex this is to solve, and the UX is awesome, so cool that you can edit the cache keys in the UI. Do you/will you handle caching individual request layers? For example if I need part of my query to always be fresh, but some expensive part can be cached for hours, is this possible? And somewhat related, what about keying on operation variables?
- mxstbr 5y ago
  Thank you for the nice words, glad to hear you like GraphCDN. We don't currently cache partial queries / request layers. For now, I would recommend splitting the query into two requests — one that loads the uncached data and one for the cached data. We're definitely thinking about this exact use case though, stay tuned!
  - graphql 5y ago
    Ah yes, the devil in the details. Partial query caching appears to be the unicorn we all would like to chase, capture, and ultimately study. GraphQL is powerful, But with great power, comes great responsibility, and it is too easy to dig your own grave if blindly jumping in with it. I wonder if we can learn a thing or two, or just flat out steal, some of the concepts used in flagship relational databases. For example, how Postgres has a query optimizer tucked away secretly under the hood that attempts to alter queries to be more efficient.
agmontpetit 5y ago
Fastly has have trouble caching POST requests because Clustering [0] doesn't work with POST requests (AFAIK there is not workaround). Does Compute@Edge have this limitation? [0]: https://developer.fastly.com/learning/vcl/clustering/ https://developer.fastly.com/learning/vcl/clustering/
- timsuchanek 5y ago
  Under the hood, we're turning the POST requests into get requests, so that limitation shouldn't affect us. As far as I know, underlying C@E behaves like the one used in a VCL service.
  - hermanradtke 5y ago
    What happens when the query string gets too long?
    - timsuchanek 5y ago
      The question that had to be asked :D First of all, with the current query string size, you can already send quite big queries (8kb), which should suffice for many use-cases. The introspection query, already quite huge for reference is 1kb. For the ones who need more - splitting the values into `Vary` headers is an option to further increase it. At some point - around 50kb there just is a hard limit. If you then want to send an even bigger query - we have persisted queries on our roadmap. Then you'd just send a hash.

by mxstbr 5y ago

cpursley 5y ago

Really interesting. Can this work with Hasura?

timsuchanek 5y ago
Yes, it does! We have more and more people who want a more powerful cache setup than what Hasura offers or are just self-hosting Hasura. Both self-hosted and Hasura cloud work!
- cpursley 5y ago
  Shut up and take my money! This will save me a bunch of time from my original plan of rolling a distributed Elixir proxy caching cluster to put in front of Hasura. I'd love to see a write up/blog post for working specifically with Hasura and their auth system. Would probably be helpful for seo purposes as well.
  - mxstbr 5y ago
    Absolutely, we'll work on that. Let us know if you have any questions or run into any trouble at support@graphcdn.io!

0xy 5y ago

This is a super interesting project and thanks for posting it. I'm just wondering why you decided on Fastly over Cloudflare Workers or AWS Lambda Edge for this?

timsuchanek 5y ago
Good question! These are the main reasons: Fastly has much faster cache purging - it can purge any content globally in about 150ms. Purging is one of THE crucial parts which make GraphCDN work. With such a fast purging, we can deliver Read-after-write consistency . In our tests with Cloudflare that took a few seconds. Additionally, in our tests, Fastly was in general just a bit faster. While we're also JavaScript and V8 fans, our edge layer on Fastly is written in Rust - which is a pleasure to use, especially for a product as ours. Fastly runs it as WASM on the edge - they can load the worker in about 40 microseconds. Fastly Compute@Edge is still in limited availability, but if you get a chance, I highly recommend checking it out!
- the_mitsuhiko 5y ago
  > we can deliver Read-after-write consistency Based on the comment on max below that's not read-after-write consistency but it just becomes eventually consistent but ideally within ~150ms. What would be the consequences of bypassing the cache for read after write, does that break your model in any way? (I guess at the very least you will lose your statistics feature)
  - timsuchanek 5y ago
    In all mutations you run through GraphCDN, we wait until the purging is done (as you said, ideally ~150ms). So that means at the point the mutation result comes back to the client, all other queries will get the new content from then on. You can bypass the cache if you want to, but it's not necessary. As it would just be like a Cache MISS, that would totally work.
    - the_mitsuhiko 5y ago

pbowyer 5y ago

Good work! What was it about this problem that made you need Compute@Edge rather than the standard Fastly/Varnish VCL-backed caching? If you tried that route, how far were you able to get with Varnish before needing the new Compute@Edge product?

gnz00 5y ago
It's been awhile but IIRC you can't inspect the body on a POST request using Varnish. Either that or any real compute in Varnish gets super gnarly. I also think Fastly has a distributed K/V for state storage that you can access if you're using their new compute.
- timsuchanek 5y ago
  That is correct, you need to first transform the POST request into something Fastly can cache. They don't have a K/V Store besides the edge dictionaries with max 8kb value size yet, but I've heard something is coming...
  - gnz00 5y ago
    Ah cool, thanks for the update. I haven't looked at their stuff in over a year. Congrats on the launch, looks really slick.
timsuchanek 5y ago
Good question pbowyer! We actually do a bunch of GraphQL parsing / calculations in the Compute@Edge layer. So without that, we would not be able to provide our current features. We also support a feature we call "scopes" - where we both support cookies and headers in the "Vary" header so you can cache user-specific data only for the user with a specific "Authorization" header for example. In order to support scopes with cookies, we needed Compute@Edge and could not make that work with VCL on its own. But believe me - we tried. Our first version was actually mostly built on top of VCL.

latch 5y ago

> We spent months building custom caching for our GraphQL API from scratch Could you not abandon GraphQL? Returning non-customized responses, while taking more bandwidth, is much more cache friendly for the client, proxies and particularly servers - where you can do simple yet powerful things (e.g. version-based invalidation and pre-generated payloads). Like, I went to https://spectrum.chat/explore https://spectrum.chat/explore and clicked on "Tech. It uses GraphQL to load the communities. Why not just hit /v1/communities?category=tech which would loosely translate into one of: -- if you want to serialize the object on each get select * from communities where tags @> array['tech'] -- if you can look up the communities in a cache by id select id from communities where tags @> array['tech'] -- if you have a high read / write and can just pre-serialize the payload and then glue together the response select summary from communities where tags @> array['tech']

timsuchanek 5y ago
I think it depends what you want as your cache invalidation policy. Simple TTL-based cache invalidation wouldn't necessarily need GraphCDN. However, no matter if REST, GraphQL or any other protocol - making sure, that when content changed, to invalidate it properly on the edge is not trivial. If I understand correctly, you're suggesting to cache on a data-layer level (below the app in the stack) instead of what we do - above the app in the stack - just in front of the client. That is also a valid approach. It'll need more custom code in your application and has the disadvantage, that your origin will still be hit on every request, while we fully cache on the edge - cached queries won't hit your server anymore and will take load off origin and have the minimum latency possible.
- latch 5y ago
  I'm suggesting that you can cache both at the data layer and "above the app". I was just highlighting one particularly powerful patterns available if you're having trouble scaling (like my parent said). > making sure, that when content changed, to invalidate it properly on the edge is not trivial The trivial way to do this is simply not to invalidate. Include a version in the cache key. When content changes, the version increases, and clients get new versions. (I think this is sometimes called lazy cache invalidation). It works well with LRU caches. It's still a hit for the top-level keys, but you avoid all the heavy data load and rendering. Granted, that doesn't solve every case, but I'm not sure it's fair to say that it'll take more custom code when the initial approach took "months building custom caching" that never really worked. Also, I think you'll find your origin under less load due to having fewer variants.

nivertech 5y ago

How GraphQL subscriptions are handled? Are they supported? Do they count as a single or multiple requests for pricing purposes?

timsuchanek 5y ago
As of now, GraphCDN only caches queries. Subscriptions are just passed through to the origin - a normal WS connection is built. One WS connection counts as one request.

eurasiantiger 5y ago

How does the invalidation work if data changes independently in the backend, i.e. without using mutations?

timsuchanek 5y ago
We give you a purging API (GraphQL as well) where you can target specific queries (`allUsers`) or types (`User`) that you want to purge. You can read more here: https://graphcdn.io/docs/cache-purging https://graphcdn.io/docs/cache-purging

ivanvanderbyl 5y ago

Max & Tim, thanks for building this. This is something I recently spent 2 weeks trying to solve, then more pressing issues came up so we threw more CPUs at the problem. So from that I can appreciate how complex this is to solve, and the UX is awesome, so cool that you can edit the cache keys in the UI. Do you/will you handle caching individual request layers? For example if I need part of my query to always be fresh, but some expensive part can be cached for hours, is this possible? And somewhat related, what about keying on operation variables?

mxstbr 5y ago
Thank you for the nice words, glad to hear you like GraphCDN. We don't currently cache partial queries / request layers. For now, I would recommend splitting the query into two requests — one that loads the uncached data and one for the cached data. We're definitely thinking about this exact use case though, stay tuned!
- graphql 5y ago
  Ah yes, the devil in the details. Partial query caching appears to be the unicorn we all would like to chase, capture, and ultimately study. GraphQL is powerful, But with great power, comes great responsibility, and it is too easy to dig your own grave if blindly jumping in with it. I wonder if we can learn a thing or two, or just flat out steal, some of the concepts used in flagship relational databases. For example, how Postgres has a query optimizer tucked away secretly under the hood that attempts to alter queries to be more efficient.

agmontpetit 5y ago

Fastly has have trouble caching POST requests because Clustering [0] doesn't work with POST requests (AFAIK there is not workaround). Does Compute@Edge have this limitation? [0]: https://developer.fastly.com/learning/vcl/clustering/ https://developer.fastly.com/learning/vcl/clustering/

timsuchanek 5y ago
Under the hood, we're turning the POST requests into get requests, so that limitation shouldn't affect us. As far as I know, underlying C@E behaves like the one used in a VCL service.
- hermanradtke 5y ago
  What happens when the query string gets too long?
  - timsuchanek 5y ago
    The question that had to be asked :D First of all, with the current query string size, you can already send quite big queries (8kb), which should suffice for many use-cases. The introspection query, already quite huge for reference is 1kb. For the ones who need more - splitting the values into `Vary` headers is an option to further increase it. At some point - around 50kb there just is a hard limit. If you then want to send an even bigger query - we have persisted queries on our roadmap. Then you'd just send a hash.