3 ms·
I think it depends on whether you are talking ad-hoc ie. a user analyzing several datasets, or pipelined ie. preprocessed joins. For pipelined joins, effectivel
by alexisread 4y ago
I think it depends on whether you are talking ad-hoc ie. a user analyzing several datasets, or pipelined ie. preprocessed joins.
For pipelined joins, effectively your data forms a DAG (directed acyclic graph, and yes we are ignoring recursion here). Providing your data services speak the same language you can create a pipeline off the first pipeline that joins the data and sticks it in a cache (eg. RDS, Elasticsearch).
Changing the underlying data should then trigger a reload of the downstream pipelines.
This is basically what Materialize.io, KSQLDB et. al. do - a reactive DAG with a database as the cache.
One issue for larger companies is that you don't control the whole DAG, so discovery, security, protocols etc. need to be coordinated by an overarching architecture for this to work.
Something like Apollo (GraphQL) is a simpler solution (in some ways) as you control the joins on the Apollo server which speak to backend (REST) APIs (other teams).
- btown 4y agoDoesn’t this mean every service must expose a way for downstream consumers (via Apollo or not) to subscribe to updates to allow them to invalidate their caches? I shudder to think how stale the DAG approach would be without this. I suppose this is doable if the company lives on Kafka, but what if it lives on RESTful microservices?
- alexisread 4y agoUsually restful services are behind a loadbalancer/reverse proxy. This proxy could also handle things like security, logging, cache invalidation (the service behind the proxy) and service registration (other downstream services). Marking APIs as immutable/up or down (for rollback) state migrations etc. Is really important from a platform perspective, so it also makes sense for APIs to have a cache hook as well. The proxy can then manage the platform state wrt change propagation.