4 ms·

As someone who’s primarily worked with monoliths, I often wonder how often this exact problem happens, but where A and B are [micro]services owned by two differ

by btown 4y ago

As someone who’s primarily worked with monoliths, I often wonder how often this exact problem happens, but where A and B are [micro]services owned by two different teams, one is required by company policy to use their APIs not their raw databases, and escalation of each of these issues e.g. query size/rate limiting runs the risk of burning political capital on top of everything else. How does one JOIN across not just tables but opaque services, in the general case? Or does every team doing microservices silently expect that one day a data team will start querying for a massive number of records-by-ID from every service, and the veterans in each team plan for this load pattern accordingly?

pwg 4y ago
> Or does every team doing microservices silently expect that one day a data team will start querying for a massive number of records-by-ID from every service, and the veterans in each team plan for this load pattern accordingly? What tends to be by far more common is that each team fails to envision that someone, somewhere, sometime in the not so distant future will want or be required to retrieve more than one "element" at a time via their APIs. And so panic ensues when "other entity" begins feeding 20 API retrievals per second at their "one-at-a-time API" and their performance goes off the cliff it was always sitting near.
- NegativeLatency 4y ago
  Extra points if you manage to do it to your own team’s APIs because of bad design and planning
  - tracker1 4y ago
    You mean like 8+ joins to get that single record and then having it bottleneck when you get a few hundred requests a minute? (not bitter at all here)
thatwasunusual 4y ago
> How does one JOIN across not just tables but opaque services, in the general case? You (should) never do that. It's as simple as that. If you create microservices that are atomically depending on each other, you are doing something _extremely_ wrong.
- btown 4y ago
  Definitely not implying that two services should mutually depend on each other. But a third service C may want to look up in B for every record available from A - say, if B reports reservations, and A reports users who are members of a certain group, and you want the reservations relevant to a specific group. The OP article describes all sorts of pitfalls for the C team if you only had access to B via a "get by IDs" API.
alexisread 4y ago
I think it depends on whether you are talking ad-hoc ie. a user analyzing several datasets, or pipelined ie. preprocessed joins. For pipelined joins, effectively your data forms a DAG (directed acyclic graph, and yes we are ignoring recursion here). Providing your data services speak the same language you can create a pipeline off the first pipeline that joins the data and sticks it in a cache (eg. RDS, Elasticsearch). Changing the underlying data should then trigger a reload of the downstream pipelines. This is basically what Materialize.io, KSQLDB et. al. do - a reactive DAG with a database as the cache. One issue for larger companies is that you don't control the whole DAG, so discovery, security, protocols etc. need to be coordinated by an overarching architecture for this to work. Something like Apollo (GraphQL) is a simpler solution (in some ways) as you control the joins on the Apollo server which speak to backend (REST) APIs (other teams).
- btown 4y ago
  Doesn’t this mean every service must expose a way for downstream consumers (via Apollo or not) to subscribe to updates to allow them to invalidate their caches? I shudder to think how stale the DAG approach would be without this. I suppose this is doable if the company lives on Kafka, but what if it lives on RESTful microservices?
  - alexisread 4y ago
    Usually restful services are behind a loadbalancer/reverse proxy. This proxy could also handle things like security, logging, cache invalidation (the service behind the proxy) and service registration (other downstream services). Marking APIs as immutable/up or down (for rollback) state migrations etc. Is really important from a platform perspective, so it also makes sense for APIs to have a cache hook as well. The proxy can then manage the platform state wrt change propagation.
mcdonje 4y ago
>Or does every team doing microservices silently expect that one day a data team will start querying for a massive number of records-by-ID from every service, and the veterans in each team plan for this load pattern accordingly? I assume here by "data team" you mean reporting. Reporting and operations groups are very different with very different needs. Microservices are useful in operations settings where the flexibility of taking modules out of a monolith and putting the network between them outweighs the performance hit. Reporting directly from microservices is a recipe for disaster. To support reporting, the microservices need to contribute data to a data lake, data warehouse, or other repository.
oftenwrong 4y ago
IME that antipattern is common in small companies. This is mainly so because it's often the most expedient way to get something "working". A related problem is creating a new RPC for every variation of a query that an external service may require. One better approach is to ensure each service's db has the data it needs already at query time. For example, each service should ingest events from elsewhere in the system, and accumulate the relevant data for its responsibilities. Joins should always happen in the db. Another approach is to keep all the data in the same RDBMS. You can slice up the data into different schemas as you see fit. I have had a lot of success with this approach, reuniting databases where people have gone a bit too microservice-wild for their actual circumstances. You can vertically scale an RDBMS to quite a large size before seeking other approaches.
tracker1 4y ago
It really depends on how much data you need... I worked in an org where the primary data was in one database, and secondary data was in another. The DBA team wrote the query to call the other (remote) db across in part of the statement, and it was horribly slow... The N+1 pattern combined with memcached on the secondary lookups was so much faster in the end.. since it was limited to a display page worth of secondary lookups. A GraphQL server can relatively effectively do this for you. In the end it really depends... if you're talking even 10-100k users, a single, well optimized SQL RDBMS is your best bet... getting past that takes deep knowledge and/or more options/skills. In the end, most don't have that next step and the trend to Micro-Service all the things is jumped to too soon in most cases (and not soon enough in others).
LennyWhiteJr 4y ago
A couple ways. If the need is not real-time and analytical, you feed the data from multiple services into a separate BI database which can do slower and more complex joins across data from multiple data sources. Or if the need is real-time, you build a paginated API with a page limit that can always be processed within the API SLA. Then you build workflows on top of the paginated API to operate on that data. Generally, unbounded operations have to be broken up at some point. It just depends on how big the data set is.
kgeist 4y ago
>How does one JOIN across not just tables but opaque services One solution we use is to have an event queue from service A to which service B is subscribed. In the event handler, service B fills its own view table with data from service A. And then it can do joins on data from multiple services because everything is in the same DB. We require services to always emit "created" and "updated" events for its objects.
lmm 4y ago
You have a shared datastore, but not an SQL RDBMS. If you're big enough you build it in-house. See "An oral history of Bank Python", posted here a while back.

by btown 4y ago

pwg 4y ago

> Or does every team doing microservices silently expect that one day a data team will start querying for a massive number of records-by-ID from every service, and the veterans in each team plan for this load pattern accordingly? What tends to be by far more common is that each team fails to envision that someone, somewhere, sometime in the not so distant future will want or be required to retrieve more than one "element" at a time via their APIs. And so panic ensues when "other entity" begins feeding 20 API retrievals per second at their "one-at-a-time API" and their performance goes off the cliff it was always sitting near.

NegativeLatency 4y ago
Extra points if you manage to do it to your own team’s APIs because of bad design and planning
- tracker1 4y ago
  You mean like 8+ joins to get that single record and then having it bottleneck when you get a few hundred requests a minute? (not bitter at all here)

thatwasunusual 4y ago

> How does one JOIN across not just tables but opaque services, in the general case? You (should) never do that. It's as simple as that. If you create microservices that are atomically depending on each other, you are doing something _extremely_ wrong.

btown 4y ago
Definitely not implying that two services should mutually depend on each other. But a third service C may want to look up in B for every record available from A - say, if B reports reservations, and A reports users who are members of a certain group, and you want the reservations relevant to a specific group. The OP article describes all sorts of pitfalls for the C team if you only had access to B via a "get by IDs" API.

alexisread 4y ago

I think it depends on whether you are talking ad-hoc ie. a user analyzing several datasets, or pipelined ie. preprocessed joins. For pipelined joins, effectively your data forms a DAG (directed acyclic graph, and yes we are ignoring recursion here). Providing your data services speak the same language you can create a pipeline off the first pipeline that joins the data and sticks it in a cache (eg. RDS, Elasticsearch). Changing the underlying data should then trigger a reload of the downstream pipelines. This is basically what Materialize.io, KSQLDB et. al. do - a reactive DAG with a database as the cache. One issue for larger companies is that you don't control the whole DAG, so discovery, security, protocols etc. need to be coordinated by an overarching architecture for this to work. Something like Apollo (GraphQL) is a simpler solution (in some ways) as you control the joins on the Apollo server which speak to backend (REST) APIs (other teams).

btown 4y ago
Doesn’t this mean every service must expose a way for downstream consumers (via Apollo or not) to subscribe to updates to allow them to invalidate their caches? I shudder to think how stale the DAG approach would be without this. I suppose this is doable if the company lives on Kafka, but what if it lives on RESTful microservices?
- alexisread 4y ago
  Usually restful services are behind a loadbalancer/reverse proxy. This proxy could also handle things like security, logging, cache invalidation (the service behind the proxy) and service registration (other downstream services). Marking APIs as immutable/up or down (for rollback) state migrations etc. Is really important from a platform perspective, so it also makes sense for APIs to have a cache hook as well. The proxy can then manage the platform state wrt change propagation.

mcdonje 4y ago

>Or does every team doing microservices silently expect that one day a data team will start querying for a massive number of records-by-ID from every service, and the veterans in each team plan for this load pattern accordingly? I assume here by "data team" you mean reporting. Reporting and operations groups are very different with very different needs. Microservices are useful in operations settings where the flexibility of taking modules out of a monolith and putting the network between them outweighs the performance hit. Reporting directly from microservices is a recipe for disaster. To support reporting, the microservices need to contribute data to a data lake, data warehouse, or other repository.

oftenwrong 4y ago

IME that antipattern is common in small companies. This is mainly so because it's often the most expedient way to get something "working". A related problem is creating a new RPC for every variation of a query that an external service may require. One better approach is to ensure each service's db has the data it needs already at query time. For example, each service should ingest events from elsewhere in the system, and accumulate the relevant data for its responsibilities. Joins should always happen in the db. Another approach is to keep all the data in the same RDBMS. You can slice up the data into different schemas as you see fit. I have had a lot of success with this approach, reuniting databases where people have gone a bit too microservice-wild for their actual circumstances. You can vertically scale an RDBMS to quite a large size before seeking other approaches.

tracker1 4y ago

It really depends on how much data you need... I worked in an org where the primary data was in one database, and secondary data was in another. The DBA team wrote the query to call the other (remote) db across in part of the statement, and it was horribly slow... The N+1 pattern combined with memcached on the secondary lookups was so much faster in the end.. since it was limited to a display page worth of secondary lookups. A GraphQL server can relatively effectively do this for you. In the end it really depends... if you're talking even 10-100k users, a single, well optimized SQL RDBMS is your best bet... getting past that takes deep knowledge and/or more options/skills. In the end, most don't have that next step and the trend to Micro-Service all the things is jumped to too soon in most cases (and not soon enough in others).

LennyWhiteJr 4y ago

A couple ways. If the need is not real-time and analytical, you feed the data from multiple services into a separate BI database which can do slower and more complex joins across data from multiple data sources. Or if the need is real-time, you build a paginated API with a page limit that can always be processed within the API SLA. Then you build workflows on top of the paginated API to operate on that data. Generally, unbounded operations have to be broken up at some point. It just depends on how big the data set is.

kgeist 4y ago

>How does one JOIN across not just tables but opaque services One solution we use is to have an event queue from service A to which service B is subscribed. In the event handler, service B fills its own view table with data from service A. And then it can do joins on data from multiple services because everything is in the same DB. We require services to always emit "created" and "updated" events for its objects.

lmm 4y ago

You have a shared datastore, but not an SQL RDBMS. If you're big enough you build it in-house. See "An oral history of Bank Python", posted here a while back.