4 ms·

I don't think so. This kind of thing comes up constantly in RDBMS. New requirement means we need to join thneeds and widgets data together. In a regular databas

I don't think so. This kind of thing comes up constantly in RDBMS. New requirement means we need to join thneeds and widgets data together. In a regular database, even NoSql, this isn't a hard problem. When the services have their own datastores, well now they need to talk to eachother

kayoone 8y ago
not necessarily if you design a decoupled event driven approach where every service can just subscribe to the data it needs.
- spricket 8y ago
  We actually tried this as well. It never made it out of testing. We ended up with copies of data in many places, which was annoying. We duplicated a lot of work for consuming the same events across multiple services and making sure they updated the "projection" the same way. However a much larger problem was overall bad tooling. Specifically the data storage requirements for an event stream eclipsed our wildest projections. We're talking many terabytes just on our local test nodes. We tried to remedy this by "compressing" past events into snapshots but the tooling for this doesn't really exist. It was far too common for a few bad events to get into the stream and cause massive chaos. We couldn't find a reasonable solution to rewind and fix past events, and replays took far too long without reliable snapshots. In the end I was convinced that the whole event driven approach was just a way of building your own "projection" databases on top of a "commit log" which was the event stream. Keeping record of past events also wasn't nearly as useful as we originally believed. We couldn't think of a single worthwhile use for our past event data that we couldn't just duplicate with an "audit" table and some triggers for data we cared about in a traditional db. Ironically we ended up tailing the commit log of a traditional db to build our projections. Around that time we all decided it was time to go back to normal RPC between services.
  - thoman23 8y ago
    I appreciate you sharing this. I'm considering embarking on this approach with my team, and everything you are mentioning is what I was worried about when I first started reading up on the microservices architecture. Now I'm seriously considering a somewhat hybrid approach: Collect all of my domain data in one giant normalized operational data store (using a fairly traditional ETL approach for this piece), and then having separate schemas for my services. The service schemas would have denormalized objects that are designed for the functional needs of the service, and would be implemented either as materialized views built off the upstream data store, or possibly with an additional "data pump" approach where activity in the upstream data store would trigger some sort of asynchronous process to copy the data into the service schemas. That way my services would be logically decoupled in the sense that if I wanted I could separate the entire schema for a given service into its own separate database later if needed. But by keeping it all in one database for now, it should make reconciliation and data quality checks easier. Note that I don't have a huge amount of data to worry about (~1-2TB) which could make this feasible.
    - spricket 8y ago
      There's two main approaches to handling "events". Using event sourcing vs direct RPC. After our disaster I highly recommend Google's approach, A structured gRPC layer between services with blocking calls. You might think you don't have much data, we didn't either, but when Kafka is firehosing updates to LoginStatus 24/7 data cost gets out of control fast. I'm going against the Martin Fowler grain hard here, but Event Sourcing in practice is largely a failure. It's bad tooling mostly as I mentioned, but please stay away. It's so bad.
- arethuza 8y ago
  "every service can just subscribe to the data it needs." Doesn't that imply that each service then has to store any data it receives in these events - potentially leading to a lot of duplication and all of the problems that can come with that (e.g. data stores getting out of sync).
  - thoman23 8y ago
    Yes, that's exactly what it implies. Like I said at the top of this thread, I'm not an expert on this approach (I've done my reading, but haven't yet spent time in the trenches), but my understanding is that you would embrace the duplication and eventual consistency. I do wonder how well it works in practice though, and how much time you would spend running cross-service reconciliation checks to make sure your independent data stores are still in sync.

kayoone 8y ago

not necessarily if you design a decoupled event driven approach where every service can just subscribe to the data it needs.

spricket 8y ago
We actually tried this as well. It never made it out of testing. We ended up with copies of data in many places, which was annoying. We duplicated a lot of work for consuming the same events across multiple services and making sure they updated the "projection" the same way. However a much larger problem was overall bad tooling. Specifically the data storage requirements for an event stream eclipsed our wildest projections. We're talking many terabytes just on our local test nodes. We tried to remedy this by "compressing" past events into snapshots but the tooling for this doesn't really exist. It was far too common for a few bad events to get into the stream and cause massive chaos. We couldn't find a reasonable solution to rewind and fix past events, and replays took far too long without reliable snapshots. In the end I was convinced that the whole event driven approach was just a way of building your own "projection" databases on top of a "commit log" which was the event stream. Keeping record of past events also wasn't nearly as useful as we originally believed. We couldn't think of a single worthwhile use for our past event data that we couldn't just duplicate with an "audit" table and some triggers for data we cared about in a traditional db. Ironically we ended up tailing the commit log of a traditional db to build our projections. Around that time we all decided it was time to go back to normal RPC between services.
- thoman23 8y ago
  I appreciate you sharing this. I'm considering embarking on this approach with my team, and everything you are mentioning is what I was worried about when I first started reading up on the microservices architecture. Now I'm seriously considering a somewhat hybrid approach: Collect all of my domain data in one giant normalized operational data store (using a fairly traditional ETL approach for this piece), and then having separate schemas for my services. The service schemas would have denormalized objects that are designed for the functional needs of the service, and would be implemented either as materialized views built off the upstream data store, or possibly with an additional "data pump" approach where activity in the upstream data store would trigger some sort of asynchronous process to copy the data into the service schemas. That way my services would be logically decoupled in the sense that if I wanted I could separate the entire schema for a given service into its own separate database later if needed. But by keeping it all in one database for now, it should make reconciliation and data quality checks easier. Note that I don't have a huge amount of data to worry about (~1-2TB) which could make this feasible.
  - spricket 8y ago
    There's two main approaches to handling "events". Using event sourcing vs direct RPC. After our disaster I highly recommend Google's approach, A structured gRPC layer between services with blocking calls. You might think you don't have much data, we didn't either, but when Kafka is firehosing updates to LoginStatus 24/7 data cost gets out of control fast. I'm going against the Martin Fowler grain hard here, but Event Sourcing in practice is largely a failure. It's bad tooling mostly as I mentioned, but please stay away. It's so bad.
arethuza 8y ago
"every service can just subscribe to the data it needs." Doesn't that imply that each service then has to store any data it receives in these events - potentially leading to a lot of duplication and all of the problems that can come with that (e.g. data stores getting out of sync).
- thoman23 8y ago
  Yes, that's exactly what it implies. Like I said at the top of this thread, I'm not an expert on this approach (I've done my reading, but haven't yet spent time in the trenches), but my understanding is that you would embrace the duplication and eventual consistency. I do wonder how well it works in practice though, and how much time you would spend running cross-service reconciliation checks to make sure your independent data stores are still in sync.