5 ms·

> "all the services needed to talk to each other" I'm not an expert by any means, but I'm pretty sure that statement indicates a problem.

jmole 8y ago
Yes, then you're just doing bad OOP with sockets instead of a language that was designed for it.
- spricket 8y ago
  Heh, definitely some truth to this. What saved us before, was our forest of code could depend on the database to maintain some sanity. And we leaned on it heavily. Hold a transaction open while 10,000 lines of code and a few N+1 queries do their business? Eh, okay, I guess. Maybe we didn't have the descipine to make microservices work. But IMO our engineering team was pretty good compared to others I've seen. All our "traditional" apps chugged along fine during the same period
  - raverbashing 8y ago
    Really, whenever someone blames "discipline" be suspicious Not even the army has perfect discipline, even with hard training. They have cross-checks, piles of processes and move slowly for the most part (Software development shouldn't aim to be like the army though)
spricket 8y ago
I don't think so. This kind of thing comes up constantly in RDBMS. New requirement means we need to join thneeds and widgets data together. In a regular database, even NoSql, this isn't a hard problem. When the services have their own datastores, well now they need to talk to eachother
- kayoone 8y ago
  not necessarily if you design a decoupled event driven approach where every service can just subscribe to the data it needs.
  - spricket 8y ago
    We actually tried this as well. It never made it out of testing. We ended up with copies of data in many places, which was annoying. We duplicated a lot of work for consuming the same events across multiple services and making sure they updated the "projection" the same way. However a much larger problem was overall bad tooling. Specifically the data storage requirements for an event stream eclipsed our wildest projections. We're talking many terabytes just on our local test nodes. We tried to remedy this by "compressing" past events into snapshots but the tooling for this doesn't really exist. It was far too common for a few bad events to get into the stream and cause massive chaos. We couldn't find a reasonable solution to rewind and fix past events, and replays took far too long without reliable snapshots. In the end I was convinced that the whole event driven approach was just a way of building your own "projection" databases on top of a "commit log" which was the event stream. Keeping record of past events also wasn't nearly as useful as we originally believed. We couldn't think of a single worthwhile use for our past event data that we couldn't just duplicate with an "audit" table and some triggers for data we cared about in a traditional db. Ironically we ended up tailing the commit log of a traditional db to build our projections. Around that time we all decided it was time to go back to normal RPC between services.
    - thoman23 8y ago
      I appreciate you sharing this. I'm considering embarking on this approach with my team, and everything you are mentioning is what I was worried about when I first started reading up on the microservices architecture. Now I'm seriously considering a somewhat hybrid approach: Collect all of my domain data in one giant normalized operational data store (using a fairly traditional ETL approach for this piece), and then having separate schemas for my services. The service schemas would have denormalized objects that are designed for the functional needs of the service, and would be implemented either as materialized views built off the upstream data store, or possibly with an additional "data pump" approach where activity in the upstream data store would trigger some sort of asynchronous process to copy the data into the service schemas. That way my services would be logically decoupled in the sense that if I wanted I could separate the entire schema for a given service into its own separate database later if needed. But by keeping it all in one database for now, it should make reconciliation and data quality checks easier. Note that I don't have a huge amount of data to worry about (~1-2TB) which could make this feasible.
Cthulhu_ 8y ago
Yeah, if you are going for a microservices architecture, you need at least one person or dedicated team in an oversight / architecture role that keeps the design and growth in check. Primarily that means saying "no" when someone wants to create a new service or open up a new line of communication. It's an exercise in limiting dependencies. And the easiest way to do that is to not build a microservices architecture; instead (and I hope I'm preaching to the choir here) build a monolith (or "a regular application") and only if you have good numbers and actual problems with scaling and the like do you start considering splitting off a section of your application. If you iterate on that long enough, MAYBE you'll end up with a microservices architecture.

jmole 8y ago

Yes, then you're just doing bad OOP with sockets instead of a language that was designed for it.

spricket 8y ago
Heh, definitely some truth to this. What saved us before, was our forest of code could depend on the database to maintain some sanity. And we leaned on it heavily. Hold a transaction open while 10,000 lines of code and a few N+1 queries do their business? Eh, okay, I guess. Maybe we didn't have the descipine to make microservices work. But IMO our engineering team was pretty good compared to others I've seen. All our "traditional" apps chugged along fine during the same period
- raverbashing 8y ago
  Really, whenever someone blames "discipline" be suspicious Not even the army has perfect discipline, even with hard training. They have cross-checks, piles of processes and move slowly for the most part (Software development shouldn't aim to be like the army though)

spricket 8y ago

I don't think so. This kind of thing comes up constantly in RDBMS. New requirement means we need to join thneeds and widgets data together. In a regular database, even NoSql, this isn't a hard problem. When the services have their own datastores, well now they need to talk to eachother

kayoone 8y ago
not necessarily if you design a decoupled event driven approach where every service can just subscribe to the data it needs.
- spricket 8y ago
  We actually tried this as well. It never made it out of testing. We ended up with copies of data in many places, which was annoying. We duplicated a lot of work for consuming the same events across multiple services and making sure they updated the "projection" the same way. However a much larger problem was overall bad tooling. Specifically the data storage requirements for an event stream eclipsed our wildest projections. We're talking many terabytes just on our local test nodes. We tried to remedy this by "compressing" past events into snapshots but the tooling for this doesn't really exist. It was far too common for a few bad events to get into the stream and cause massive chaos. We couldn't find a reasonable solution to rewind and fix past events, and replays took far too long without reliable snapshots. In the end I was convinced that the whole event driven approach was just a way of building your own "projection" databases on top of a "commit log" which was the event stream. Keeping record of past events also wasn't nearly as useful as we originally believed. We couldn't think of a single worthwhile use for our past event data that we couldn't just duplicate with an "audit" table and some triggers for data we cared about in a traditional db. Ironically we ended up tailing the commit log of a traditional db to build our projections. Around that time we all decided it was time to go back to normal RPC between services.
  - thoman23 8y ago
    I appreciate you sharing this. I'm considering embarking on this approach with my team, and everything you are mentioning is what I was worried about when I first started reading up on the microservices architecture. Now I'm seriously considering a somewhat hybrid approach: Collect all of my domain data in one giant normalized operational data store (using a fairly traditional ETL approach for this piece), and then having separate schemas for my services. The service schemas would have denormalized objects that are designed for the functional needs of the service, and would be implemented either as materialized views built off the upstream data store, or possibly with an additional "data pump" approach where activity in the upstream data store would trigger some sort of asynchronous process to copy the data into the service schemas. That way my services would be logically decoupled in the sense that if I wanted I could separate the entire schema for a given service into its own separate database later if needed. But by keeping it all in one database for now, it should make reconciliation and data quality checks easier. Note that I don't have a huge amount of data to worry about (~1-2TB) which could make this feasible.

Cthulhu_ 8y ago

Yeah, if you are going for a microservices architecture, you need at least one person or dedicated team in an oversight / architecture role that keeps the design and growth in check. Primarily that means saying "no" when someone wants to create a new service or open up a new line of communication. It's an exercise in limiting dependencies. And the easiest way to do that is to not build a microservices architecture; instead (and I hope I'm preaching to the choir here) build a monolith (or "a regular application") and only if you have good numbers and actual problems with scaling and the like do you start considering splitting off a section of your application. If you iterate on that long enough, MAYBE you'll end up with a microservices architecture.