4 ms·
> Split up the monolith into multiple interconnected services, each with its own data store that could be scaled on its own terms. Just to note: you don't have
by deathanatos 3y ago
> Split up the monolith into multiple interconnected services, each with its own data store that could be scaled on its own terms.
Just to note: you don't have to split out all the possible microservices at this junction. You can ask, "what split would have the most impact?"
In my case, we split out some timeseries data from Mongo into Cassandra. Cass's table structure was a much better fit — that dataset had a well defined schema, so Cass could pack the data much more efficiently; for that subset, we didn't need the flexibility of JSON docs. And it was the bulk of our data, and so Mongo was quite happy after that. Only a single split was required. (And technically, we were a monolith before and after: the same service just ended up writing to two databases.)
Ironically, later, an airchair architect wanted to merge all the data into a JSON document store, which resulted in numerous "we've been down that road, and we know where it goes" type discussions.
- kreetx 3y agoIn a way, in the article they also did a split: specific heavy select queries were offloaded to a replica.
- agentultra 3y agoThey could probably squeeze more depending on their workload patterns. RDBMS' typically optimize for fast/convenient writes. If your write load would be fine with a small increase in latency then you can do a lot of de-normalization so that your reads can avoid using tonnes of joins, aggregates, windows, etc at read-time. Update write path so that you update all of the de-normalized views at write time. Depending on your read load and application structure you can get a lot more scale with caching. Decent article.
- mamcx 3y agoIs interesting that the idea of micro services is throw like a obvious "solution". Is not. "Scale-up" MUST be the "obvious" solution. What is missed by many, and this article touch (despite saying that micro-services is a "solid" choice) is that "Scale-up" is "scale-out" without breaking the consistency of the DB. Is a lot you can do to squeeze, and is rare you need to ignore join, data validations and other anti-patterns that are normally trow casual when problems of performance happens.
- deathanatos 3y agoI don't know what to tell you other than I've seen vertical scaling hit its ceiling, several times. The OP lists "scale vertically first" as a given; to an extent, I agree with it, and the comment you're responding to is made with that as a base assumption. There are sometimes diminishing returns to simple scaling; e.g., in my current job, each new disk we add adds 1/n disks' worth of capacity. Each scaling step happens quicker and quicker (assuming growth of the underlying system). Eventually, you hit the wall in the OP, in that you need design level changes, not just quick fixes. The situation I mention in my comment was one of those: we'd about reached the limits of what was possible with the setup we had. We were hitting things such as bringing in new nodes was difficult: the time for the replica to replicate was getting too long, and Mongo, at the time, had some bug that caused like a ~30% chance that the replica would SIGSEGV and need to restart the replication from scratch. Operationally, it was a headache, and the split moved a lot of data out that made these cuts not so bad. (Cassandra did bring its own challenges, but the sum of the new state was that it was better than where we were.) Consistency is something you must pay attention to. In our case, the old foreign key between the two systems was the user ID, and we had specific checks to ensure consistency of it.
- kedean 3y agoFunny enough I frequently have the opposite problem, justifying repeatedly why Cassandra is a bad fit for relatively short lived, frequently updated data (tombstones, baby).
- deathanatos 3y agoI'd agree with you there. The specific data that went into Cassandra in our case was basically immutable. (And somehow, IIRC, we still had issues around tombstones. I am not a fan of them.) Cassandra's tooling left much to be desired around inspecting the exact state of tombstones within the cluster.
- alexchamberlain 3y agoThe other thing worth noting is a server can read from 2 datastores - which as a sibling comment says, they ended up doing with a read replica. There's nothing preventing you from reading from Postgres and Redis in the same process!