4 ms·
In short, yes. The database is the bottleneck because it's much harder to scale than applications. The path of evolution in the industry: 1. Stateful applica
by namelosw 7y ago
In short, yes.
The database is the bottleneck because it's much harder to scale than applications.
The path of evolution in the industry:
1. Stateful application - usually only 1 server, not distributed at all. It's very hard to scale.
2. An obvious solution is to make application stateless and having a centralized state. Then the application is very easy to scale and operate. And databases slowly became the bottleneck because there are much more applications than databases.
3. Then not so obvious solution is to split your problem domain into solution contexts, where every context become an application and each application have a database to talk to. So the databases are still not really distributed but it's sort of distributed by your sub-domain of business. (I think it's the industry mainstream or becoming mainstream now)
4. Then the non-obvious solution is to have truly distributed states split across your application. Basically, the goal is you can distribute any object across a whole lot cluster, in a more or less transparency way which lets you treat the cluster as a whole without concern about individual machines. (There are some cutting edge stuff yet to become mainstream, like Orleans / Lasp-lang or just cluster/shared actor like Akka, or just distributed Erlang, etc)
It's an interesting example that short-term solution is in a totally different direction from the powerful solution. To address this in short-term is to extract and centralize the state, while to address this in the long-term is to truly distribute the state.
- jchrisa 7y agoOne example of #4 that is making headway in production systems is the Calvin protocol (a RAFT based write-ahead-log with lock-free deterministic transaction application). It's implemented by my employer FaunaDB. You can learn more here https://fauna.com/blog/consistency-without-clocks-faunadb-transaction-protocol https://fauna.com/blog/consistency-without-clocks-faunadb-tr...
- dranka 7y agoA variant of no 3 is used for some large scale telecom equipment supporting multi millions of attached users with serious uptime requirements. State is distributed and handled locally and replicated to a sibling node for fallback protection in case a node goes down. The replication is dynamic so new sibling relationships are established when nodes come and go. There is also some more persistent state (like billing information) that is used to recover in case of total node failure but a lot of the transient state can be rebuilt when users are forced to reattach
- idlewords 7y agoI want to quibble with you on one thing—the database is often easy to scale, by three orders of magnitude, by configuring it correctly, and then moving it onto dedicated hardware.
- tracker1 7y agoSimilarly, very few applications need more performance than you can get by scaling vertically and having a hot spare replica database. Although a distributed data store on a few cheaper VPSes/nodes is probably less expensive.
- paulddraper 7y agoAn easy and often effective solution is read replicas. Operations that need ACID get ACID and operations that don't, don't. And every production-grade DB make read replicas easy. Yes, multi-master is the interesting and challenging thing to us, but single-master still has a lot of juice.
- merb 7y agoactually before 3 and 4 you can scale reads which often is the bottleneck and most major databases can scale reads pretty good these days.