4 ms·
At the hardware level, one thing that has changed over the years is the exponential growth in network speeds and feeds – within a datacenter and across datacent
by vinay_ys 3y ago
At the hardware level, one thing that has changed over the years is the exponential growth in network speeds and feeds – within a datacenter and across datacenters.
Layer-3 IP Clos networks are pretty much a standard within a datacenter, making multiple redundant paths available. Across datacenters, optical circuit switched networks with lots of redundant paths are very common.
Modern network fabrics also have end-to-end mechanisms for bandwidth engineering based on per message QoS/priority set by applications as well as very accurate time-keeping.
Observability mechanisms and operational practices have also tremendously improved. A single node becoming unreachable from all other nodes is quite possible. But a single node being reachable to some subset of other nodes in the cluster and not others is nearly impossible. Only way a network partition can happen is due to misconfiguration. Purely layer-3 IP clos networks with BGP advertised multiple ECMP routes are a lot simpler than layer-2 networks of the past. With modern switch operating systems, modern devops practices, Infrastructure-as-Code etc. misconfiguration is nearly impossible.
At a higher layer, disaggregated storage and compute architectures are standard for any distributed database system. In this model, storage fabric is quite simple and much more distributed and redundant. Today, it is a lot easier to guarantee a consistent and highly available transactional write to a quorum of simple storage nodes spread across multiple datacenters in a region. On top of that, building higher-level isolation guaranteeing database transactions in a tier of ephemeral compute nodes is relatively much simpler.
This kind of system is extremely resilient to failures at all levels – node failures are detected and lost processing capacity recovered in seconds; failed transactions are retried in milliseconds; loss of redundancy due to failed storage nodes are rebuilt in seconds; loss of network capacity due to failed circuits or intermediate switches are routed around in microseconds.
So, in today's context, we build databases that provide flexible transaction isolation guarantees that is selected appropriately on a per transaction basis by the application. Application also specifies the latency budget it has available per transaction. With all these improvements, thankfully, application developers don't have worry about CAP theorem like they had to in the heydays of NoSQL databases in 2012.
- chx 3y agoCould you provide links to these things for those who are stuck in the past and want to learn more?
- vinay_ys 3y agoWell, there's no one comprehensive link I can suggest. But you can find plenty of links on arxiv.org for research publications on these topics. You can look at the ones from big companies like Google, Microsoft and Meta/Facebook. You can also search in Youtube for various conference talks on the same topics.
- deleted 3y ago[deleted]
- vendiddy 3y agoThanks for the write-up. Was interesting to read!
- belter 3y agoThis is a variation on Google's argument that "our network is so resilient we don't need to worry about Partitions so we can offer C and A and P" but is a weak one, more and more on the face of real present experiences. Due to the quorum requirements, probability of failure increases with larger quorum sizes, as the number of intersections (and therefore potential points of failure) increases with more complex quorum arrangements.
- vinay_ys 3y ago> Due to the quorum requirements, probability of failure increases with larger quorum sizes Well, that's not true due to two key design points: - Size of the read or write quorum can be chosen on a per transaction basis depending on what level of availability and consistency you want. The design goal is to lower the likelyhood of the minimum number of quorum members you need for your transaction will not be available. - Given the transaction processing nodes are ephemeral and independent from storage nodes, if a transaction processing node which is designated as a member of a quorum dies, it can be quickly and instantly replaced with a new one and all in-flight operations will be retried by the smart clients.
- belter 3y agoWhile all those design strategies are indeed valid, and adaptable dynamic quorum sizes, have the potential to manage certain trade-offs, the practical implementation of such systems presents such a challenges, due to the inherent complexities of distributed systems that...Tackling these complexities with those strategies should be approached with a degree of humility. Overheads and latency issues in node replacement, particularly under conditions of unexpected high load or network difficulties, compound these challenges. These issues often manifest in correlated failures, which are more common than isolated node failures. In this landscape we are still in compromise and trade-offs territory. I would refer to these two papers as insightful demonstrations of these challenges: "Consistency vs. Availability in Distributed Real-Time Systems" - https://ar5iv.labs.arxiv.org/html/2301.08906v1 https://ar5iv.labs.arxiv.org/html/2301.08906v1 "Consistency models in distributed systems: A survey on definitions, disciplines, challenges and applications" - https://ar5iv.labs.arxiv.org/html/1902.03305 https://ar5iv.labs.arxiv.org/html/1902.03305
- dalyons 3y agoWhat database systems are you aware of that take advantage of these modern network advances? Sounds like spanner , but any others?
- SJC_Hacker 3y agoThis may not hold across geographic boundaries, which is the real problem. Like between Asia, Europe, and North America. Its not so much an a complete outage problem, as it is about bandwidth dropping very low on occassion. When you're talking about say, high frequency trading systems, this is a big deal. And anyone who deploys a service with all their data in a single data center, is a pretty small time player, or they don't need the uptime.