8 ms·

You Can’t Sacrifice Partition Tolerance (2010)

93 points by xk3 5y ago

fred256 5y ago
(2010)
- dang 5y ago
  Added. Thanks!
RcouF1uZ4gsC 5y ago
The OP is correct in theory, but not in practice. If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. See https://cloud.google.com/blog/products/databases/inside-cloud-spanner-and-the-cap-theorem https://cloud.google.com/blog/products/databases/inside-clou... Notice who the author is: Eric Brewer
- lostcolony 5y ago
  Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?)
  - linuxhansl 5y ago
    You try another connection (after some timeout)? Or you start N requests in parallel and then pick the first response?
    - whateveracct 5y ago
      aren't retry policies inherently compromising on "A"? they fail and cause things to be slow when they take effect
    - lostcolony 5y ago
      "After some timeout" - exactly. During that time you effectively have a partition; you have either accepted the write, meaning you are no longer consistent, or you are blocking on the write, meaning you are no longer available. If you have multiple, let's expand out to 3 instances (not necessary, but helps illustrate some of the issues better). A tells B something 10x. A also tells C something 10x. Even assuming you don't get inconsistent responses (i.e., B has both accepted and rejected the write), what happens if B accepts it, and C rejects it? Likewise, if you have high load, you're going to have race conditions leading to lack of consistency as well (A tells B and C x = 2, meanwhile C tells A and B that x = 3). You are favoring availability > consistency.
  - throwaway3699 5y ago
    Then you have downtime. If the downtime is small enough, then the trade-off can be worthwhile.
    - owenmarshall 5y ago
      You sacrificed A.
      - throwaway3699 5y ago
        So does Spanner. The point is that Spanner has so little downtime that it's not really an issue.
        lostcolony 5y ago
        Spanner is a strongly consistent system leveraging Paxos. Yes, with sufficient redundancy you can reduce the likelihood of a partition. You still want to know what the guarantees of the system are. Spanner guarantees that if a transaction can't be agreed to by a quorum of nodes, it will be rejected. Ergo, favoring consistency. That guarantee matters, no matter how infrequent partitions are.
    - lostcolony 5y ago
      So...not available.
  - toast0 5y ago
    > Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?) If you really need things to arrive, you'd be sending them on more than one connection. Of course, the difficulty with redundant fiber connections is that really the only way to determine if redundant fiber connections are in the same bundle is with a backhoe.
- linuxhansl 5y ago
  Not sure why you got downvoted. I came here to say the same. While it is in theory true that you cannot have a CA system, in practice if you control your H/W and data centers as well, you can build enough redundancy that a P is not more likely than (say) being hit by a meteor. So still not a CA system, but practically CA 99.9..% (add as many 9's as you have money). Edit: Spelling.
  - __jem 5y ago
    99.9 isn't "A" though -- that's something like 10 hours of downtime per year. Even at 6 9s, 30 seconds is plenty of time to end up in a split brain situation!
- deleted 5y ago
  [deleted]
- redis_mlc 5y ago
  > If you have a lot of money and are willing and able to build a private network with enough redundancy Google has done this. Possibly AT&T also. > Notice who the author is: Eric Brewer Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes. As a DBA, I find CAP totally inadequate for any business purpose. Try explaining to a business owner, "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?"
  - packetslave 5y ago
    Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes Teacher, academic, and vice president of infrastructure at Google. He wrote about the CAP implications of Spanner here as well: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-...
  - Dylan16807 5y ago
    I think you're conflating different things there. If you built a highly redundant network, and it still partitioned, then the actual problem is that you somehow lost 80% of your lines to the internet at the same time. So your response back needs to be that your HA setup kept things running for a while but it would take actual magic to keep going once just about all the servers are unplugged simultaneously. That's not a problem with CAP, that's a problem with explaining the wrong part of the issue.
  - lostcolony 5y ago
    >>> "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?" "All that money spent on HA was what allowed us to accept purchases -at all-"
- lmm 5y ago
  > If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. Actually I'd say this is something that's correct in theory but not in practice. In theory you can build a network so redundant that you will never experience a partition. In practice you will always have bugs in your redundancy arrangement, operational errors by sysadmins, and so on. > If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. You still have to choose whether to drop C, A, or both when P does happen. Five 9s means about an hour of downtime a year; maybe for some businesses that's small enough to be irrelevant, but by no means all.
deleted 5y ago
[deleted]
eloff 5y ago
You can make partition tolerance arbitrarily low in practice with enough work and resources. But you can never eliminate it. Note it is not just network failures here, because a dead machine and an inaccessible machine are both unavailable. So in theory you have to choose in the event of P do you sacrifice A or C. That's CAP in a nutshell. You have to plan for P so choose your failure mode in that case. And this is where people went wrong. They figured you can't sacrifice availability, so then you can't have consistency. If you can't have consistency in failure modes, you may as well not have it at all. That's really dumb, because in reality you're always going to have some downtime. Most use cases can afford a little downtime, rather just work to keep it to acceptable levels. By sacrificing consistency you enter a realm where logic and cause and effect no longer apply - it's hell to program with that model - and the resulting bugs will cost you availability anyway. CP should be the default choice.
- zffr 5y ago
  > By sacrificing consistency you enter a realm where logic and cause and effect no longer apply - it's hell to program with that model - and the resulting bugs will cost you availability anyway. Sacrificing strong consistency doesn’t mean your system won’t be eventually consistent. CRDTs are a great example of how you can get availability and simple consistency semantics.
  - eloff 5y ago
    Eventually consistent generally means things can be all over the place. You can temporarily see extremely weird and unintuitive things and then modify data based on that incorrect world view. That means eventually consistent turns into permanently inconsistent in unpredictable ways. CRDTs are interesting, but over hyped. They only work for very simple situations and very limited data structures.
    - lostcolony 5y ago
      Eventual consistency is a stronger guarantee than you get with a remote git repo. It actually works fine for a large number of use cases. CRDTs are just a way of implementing eventual consistency in provably predictable and 'reasonable' manners. I will agree they are very limited data structures, but I disagree that they only work for very simple situations. I'd liken them to an immutable list in FP. Very limited data structure, but can be used cleverly to create very complicated ones that maintain the same invariants.
deleted 5y ago
[deleted]
dang 5y ago
Some past threads: You Can’t Sacrifice Partition Tolerance (2010) - https://news.ycombinator.com/item?id=11700856 https://news.ycombinator.com/item?id=11700856 - May 2016 (21 comments) You Can’t Sacrifice Partition Tolerance (2010) - https://news.ycombinator.com/item?id=8214310 https://news.ycombinator.com/item?id=8214310 - Aug 2014 (4 comments) You Can't Sacrifice Partition Tolerance (response to Stonebraker) - https://news.ycombinator.com/item?id=1821887 https://news.ycombinator.com/item?id=1821887 - Oct 2010 (13 comments) You Can't Sacrifice Partition Tolerance - https://news.ycombinator.com/item?id=1768312 https://news.ycombinator.com/item?id=1768312 - Oct 2010 (50 comments)
- chii 5y ago
  i wish that HN can actually aggregate these old threads together in the resubmission directly (and automagically?). And may be randomize the comment sorting so that the discussion can continue from the existing comments, rather than a rehash.
  - froh 5y ago
    regarding the old comment sorting, I wonder if recent comments percolate the commented-on comment up anyhow? p-c1-c2 would an activity on C2 influence the sorting of p amongst its peers?
    - chii 5y ago
      depends - but i often see a single, long thread of comments from the highest "voted" comment. this makes it hard to discover lower ranked comments as it's now below the fold, and thus less likely to get votes either. That's why i suggested a random sort, so that each top-level comment has a chance of getting visiblity and thus, remove groupthink and biases from discussion. Of course, the top voted comments would have value - so that's why i suggested only doing this random sorting from aggregated old submissions, rather than for all submissions.
      - froh 5y ago
jimsimmons 5y ago
Doesn’t a distributed consensus protocol solve this exact problem?
forty 5y ago
Good article but I'm still struggling to understand CAP in practice. In general, the whole distributed system will include some kind of database and some kind of application servers. What would AP mean in this context? That application server nodes are able to respond meaningful (ie non failing) responses even if they are unable to communicate with the DB? Wouldn't that require every application node to have a whole local copy of the data it requires to work?
- TBurette 5y ago
  The CAP theorem applies whenever you have a distributed data store. This means a system you can write data to and read back from. This system is made of multiple nodes connected through a network and the clients can connect and make read/write requests from any node. If you don't meet those conditions CAP doesn't apply to you. For example : - an SQL database with master/slave replication but all the clients only ever access the database from the same nod (the clients will always have a consistent view of the one node they access) - multiple processes on the same machine doesn't apply (it's a concurrent datastore but not a distributed one) - a single SQL database instance doesn't apply (data is not distributed even if there are multiple clients. Confusingly enough if clients can keep some data and work in offline mode, then it would apply because the clients would be considered as a node of the system) CAP was clearly created in the context of distributed database : SQL, noSQL, CDN,... Can it apply to an entire system made of multiple components such as database, multiple services,... ? Things get muddier but I'd say yes as long as you check all the boxes: a system with data spread on multiple nodes that clients can access to where the connection can get severed. Application servers with no local copy losing connection to the server : no. Where is the distribution of data? Where can a network failure partition the data? There is not much A or C choice to make when a partition occurs. Applying CAP wouldn't be very interesting here. However application servers with local copy : yes CAP applies. In case of a partition there is a trade-off to be made between C and A. To complicate things further, CAP has basically two versions. There is Brewer's version which talks in general terms is not formal and gets hazy when you drill down to the details. The other version is the one of Lynchs's Paper that provided a proof for the CAP theorem. In that version the definitions are very strict and consequently many real worlds aspects don't fit. Depending on who you ask you may get different answers. For example, take the context of a database distributed on multiple servers in a datacenter with a reverse proxy that can detect partitions and always redirect read/write queries to non-partitioned nodes. In that context you could say that you have all three C, A and P because there is never going to be a request on a partitioned node. According to Lynch's version of CAP this is not C consistent because you have non-failing nodes that can read stale data (the question of whether there is going to be any request or not is outside the purview of the proof).
tracnar 5y ago
What's often missing from these discussions is that the trade off in the CAP theorem can change across the system and over time. If you have a cache on your client, you're trading off consistency for availability, same if you do an optimistic update on the client-side. But that does not change the consistency of your database core. You can also "downgrade" to eventual consistency if you detect a network problem. I feel that a more principled way to navigate between these trade offs and awareness from the application and database of the current consistency guarantees of the data would help to make more robust systems. E.g. you'd want strong consistency by default, with a fallback to strong eventual consistency when the network is poor and the use-case allows it (e.g. not for changing your password, but ok for posting a comment). It needs to be reflected in the UI, and ideally you want the trade off to be decided at each level in a consistent way (client, local database, core database, ...).
GuB-42 5y ago
The title should be more explicitly, "You can't sacrifice partition tolerance in a distributed system". But you can if you don't have a distributed system. I think it is a good argument for monoliths, "real" non-cloud servers and mainframes. In reality, there are usually backups, and hardware is not perfect so the whole system is most likely somewhat CP, but having everything on the same machine means you have less to worry about P in your application. Edit: BTW, I know that monoliths don't scale, but you don't always need to scale, or not yet.
- deegles 5y ago
  I think about this a lot. Amazon has (among others) the m5zn.12xlarge EC2 instance with 48 vCPUs, 192GB memory, 100gbps network and 19gbps EBS bandwidth... you could run a heck of a monolith before needing to scale horizontally.

93 points by xk3 5y ago

fred256 5y ago

(2010)

dang 5y ago
Added. Thanks!

RcouF1uZ4gsC 5y ago

The OP is correct in theory, but not in practice. If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. See https://cloud.google.com/blog/products/databases/inside-cloud-spanner-and-the-cap-theorem https://cloud.google.com/blog/products/databases/inside-clou... Notice who the author is: Eric Brewer

lostcolony 5y ago
Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?)
- linuxhansl 5y ago
  You try another connection (after some timeout)? Or you start N requests in parallel and then pick the first response?
  - whateveracct 5y ago
    aren't retry policies inherently compromising on "A"? they fail and cause things to be slow when they take effect
  - lostcolony 5y ago
    "After some timeout" - exactly. During that time you effectively have a partition; you have either accepted the write, meaning you are no longer consistent, or you are blocking on the write, meaning you are no longer available. If you have multiple, let's expand out to 3 instances (not necessary, but helps illustrate some of the issues better). A tells B something 10x. A also tells C something 10x. Even assuming you don't get inconsistent responses (i.e., B has both accepted and rejected the write), what happens if B accepts it, and C rejects it? Likewise, if you have high load, you're going to have race conditions leading to lack of consistency as well (A tells B and C x = 2, meanwhile C tells A and B that x = 3). You are favoring availability > consistency.
- throwaway3699 5y ago
  Then you have downtime. If the downtime is small enough, then the trade-off can be worthwhile.
  - owenmarshall 5y ago
    You sacrificed A.
    - throwaway3699 5y ago
      So does Spanner. The point is that Spanner has so little downtime that it's not really an issue.
      - lostcolony 5y ago
        Spanner is a strongly consistent system leveraging Paxos. Yes, with sufficient redundancy you can reduce the likelihood of a partition. You still want to know what the guarantees of the system are. Spanner guarantees that if a transaction can't be agreed to by a quorum of nodes, it will be rejected. Ergo, favoring consistency. That guarantee matters, no matter how infrequent partitions are.
  - lostcolony 5y ago
    So...not available.
- toast0 5y ago
  > Redundancy doesn't solve CAP. Even with a dozen connections you can end up with issues (A sent a packet, B received it, the connection you pick dropped the ack; what happens?) If you really need things to arrive, you'd be sending them on more than one connection. Of course, the difficulty with redundant fiber connections is that really the only way to determine if redundant fiber connections are in the same bundle is with a backhoe.
linuxhansl 5y ago
Not sure why you got downvoted. I came here to say the same. While it is in theory true that you cannot have a CA system, in practice if you control your H/W and data centers as well, you can build enough redundancy that a P is not more likely than (say) being hit by a meteor. So still not a CA system, but practically CA 99.9..% (add as many 9's as you have money). Edit: Spelling.
- __jem 5y ago
  99.9 isn't "A" though -- that's something like 10 hours of downtime per year. Even at 6 9s, 30 seconds is plenty of time to end up in a split brain situation!
deleted 5y ago
[deleted]
redis_mlc 5y ago
> If you have a lot of money and are willing and able to build a private network with enough redundancy Google has done this. Possibly AT&T also. > Notice who the author is: Eric Brewer Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes. As a DBA, I find CAP totally inadequate for any business purpose. Try explaining to a business owner, "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?"
- packetslave 5y ago
  Note that Professor Brewer is a teacher/academic, and CAP is intended for instructional purposes Teacher, academic, and vice president of infrastructure at Google. He wrote about the CAP implications of Spanner here as well: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf https://storage.googleapis.com/pub-tools-public-publication-...
- Dylan16807 5y ago
  I think you're conflating different things there. If you built a highly redundant network, and it still partitioned, then the actual problem is that you somehow lost 80% of your lines to the internet at the same time. So your response back needs to be that your HA setup kept things running for a while but it would take actual magic to keep going once just about all the servers are unplugged simultaneously. That's not a problem with CAP, that's a problem with explaining the wrong part of the issue.
- lostcolony 5y ago
  >>> "Yeah, when our network partitioned, we couldn't accept purchases in both partitions." Their response will be, "So all that money you spent on HA was really based on a lie?" "All that money spent on HA was what allowed us to accept purchases -at all-"
lmm 5y ago
> If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. Actually I'd say this is something that's correct in theory but not in practice. In theory you can build a network so redundant that you will never experience a partition. In practice you will always have bugs in your redundancy arrangement, operational errors by sysadmins, and so on. > If you have a lot of money and are willing and able to build a private network with enough redundancy, you can have both C and A in practice, and P will be low enough not to practically affect the service. You still have to choose whether to drop C, A, or both when P does happen. Five 9s means about an hour of downtime a year; maybe for some businesses that's small enough to be irrelevant, but by no means all.

deleted 5y ago

[deleted]

eloff 5y ago

You can make partition tolerance arbitrarily low in practice with enough work and resources. But you can never eliminate it. Note it is not just network failures here, because a dead machine and an inaccessible machine are both unavailable. So in theory you have to choose in the event of P do you sacrifice A or C. That's CAP in a nutshell. You have to plan for P so choose your failure mode in that case. And this is where people went wrong. They figured you can't sacrifice availability, so then you can't have consistency. If you can't have consistency in failure modes, you may as well not have it at all. That's really dumb, because in reality you're always going to have some downtime. Most use cases can afford a little downtime, rather just work to keep it to acceptable levels. By sacrificing consistency you enter a realm where logic and cause and effect no longer apply - it's hell to program with that model - and the resulting bugs will cost you availability anyway. CP should be the default choice.

zffr 5y ago
> By sacrificing consistency you enter a realm where logic and cause and effect no longer apply - it's hell to program with that model - and the resulting bugs will cost you availability anyway. Sacrificing strong consistency doesn’t mean your system won’t be eventually consistent. CRDTs are a great example of how you can get availability and simple consistency semantics.
- eloff 5y ago
  Eventually consistent generally means things can be all over the place. You can temporarily see extremely weird and unintuitive things and then modify data based on that incorrect world view. That means eventually consistent turns into permanently inconsistent in unpredictable ways. CRDTs are interesting, but over hyped. They only work for very simple situations and very limited data structures.
  - lostcolony 5y ago
    Eventual consistency is a stronger guarantee than you get with a remote git repo. It actually works fine for a large number of use cases. CRDTs are just a way of implementing eventual consistency in provably predictable and 'reasonable' manners. I will agree they are very limited data structures, but I disagree that they only work for very simple situations. I'd liken them to an immutable list in FP. Very limited data structure, but can be used cleverly to create very complicated ones that maintain the same invariants.

deleted 5y ago

[deleted]

dang 5y ago

Some past threads: You Can’t Sacrifice Partition Tolerance (2010) - https://news.ycombinator.com/item?id=11700856 https://news.ycombinator.com/item?id=11700856 - May 2016 (21 comments) You Can’t Sacrifice Partition Tolerance (2010) - https://news.ycombinator.com/item?id=8214310 https://news.ycombinator.com/item?id=8214310 - Aug 2014 (4 comments) You Can't Sacrifice Partition Tolerance (response to Stonebraker) - https://news.ycombinator.com/item?id=1821887 https://news.ycombinator.com/item?id=1821887 - Oct 2010 (13 comments) You Can't Sacrifice Partition Tolerance - https://news.ycombinator.com/item?id=1768312 https://news.ycombinator.com/item?id=1768312 - Oct 2010 (50 comments)

chii 5y ago
i wish that HN can actually aggregate these old threads together in the resubmission directly (and automagically?). And may be randomize the comment sorting so that the discussion can continue from the existing comments, rather than a rehash.
- froh 5y ago
  regarding the old comment sorting, I wonder if recent comments percolate the commented-on comment up anyhow? p-c1-c2 would an activity on C2 influence the sorting of p amongst its peers?
  - chii 5y ago
    depends - but i often see a single, long thread of comments from the highest "voted" comment. this makes it hard to discover lower ranked comments as it's now below the fold, and thus less likely to get votes either. That's why i suggested a random sort, so that each top-level comment has a chance of getting visiblity and thus, remove groupthink and biases from discussion. Of course, the top voted comments would have value - so that's why i suggested only doing this random sorting from aggregated old submissions, rather than for all submissions.
    - froh 5y ago

jimsimmons 5y ago

Doesn’t a distributed consensus protocol solve this exact problem?

forty 5y ago

Good article but I'm still struggling to understand CAP in practice. In general, the whole distributed system will include some kind of database and some kind of application servers. What would AP mean in this context? That application server nodes are able to respond meaningful (ie non failing) responses even if they are unable to communicate with the DB? Wouldn't that require every application node to have a whole local copy of the data it requires to work?

TBurette 5y ago
The CAP theorem applies whenever you have a distributed data store. This means a system you can write data to and read back from. This system is made of multiple nodes connected through a network and the clients can connect and make read/write requests from any node. If you don't meet those conditions CAP doesn't apply to you. For example : - an SQL database with master/slave replication but all the clients only ever access the database from the same nod (the clients will always have a consistent view of the one node they access) - multiple processes on the same machine doesn't apply (it's a concurrent datastore but not a distributed one) - a single SQL database instance doesn't apply (data is not distributed even if there are multiple clients. Confusingly enough if clients can keep some data and work in offline mode, then it would apply because the clients would be considered as a node of the system) CAP was clearly created in the context of distributed database : SQL, noSQL, CDN,... Can it apply to an entire system made of multiple components such as database, multiple services,... ? Things get muddier but I'd say yes as long as you check all the boxes: a system with data spread on multiple nodes that clients can access to where the connection can get severed. Application servers with no local copy losing connection to the server : no. Where is the distribution of data? Where can a network failure partition the data? There is not much A or C choice to make when a partition occurs. Applying CAP wouldn't be very interesting here. However application servers with local copy : yes CAP applies. In case of a partition there is a trade-off to be made between C and A. To complicate things further, CAP has basically two versions. There is Brewer's version which talks in general terms is not formal and gets hazy when you drill down to the details. The other version is the one of Lynchs's Paper that provided a proof for the CAP theorem. In that version the definitions are very strict and consequently many real worlds aspects don't fit. Depending on who you ask you may get different answers. For example, take the context of a database distributed on multiple servers in a datacenter with a reverse proxy that can detect partitions and always redirect read/write queries to non-partitioned nodes. In that context you could say that you have all three C, A and P because there is never going to be a request on a partitioned node. According to Lynch's version of CAP this is not C consistent because you have non-failing nodes that can read stale data (the question of whether there is going to be any request or not is outside the purview of the proof).

tracnar 5y ago

What's often missing from these discussions is that the trade off in the CAP theorem can change across the system and over time. If you have a cache on your client, you're trading off consistency for availability, same if you do an optimistic update on the client-side. But that does not change the consistency of your database core. You can also "downgrade" to eventual consistency if you detect a network problem. I feel that a more principled way to navigate between these trade offs and awareness from the application and database of the current consistency guarantees of the data would help to make more robust systems. E.g. you'd want strong consistency by default, with a fallback to strong eventual consistency when the network is poor and the use-case allows it (e.g. not for changing your password, but ok for posting a comment). It needs to be reflected in the UI, and ideally you want the trade off to be decided at each level in a consistent way (client, local database, core database, ...).

GuB-42 5y ago

The title should be more explicitly, "You can't sacrifice partition tolerance in a distributed system". But you can if you don't have a distributed system. I think it is a good argument for monoliths, "real" non-cloud servers and mainframes. In reality, there are usually backups, and hardware is not perfect so the whole system is most likely somewhat CP, but having everything on the same machine means you have less to worry about P in your application. Edit: BTW, I know that monoliths don't scale, but you don't always need to scale, or not yet.

deegles 5y ago
I think about this a lot. Amazon has (among others) the m5zn.12xlarge EC2 instance with 48 vCPUs, 192GB memory, 100gbps network and 19gbps EBS bandwidth... you could run a heck of a monolith before needing to scale horizontally.