15 ms·

This is very interesting. From the little I understand (sorry for using AWS terms as I am more versed with AWS than GCE) this can happen to AWS as well right? e

This is very interesting. From the little I understand (sorry for using AWS terms as I am more versed with AWS than GCE) this can happen to AWS as well right? even if your software is deployed to multiple AZs / multiple regions, if bad routing / network configuration makes it through the various protection mechanisms then basically no amount of redundancy can help if your service is part of the non functional IP block. I mean it seems no matter how redundant you are, there will always be somewhere along the line a single point of failure, even if it has multiple mechanism to prevent it from happening, if all of these mechanisms fail, then it's still a single point. What prevents this from happening at Azure / AWS? Is there anything that general internet routing protocols need to change to prevent it from happening? e.g. I'm sure that we will never hear that Bank of X has transferred a billion dollar to an account but because of propagation errors it published only the credit but didn't finish the debit and now we have two billionaires. This two or more phase commit is pretty much bulletproof in banking as far as I know, and banks are not known to be technologically more advanced than Google, how come internet routing is so prone to errors that can an entire cloud service unavailable for even a small period of time? I'm far from knowing much about networking (although I took some graduate networking courses, I still feel I know practically nothing about it...) So I would appreciate if someone versed in this ELI5 whether it can happen in AWS and Azure regardless of how redundant you are, (which leads to a notion of cross cloud provider redundancy which I'm sure is used in some places) and whether the banking analogy is fair and relevant, and if there are any RFCs to make world-blackout routing nightmares less likely to happen.

poooogles 11y ago
I'm not sure the AWS network follows the same setup, AWS has very distinct blocks between the US/EU/APAC compared to GCP where you can inherit the same IP if you quickly delete/recreate instances in different regions?
- Swannie 11y ago
  I was going to post the same comment too. My understanding, from the odd bits and bobs of information I have, is that AWS regions are typically managed somewhat independently.
ar0 11y ago
Of course things such as mismatched account balances (i.e. the account balance does not equal all credits - debits) or erroneous postings due to bugs happen in the banking IT world. It's just that they are not that visible because only the people affected and that are quick enough in checking their balances will learn about it and after a few hours or days, when they notice the mistake, they will just fix the entries. (And if you thought you were clever and transferred all the funny money away, they are going to sue you to get it back; see e.g. this Quora thread for bank errors and legality: https://www.quora.com/If-my-bank-mistakenly-deposits-1-000-000-into-my-account-am-I-legally-allowed-to-spend-it-before-they-realize-their-mistake https://www.quora.com/If-my-bank-mistakenly-deposits-1-000-0...) EDIT: Also, to answer the question: I think distributed computing is hard. The bank will usually have all their account balances on one huge central mainframe in one location, so you do not need to rely on computers talking to each other. And also, a bank does not really need to publish credits and debits at the same time - they just have to make sure your account is debited at or before the other account is credited (in fact, with most money transfers between banks there will be days between these two). So they can just debit your account, check whether this has worked and then send the money on its journey afterwards and be done with it. If a bug happens and the money does not show up at the recipient, they will complain, the bank can look into it and fix it - no (or not much to the bank, anyways) harm done.
cavisne 11y ago
AWS dont attempt to do anycast, which is annoying but also means this can't happen

poooogles 11y ago

I'm not sure the AWS network follows the same setup, AWS has very distinct blocks between the US/EU/APAC compared to GCP where you can inherit the same IP if you quickly delete/recreate instances in different regions?

Swannie 11y ago
I was going to post the same comment too. My understanding, from the odd bits and bobs of information I have, is that AWS regions are typically managed somewhat independently.

ar0 11y ago

Of course things such as mismatched account balances (i.e. the account balance does not equal all credits - debits) or erroneous postings due to bugs happen in the banking IT world. It's just that they are not that visible because only the people affected and that are quick enough in checking their balances will learn about it and after a few hours or days, when they notice the mistake, they will just fix the entries. (And if you thought you were clever and transferred all the funny money away, they are going to sue you to get it back; see e.g. this Quora thread for bank errors and legality: https://www.quora.com/If-my-bank-mistakenly-deposits-1-000-000-into-my-account-am-I-legally-allowed-to-spend-it-before-they-realize-their-mistake https://www.quora.com/If-my-bank-mistakenly-deposits-1-000-0...) EDIT: Also, to answer the question: I think distributed computing is hard. The bank will usually have all their account balances on one huge central mainframe in one location, so you do not need to rely on computers talking to each other. And also, a bank does not really need to publish credits and debits at the same time - they just have to make sure your account is debited at or before the other account is credited (in fact, with most money transfers between banks there will be days between these two). So they can just debit your account, check whether this has worked and then send the money on its journey afterwards and be done with it. If a bug happens and the money does not show up at the recipient, they will complain, the bank can look into it and fix it - no (or not much to the bank, anyways) harm done.

cavisne 11y ago

AWS dont attempt to do anycast, which is annoying but also means this can't happen