3 ms·

"Why is this important?" Because without multi-region or multi-AZ durability, Wave could lose balances. For example, in the event of a DC fire. "Why can't the

by _vvhw 5y ago

"Why is this important?" Because without multi-region or multi-AZ durability, Wave could lose balances. For example, in the event of a DC fire. "Why can't they get by with a traditional main server plus some backups with a sync" Because without distributed strict serializability, Wave would also be risking the loss of data back to the last backup. However, I'm assuming here that Wave are running consensus around Postgres. That they're not taking any of these chances. But if they're doing it right, then Postgres just doesn't seem like the most boring way to get distributed failover safely, especially when the data is as valuable as account balances, where so much compliance is at stake. Why not simply use a distributed database to begin with?

knorker 5y ago
Not losing transactions with a database is not exactly terra incognita. > Because without distributed strict serializability Depends what you mean by "distributed". A simple 2-phase commit would do it. Or hell, just a write-ahead log on the application layer. > Postgres just doesn't seem like the best way to get distributed failover safely That may be so. But are you arguing in favour of blockchain? When was the last time a real bank forgot your bank balance? They never used blockchains for this.
- _vvhw 5y ago
  "But are you arguing in favour of blockchain?" No, please see my original comment, where I made this clear: "I don't mean to advocate for cryptocurrency in any way." By "modern distributed database", I don't mean blockchain. I just mean "modern distributed database", e.g. things like FoundationDB, Spanner, Aurora or CockroachDB. "A simple 2-phase commit would do it. Or hell, just a write-ahead log on the application layer." Of course, and that leads into my second question, also in my original comment: How are storage faults in the middle of the committed WAL handled? Or is the rest of the committed WAL simply discarded, conflated with a torn write from a system crash? These questions are important when it comes to storing balances safely. Banks can use all kinds of techniques for defense-in-depth (not to suggest they do), they're not limited to Postgres. But a distributed database is a good thing these days, no?
  - knorker 5y ago
    Gotcha. I'm also excited about the modern distributed databases. They're pretty awesome. But yeah, a HN comment is not the right place to design a resilient DB setup. Like I said it's not terra incognita, but it's also not just "spin up a postgres".
    - _vvhw 5y ago
      "Gotcha. I'm also excited about the modern distributed databases. They're pretty awesome." Awesome! "HN comment is not the right place to design a resilient DB setup." Why not? In my experience, HN is always the place to talk about building stuff, running stuff, and how to do that better. :)
      - knorker 5y ago
        Well, one reason is "you're posting often, slow down". I tried to reply yesterday, but was blocked. The gist of my reply yesterday was "depends on requirements, and even they will be huge. And if we just design from made up requirements that's a lot of work to produce a design that nobody will use".
- jimmydorry 5y ago
  Pretty regularly [1], depending on the bank. Every year, there will be several hour-long outages or out right data loss incidents (not all get publicized). https://www.wcjb.com/2020/08/05/bank-of-america-glitch-accounts-show-0-balance/ https://www.wcjb.com/2020/08/05/bank-of-america-glitch-accou... Would you also class going to pay for something and not being able to because the payment network is down as "forgetting your bank balance"? Because those happen even more frequently; the cause being the POS terminal not being able to check your balance, so naievely defaulting it to $0 and giving an error about the network not being available.
  - knorker 5y ago
    Having an outage is not the same as forgetting your bank balance, even if you see a scary $0 when you log in during the outage. But I agree it's definitely a thing that can prevent you accomplishing the thing you want to do with your money, during the outage. But if we're talking "outage" then bitcoin fees going over $25 is probably also fair to classify as an outage as a currency. And that happens too.
    - randomhodler84 5y ago
      High fees In 2017 during an explicit attack on the network with spam, such that the spammer doesn’t have much BTC anymore. I haven’t paid over 1sat/vB for years. It’s the cheapest it can be onchain. Negligible. Ethereum on the other hand is broken with it’s gas calculation model with the unit of account being worth so much. Don’t touch ethereum these days, it’s bad.
      - knorker 5y ago
        2017 is not the only time this has happened. It was $62 in Apr 2021, for example. CC payments and bank transfers are also way faster than bitcoin, even when bitcoin isn't having an outage. In the US bank transfers are slow, for some reason, but in Europe they're not.
        randomhodler84 5y ago
lkrubner 5y ago
> Because without multi-region or multi-AZ durability, Wave could lose balances. For example, in the event of a DC fire. I think you are making a bunch of assumptions that you haven't written down, so it is hard for me to guess what your assumptions are. In the boring, normal, standard setup that people have been using Postgres for in the last 20 years, data does not get lost so long as you have a main server and a sufficient number of backups running in sync, possibly at some distance. A question like "How not to lose data when your main server is destroyed by a fire" is an old question that database engineers have been working on for 40 years, and for which there are many good answers. But I think you're imagining some kind of automatic link between the distributed nature of the blockchain leading to a distributed Postgres. But you would need to write out what your assumptions actually are before I could respond to them.
- _vvhw 5y ago
  "But I think you're imagining some kind of automatic link between the distributed nature of the blockchain leading to a distributed Postgres. But you would need to write out what your assumptions actually are before I could respond to them." I think the assumption here is that I'm advocating blockchain, which I'm definitely not. Please see my original comment.

by _vvhw 5y ago

knorker 5y ago

Not losing transactions with a database is not exactly terra incognita. > Because without distributed strict serializability Depends what you mean by "distributed". A simple 2-phase commit would do it. Or hell, just a write-ahead log on the application layer. > Postgres just doesn't seem like the best way to get distributed failover safely That may be so. But are you arguing in favour of blockchain? When was the last time a real bank forgot your bank balance? They never used blockchains for this.

_vvhw 5y ago
"But are you arguing in favour of blockchain?" No, please see my original comment, where I made this clear: "I don't mean to advocate for cryptocurrency in any way." By "modern distributed database", I don't mean blockchain. I just mean "modern distributed database", e.g. things like FoundationDB, Spanner, Aurora or CockroachDB. "A simple 2-phase commit would do it. Or hell, just a write-ahead log on the application layer." Of course, and that leads into my second question, also in my original comment: How are storage faults in the middle of the committed WAL handled? Or is the rest of the committed WAL simply discarded, conflated with a torn write from a system crash? These questions are important when it comes to storing balances safely. Banks can use all kinds of techniques for defense-in-depth (not to suggest they do), they're not limited to Postgres. But a distributed database is a good thing these days, no?
- knorker 5y ago
  Gotcha. I'm also excited about the modern distributed databases. They're pretty awesome. But yeah, a HN comment is not the right place to design a resilient DB setup. Like I said it's not terra incognita, but it's also not just "spin up a postgres".
  - _vvhw 5y ago
    "Gotcha. I'm also excited about the modern distributed databases. They're pretty awesome." Awesome! "HN comment is not the right place to design a resilient DB setup." Why not? In my experience, HN is always the place to talk about building stuff, running stuff, and how to do that better. :)
    - knorker 5y ago
      Well, one reason is "you're posting often, slow down". I tried to reply yesterday, but was blocked. The gist of my reply yesterday was "depends on requirements, and even they will be huge. And if we just design from made up requirements that's a lot of work to produce a design that nobody will use".
jimmydorry 5y ago
Pretty regularly [1], depending on the bank. Every year, there will be several hour-long outages or out right data loss incidents (not all get publicized). https://www.wcjb.com/2020/08/05/bank-of-america-glitch-accounts-show-0-balance/ https://www.wcjb.com/2020/08/05/bank-of-america-glitch-accou... Would you also class going to pay for something and not being able to because the payment network is down as "forgetting your bank balance"? Because those happen even more frequently; the cause being the POS terminal not being able to check your balance, so naievely defaulting it to $0 and giving an error about the network not being available.
- knorker 5y ago
  Having an outage is not the same as forgetting your bank balance, even if you see a scary $0 when you log in during the outage. But I agree it's definitely a thing that can prevent you accomplishing the thing you want to do with your money, during the outage. But if we're talking "outage" then bitcoin fees going over $25 is probably also fair to classify as an outage as a currency. And that happens too.
  - randomhodler84 5y ago
    High fees In 2017 during an explicit attack on the network with spam, such that the spammer doesn’t have much BTC anymore. I haven’t paid over 1sat/vB for years. It’s the cheapest it can be onchain. Negligible. Ethereum on the other hand is broken with it’s gas calculation model with the unit of account being worth so much. Don’t touch ethereum these days, it’s bad.
    - knorker 5y ago
      2017 is not the only time this has happened. It was $62 in Apr 2021, for example. CC payments and bank transfers are also way faster than bitcoin, even when bitcoin isn't having an outage. In the US bank transfers are slow, for some reason, but in Europe they're not.
      - randomhodler84 5y ago

lkrubner 5y ago

> Because without multi-region or multi-AZ durability, Wave could lose balances. For example, in the event of a DC fire. I think you are making a bunch of assumptions that you haven't written down, so it is hard for me to guess what your assumptions are. In the boring, normal, standard setup that people have been using Postgres for in the last 20 years, data does not get lost so long as you have a main server and a sufficient number of backups running in sync, possibly at some distance. A question like "How not to lose data when your main server is destroyed by a fire" is an old question that database engineers have been working on for 40 years, and for which there are many good answers. But I think you're imagining some kind of automatic link between the distributed nature of the blockchain leading to a distributed Postgres. But you would need to write out what your assumptions actually are before I could respond to them.

_vvhw 5y ago
"But I think you're imagining some kind of automatic link between the distributed nature of the blockchain leading to a distributed Postgres. But you would need to write out what your assumptions actually are before I could respond to them." I think the assumption here is that I'm advocating blockchain, which I'm definitely not. Please see my original comment.