3 ms·

I think a lot of its probably technical debt. So much internally still relies on legacy systems in US-East-1, and every time this happens I'm sure theres a disc

by esskay 1y ago

I think a lot of its probably technical debt. So much internally still relies on legacy systems in US-East-1, and every time this happens I'm sure theres a discussion internally about decoupling that reliance which then turns into a massive diagram that looks like a family tree dating back a thousand years of all the things that need to be changed to stop it happening.

abujazar 1y ago
Sounds plausible. It's also a "fat and happy" symptom not to be able to fix deep underlying issues despite an ever growing pile of cash in the company.
- dv_dt 1y ago
  Fixing deep underlying issues tends to fare poorly on performance reviews because success is not an easily traceable victory event. It is the prolonged absence of events like this, and it's hard to prove a negative.
Nextgrid 1y ago
There's also the issue of sometimes needing actual strong consistency. Things like auth or billing for example where you absolutely can't tolerate eventual consistency or split-brain situations, in which case you need one region to serve as the ultimate source of truth.
- kassner 1y ago
  > billing […] can't tolerate eventual consistency Interesting point that banks actually tolerate a lot more eventual consistency than most software that just use a billing backend ever do. Stuff like 503-ing a SaaS request because the billing system was down and you couldn’t check for limits, could absolutely be locally cached and eventual consistency would hurt very little. Unless your cost is quite high, I would much rather prefer to keep the API up and deal with the over-usage later.
- chmod775 1y ago
  Banking/transactions is full of split-brains where everyone involved prays for eventual consistency. If you check out with a credit card, even if everything looked good then, the seller might not see the money for days or might never receive it at all.
- smj-edison 1y ago
  Interestingly, TigerBeetle manages to have distributed strict consistency over 6 machines.
- mdavidn 1y ago
  Banking is full of examples of eventually consistent systems. ACH, credit card transactions, blockchain...

by esskay 1y ago

abujazar 1y ago

Sounds plausible. It's also a "fat and happy" symptom not to be able to fix deep underlying issues despite an ever growing pile of cash in the company.

dv_dt 1y ago
Fixing deep underlying issues tends to fare poorly on performance reviews because success is not an easily traceable victory event. It is the prolonged absence of events like this, and it's hard to prove a negative.

Nextgrid 1y ago

There's also the issue of sometimes needing actual strong consistency. Things like auth or billing for example where you absolutely can't tolerate eventual consistency or split-brain situations, in which case you need one region to serve as the ultimate source of truth.

kassner 1y ago
> billing […] can't tolerate eventual consistency Interesting point that banks actually tolerate a lot more eventual consistency than most software that just use a billing backend ever do. Stuff like 503-ing a SaaS request because the billing system was down and you couldn’t check for limits, could absolutely be locally cached and eventual consistency would hurt very little. Unless your cost is quite high, I would much rather prefer to keep the API up and deal with the over-usage later.
chmod775 1y ago
Banking/transactions is full of split-brains where everyone involved prays for eventual consistency. If you check out with a credit card, even if everything looked good then, the seller might not see the money for days or might never receive it at all.
smj-edison 1y ago
Interestingly, TigerBeetle manages to have distributed strict consistency over 6 machines.
mdavidn 1y ago
Banking is full of examples of eventually consistent systems. ACH, credit card transactions, blockchain...