4 ms·

Durability is also overrated. Jokes aside, I’ve been responsible for a system that processed ~1 billion monetary transactions per day. Even with a fanatical fo

Durability is also overrated. Jokes aside, I’ve been responsible for a system that processed ~1 billion monetary transactions per day. Even with a fanatical focus on consistency and correctness it was still always off. With the philosophy promoted on the OP the chaos would have been complete… is my gut feeling at least.

mousetree 4y ago
Out of interest, what system processed 1 billion monetary transactions a day?
- bjornsing 4y ago
  Bookkeeping / billing / analytics system for a high volume online API.
- playingalong 4y ago
  High-frequency something would be my guess;)
vidarh 4y ago
Durability often is overrated. E.g. lets say your system is storing billing records for an API. Customers won't complain about being under-charged, so a loss of data for a system like that is fine as long as reducing the durability saves more money than it costs in losts ability to charge your customers. E.g. if rolling up and syncing summaries to disk every 10 seconds were to on average lose you 0.1% of records but cut your server cost for that system by 80%, it's a simple question of whether those server costs are higher or lower than .1% of your revenue (all numbers taken out of thin air). We make sacrifices in terms of both consistency and durability and all kinds of things all the time. The problem is that we often don't take the time to analyse what kind of inconsistency and loss is acceptable from a business point of view before designing a solution that often tries to aim for a level of consistency or durability that may not be needed. You can afford to sacrifice all kinds of guarantees as long as you have reliable models for how sacrificing them impacts you and your customers so that you know the sacrifices represent acceptable tradeoffs. The risk you're taking is that you're sacrificing safety margins, and so it's something you should only do if you understand both the domain and your systems failure modes very well.

mousetree 4y ago

Out of interest, what system processed 1 billion monetary transactions a day?

bjornsing 4y ago
Bookkeeping / billing / analytics system for a high volume online API.
playingalong 4y ago
High-frequency something would be my guess;)

vidarh 4y ago

Durability often is overrated. E.g. lets say your system is storing billing records for an API. Customers won't complain about being under-charged, so a loss of data for a system like that is fine as long as reducing the durability saves more money than it costs in losts ability to charge your customers. E.g. if rolling up and syncing summaries to disk every 10 seconds were to on average lose you 0.1% of records but cut your server cost for that system by 80%, it's a simple question of whether those server costs are higher or lower than .1% of your revenue (all numbers taken out of thin air). We make sacrifices in terms of both consistency and durability and all kinds of things all the time. The problem is that we often don't take the time to analyse what kind of inconsistency and loss is acceptable from a business point of view before designing a solution that often tries to aim for a level of consistency or durability that may not be needed. You can afford to sacrifice all kinds of guarantees as long as you have reliable models for how sacrificing them impacts you and your customers so that you know the sacrifices represent acceptable tradeoffs. The risk you're taking is that you're sacrificing safety margins, and so it's something you should only do if you understand both the domain and your systems failure modes very well.