4 ms·
> The culprit was insufficient testing The culprit was not insufficient testing! The culprit was insufficient testing, plus lacks of restorable backups, on mu
by lsb 7y ago
> The culprit was insufficient testing
The culprit was not insufficient testing!
The culprit was insufficient testing, plus lacks of restorable backups, on multiple levels.
Last level of backup: there's 5 million records, use a few dozen reams of paper and print out the account totals for every single account.
For all of the complex systems you can think of (planes, living organisms, etc), the reason things usually go so well is that there are multiple levels of checks and balances. Everything is usually veering towards entropy, and fail-safe systems try to ensure when things go wrong that they fail safely.
- 3fe9a03ccd14ca5 7y agoThe problem was insufficient technical leadership and engineering rigor across the board.
- ars 7y agoIt's not possible to restore from backup when the transaction is between your bank and a different bank. You might restore from backup, but the other bank won't. The issues here are far more complex than just keeping track of the current balance: a first year CS student could do that part.
- mcny 7y agoWait. I don’t understand. If you have the opening balance and have all the transaction logs, what are you missing? I don’t see why it shouldn’t be possible to feed data to both the new system and the old system for a while...
- jyounker 7y agoBut that requires forethought, planning, and a dedication to testing.
- Forge36 7y agoFrom the article it appears even the naive first year approach would have significantly reduced the scope of the problem.
- LoSboccacc 7y agoBanks handle balance discrepancies all the time all day long, all operations that cross a bank boundary are subject to reconciliation further down the line, with massive expert systems handling most of the obvious cases automatically and operators handling the flagged transactions
- bjohnson225 7y agoBlaming testing feels like a cop-out - it's easier to say 'if we tested a little more everything would have been fine' rather than 'our leadership was too incompotent to recognise that the plan was fundamentally flawed from day one, that releasing in the way we did maximized the risk to customers, or that we decided not to listen to any concerns which would have compromised the timeline'
- coleca 7y agoExactly. Had they done all the proper testing, they would have likely discovered all these issues, it's not a stretch to think that the management that allowed the project to get to the state it was in would have also pushed forward with the go-live despite the bugs that were discovered in testing. And just because they allocated time to testing doesn't mean they would have allocated sufficient time to remediation and re-testing. I've seen it happen where bugs were found late stage in a big-bang style migration, they were remediated, but there wasn't enough time left to conduct another massive mock go-live to make sure it worked. So you hope and pray...
- darkerside 7y agoTesting and restoring backups are both mitigations of problems, not the actual problems.