4 ms·

How do you know it wasn’t the IT system that caused the problem?

by wil421 6y ago

DaiPlusPlus 6y ago
That presumes that the "IT system" is capable of making huge actions by itself (or by itself via a bug or operator error): that immediately makes alarm-bells ring in my head because it says the IT system's security/authorization system isn't operating on the basis of least-privilege. There are ways to engineer automated processes to prevent things like this from ever happening, for example, all transactions over, say, $10m, could be required to be signed using PKI keys held only on smartcards physically held by those involved - and the big-fat-table that holds transactions in their IBM Z-series would be automatically audited to undo any transactions lacking the necessary signatures. And all of this should be evident in the printed hardcopy that the gov-level auditors would love to see. ...and that's just an idea off the top of my head right now at 5am (and a bottle of IPA) and I've never worked in fintech or banking. Given that shady stuff happens at the highest-levels inside legacy banking institutions (e.g. HSBC and the South American drugs trade - or Deutsche Bank and sanctioned Russian entities... and the current White House occupant) - and because I know that these banks do hire great engineers for their internal systems - that "clerical errors" don't happen like this - something fishy is afoot and I guarantee that we'll never know the truth.
- sukilot 6y ago
  Is it so hard to imagine that an IT system would have bugs?
  - mschuster91 6y ago
    I (and for what it's worth also regulators) expect that a bank has procedures in place to catch bugs that lead to such "mistakes".
  - DaiPlusPlus 6y ago
    By analogy: think about how a preemptive multitasking OS with protected virtual memory means that a misbehaving user process cannot bring down the machine: the kinds of bugs in user programs in the time of DOS or Windows 3.0 that would destroy the world (e.g. a user process overwriting kernel memory, or a single-threaded user process failing to yield to the kernel) just can’t happen with a modern kernel. The same kinds of security and safety guarantees that stem from the overall system architecture can exist in banking systems - that’s my point. So yes, until everyone switches to Haskell we will always have bugs - but the kinds of bugs can be limited - as can the scale of their impact - with good system design.
- oasisbob 6y ago
  The problem, having worked in banking, is that the core system is separate from the loans system, almost certainly. More complexity. Probably some semi-automated batch handling which is far more brittle and janky than anyone suspects. But regardless, at some point, this system still thinks the transaction has been authorized. I'm sure there's a huge audit log showing who/where/why the mistake was made. But I've also seen end-of-period processing at a smaller financial institution where humans are involved and a few spreadsheets hairpin their ways through different systems with lots of manual intervention and QA. One step was literally very close to "audit 0.01% of payments before sending the CSV into the lockbox escrow processing system...

How do you know it wasn’t the IT system that caused the problem?

by wil421 6y ago

How do you know it wasn’t the IT system that caused the problem?

That presumes that the "IT system" is capable of making huge actions by itself (or by itself via a bug or operator error): that immediately makes alarm-bells ring in my head because it says the IT system's security/authorization system isn't operating on the basis of least-privilege. There are ways to engineer automated processes to prevent things like this from ever happening, for example, all transactions over, say, $10m, could be required to be signed using PKI keys held only on smartcards physically held by those involved - and the big-fat-table that holds transactions in their IBM Z-series would be automatically audited to undo any transactions lacking the necessary signatures. And all of this should be evident in the printed hardcopy that the gov-level auditors would love to see. ...and that's just an idea off the top of my head right now at 5am (and a bottle of IPA) and I've never worked in fintech or banking. Given that shady stuff happens at the highest-levels inside legacy banking institutions (e.g. HSBC and the South American drugs trade - or Deutsche Bank and sanctioned Russian entities... and the current White House occupant) - and because I know that these banks do hire great engineers for their internal systems - that "clerical errors" don't happen like this - something fishy is afoot and I guarantee that we'll never know the truth.

sukilot 6y ago
Is it so hard to imagine that an IT system would have bugs?
- mschuster91 6y ago
  I (and for what it's worth also regulators) expect that a bank has procedures in place to catch bugs that lead to such "mistakes".
- DaiPlusPlus 6y ago
  By analogy: think about how a preemptive multitasking OS with protected virtual memory means that a misbehaving user process cannot bring down the machine: the kinds of bugs in user programs in the time of DOS or Windows 3.0 that would destroy the world (e.g. a user process overwriting kernel memory, or a single-threaded user process failing to yield to the kernel) just can’t happen with a modern kernel. The same kinds of security and safety guarantees that stem from the overall system architecture can exist in banking systems - that’s my point. So yes, until everyone switches to Haskell we will always have bugs - but the kinds of bugs can be limited - as can the scale of their impact - with good system design.
oasisbob 6y ago
The problem, having worked in banking, is that the core system is separate from the loans system, almost certainly. More complexity. Probably some semi-automated batch handling which is far more brittle and janky than anyone suspects. But regardless, at some point, this system still thinks the transaction has been authorized. I'm sure there's a huge audit log showing who/where/why the mistake was made. But I've also seen end-of-period processing at a smaller financial institution where humans are involved and a few spreadsheets hairpin their ways through different systems with lots of manual intervention and QA. One step was literally very close to "audit 0.01% of payments before sending the CSV into the lockbox escrow processing system...