9 ms·

TSB Train Wreck: Massive Bank IT Failure Going into Fifth Day

97 points by pm24601 8y ago

Spooky23 8y ago
This is one reason that I refuse to give up on paper statements mailed to my home.
- kozhevnikov 8y ago
  I download all PDFs and back them up as any other important data. I fail to see how paper statements solve anything besides collecting dust.
  - sillysaurus3 8y ago
    The solution is for the bank to just email you the actual PDF as an attachment. EDIT: Nope, I'm wrong. That would involve sending sensitive data over unencrypted channels, which is a no-go. Personally I wouldn't care and I'd want that, but that's irrelevant.
    - therealidiot 8y ago
      Perhaps PDFs that have been encrypted with some pre-agreed password? I receive payslips as encrypted PDFs by email
    - tgragnato 8y ago
      > That would involve sending sensitive data over unencrypted channels, which is a no-go. Why do you think that? My inbox is plenty of (unsigned) PDFs coming from mx<n°>.<mybank>.eu ... Headers show they're relayed from <redacted>.internal.<mybank>.eu (private IP included), that spf authentication is ok, but dkim is not even provided.
      - Silhouette 8y ago
        You're talking about authentication and integrity: can you confirm that the statement really came from your bank and has not been modified? The GP was talking about encryption: can anyone else involved with any system your email passed through read your intimate financial details?
        tgragnato 8y ago
        S/MIME no, PGP no, TLS yes. TLS means in transit interception is not easy and SPF that the address is not spoofed, but without DKIM I'm not able to verify that the content is not forged. As such, for example, my email provider is able to read those emails, and probably to modify them without me noticing. If something doesn't grant confidentiality, data integrity and authentication, it's not encryption. I don't think TLS alone is enough to qualify an email as an "encrypted channel".
        Silhouette 8y ago
        In general, you can't control how an email will travel from the original sender's system to your own, or from your own to its final recipient. Such is the nature of SMTP. A lot of modern email systems will be encrypting their connections on various hops, but it's not something you can guarantee for mail that leaves your own organisation, and even if all of the hops were encrypting the data in motion, there are still plenty of vulnerabilities while the data is at rest anywhere along the way. In short, banks sending out PDF statements by email would be dangerous, as sillysaurus3 originally implied.
        tgragnato 8y ago
        It's dangerous, but happened anyway in my case. And without the mitigations that are already available. I'm not trying to argue that it's ok, I'm saying that in my experience bank employees use email for confidential communications.
        8y ago
  - codeulike 8y ago
    We know that PDFs are editable, but everyone seems to act like they're not, at some point thats going to turn into a mini-crisis.
    - kozhevnikov 8y ago
      Everything is editable in a hex editor, but PDFs can be certificate signed allowing for document authenticity and integrity verification.
      - pbhjpbhj 8y ago
        And of course the banks sign their PDFs ...
        kozhevnikov 8y ago
        That's not PDF's fault, just like you can't blame Git for developers not signing their commits.
    - 21 8y ago
      Paper is also pretty editable or creatable. That's why paper money has so many anti-edit features today.
      - codeulike 8y ago
        It would take a lot more effort to edit a paper bank statement compared to a PDF one
        aninhumer 8y ago
        Not really? The paper statement is usually just a printout of the PDF.
        IncRnd 8y ago
        And, the purpose of a hard copy is to show the PDF contents from a specific time, whereas the online PDF may have been altered several times.
  - tomp 8y ago
    I tried doing that, but then I realized that I can't even select-copy-paste the data from PDF, after which I decided to continue with paper statements... there's no real advantage to PDF if I can't even export the data!
  - Spooky23 8y ago
    That works too. It’s lower effort for me to take an envelope and put it in a “2018 statements” folder than to go download statements. It’s also easier to throw out that folder in 2020 than to sort out digital documents.
- davidgrenier 8y ago
  You could be downloading any kind of snapshot or the online statement and back it up somewhere.
krona 8y ago
As is usually the case, the owner outsourced to IBM last year. What happened next was all but inevitable.
- 88 8y ago
  The CEO brought IBM in this week to clean up the mess. It doesn’t appear they were involved in the original migration.
  - krona 8y ago
    The big support contacts started rolling in years ago. I know, because I was there.
- s0l1dsnak3123 8y ago
  TSB's parent company had appointed an offshore 3rd party to implement the work. The CEO of TSB had hired IBM because he did not trust the offshore company's ability to deliver anymore. I agree it's likely going to cost even more in technical debt and/or setbacks if they bring in fresh eyes at this point!
21 8y ago
This is one of the reasons I have 4 bank accounts in 2 different countries, and also cash at home. I can't understand people with one account, one card, and 10 pounds in their pockets. See also recent Swedish fears that their cashless society is very vulnerable to a digital attack.
- adav 8y ago
  Lots of working people don't feel they have the luxury to take time away from their busy lives in order to plan their finances in any great detail. I feel bad for the those now on TSB who were originally Lloyds customers even though, like you, I have several accounts and cards spread across different banks.
- Silhouette 8y ago
  I haven't gone quite as far as that, but I agree with the principle. As much as reasonably possible, I keep facilities like personal banking, personal credit cards (always at least two), any mortgage or other secured loan, and any business financial services with different organisations that aren't part of the same group. There's often small print in bank terms and conditions about being allowed to grab money from anything of yours they have access to if there is a problem with anything else, and frankly I don't trust them not to abuse that, particularly if they've already made a mistake that caused a serious problem in the first place. Also, the FSCS compensation limits tend to be per person per firm, so dividing assets can sometimes increase the amount covered if anything really bad happens to the financial firms. And yes, I do also have some emergency cash hidden in safe places. I don't get crazy paranoid about it, but it's a substantial amount, certainly enough to fill my tank and buy essential provisions for a while. I would be concerned about becoming a truly cashless society as well, for this among other reasons.
sbarre 8y ago
I feel bad for the rank-and-file developers (the ones without decision-making power) who are stuck trying to fix all this in what are probably insanely bad working conditions right now. Yeah yeah "get a different job" or "you made the mess, you fix it" but massive failures like this aren't orchestrated by the front-line developers, they happen because people many steps removed from the work make bad executive decisions. A deadline was probably set long ago, and no one adjusted it when unexpected problems and complexities arose.
- joelhaasnoot 8y ago
  Sure, this definitely is true, but when your website or app is throwing an IndexOutOfBoundsException left and a InvalidBeanException right, your code quality also has issues. Security auditors (good or bad as they may be) would likely have your head for any leaked exception, for all they know the application may also be leaking other info.
  - edf13 8y ago
    My guess is the code just wasn't ready for production... someone higher up was screaming they didn't care and a deadline is a deadline so push it out.
    - EnderMB 8y ago
      Alongside this, it's all down to risk. I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great. For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on. A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick. The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.
onion2k 8y ago
This article is 3 days old. Customers are still having issues, so it's 8 days so far.
planetjones 8y ago
British Banks have been playing "fast and loose" with their IT systems for a long time now. TSB won't be the last time we see such a catastrophe. It is not the first in recent memory either. "Fast and Loose" may mean rushing changes through with unrealistic timelines, outsourcing the software development or/and infrastructure management to the cheapest location possible, sacking onshore experts to save money, etc. The UK Government is fully complicit, as their support for RBS (owned by the Government themselves) during their ruthless cost-cutting showed. Banks are of systemic importance. People suffer if they can't make payments on time or can't access money that's needed for something essential. But the systemic importance is not reflected in the legislation: we have a situation where Banks outsource their IT to Indian service companies and do their upmost to move the "heavy lifting" of software development to the cheapest location they can find, while the social networks and search engines are recruiting the top talent.
- jackweirdy 8y ago
  Indeed. Banks are technology companies. British high street banks either don't understand or refuse to believe that they are technology companies.
  - gaius 8y ago
    Like the Civil Service, the senior ranks are selected by where their parents sent them to school, and how well they learned to conjugate Latin verbs while they were there. Technical skill is not only viewed with contempt it is actually a bar to promotion. Investment banking is actually a lot saner - Thatcher’s reforms in the 80s went a long way towards destroying the Old Boys Network in that industry niche.
- collyw 8y ago
  I worked for one of the biggest names in the business in ~ 2007 and was fairly shocked at how poor quality the IT systems were (and i wasn't particularly experienced then). Probably the worst job I have had. Crap work (might as well have been working in a factory for the creativity it and genuine problem solving that it involved) aggressive style management seemed to be what got promoted. And high turnover - I don't think anyone in my team lasted much over a year. The thing is they had the money to throw at the problem. Just keep on hiring young devs at a bit above market rates. Th name would look good on your CV.
xedarius 8y ago
I am a TSB customers and the service has been insane. Here's some of the highlights - Login screen regularly throws a message from the backend saying 'java.lang.NullPointerException' (so now I know what the backend written in) - My balance was totalled incorrectly, which made me appear to be very wealthy. - Payments had gone missing from my statement (how do I know they were missing, one of them was my pay) - When I did login I tried to transfer some funds and was met with 'Payment successful' followed by a java.lang.IndexOutOfBoundsException. Had the payment worked? Who knows - They have a fundamental scaling issue. I've never seen anything like it. I do not get the impression this will be fixed quickly.
- 3pt14159 8y ago
  Why don’t you just bank elsewhere?
  - gaius 8y ago
    Because it’s a chicken and egg situation
  - sbarre 8y ago
    If I was a TSB customer, the second this was all resolved I would move 100% of my business away from them and never look back. I'm sure there are plenty of people waiting to do just that.
  - swarnie_ 8y ago
    British banks are developing a bit a of a habit for massive IT failures. Moving is fine but someone else will fail next month.
    - twic 8y ago
      Although, as the article mentions, TSB are owned by a Spanish bank, Sabadell, and it was the move to Sabadell's platform, developed in Spain, that caused the trouble!
  - xedarius 8y ago
cesarb 8y ago
I've been wondering: if this issue isn't solved soon, how long before we see a run on that bank?
- 21 8y ago
  Jokes on you, you can't do a run on a bank where you can't login or take any money out. More seriously, I would definitely move my money somewhere else if I had an account there. Not because I would worry that it will disappear (government insurance & all that), but just because this looks like the kind of problem which will take a long time to fix, and which could strike back.
  - sbarre 8y ago
    I can only imagine the bad actors that must be hammering TSB's websites and interfaces as we speak, trying to find and exploit holes...
    - spydum 8y ago
      Exactly.. you have clearly under-tested code that is in debug mode printing out all sorts of detailed error messages and systems team that is overloaded with errors and problems to triage. Talk about a ripe target. That is stuff of nightmares
      - 21 8y ago
        Even worse, there are reports of people seeing accounts which don't belong to them.
  - PeterisP 8y ago
    You definitely can do a run on a bank where you can't log in - you come to the office (or mail a registered letter) with written instructions to transfer all your money elsewhere and close the accounts, and legally they have a quite limited time to comply, I believe one or two business days at most, until various EU consumer protection laws for financial products kick in. Furthermore, there's a lot of process for switching banks that can be done by the customer through the bank they're switching to. While your savings may be stuck in TSC for some days, you don't need to go to TSC at all if you'd choose to switch your wages and recurring bill payments to somewhere else, you just go to the new bank.
kennydude 8y ago
This kind of thing gives me confidence in "challenger banks" like Monzo who are building their systems in-house and I hope with a lot more testing than this. As far as I know Monzo is using microservices (a bank is probably big enough for that to be the right choice) to try and limit the damage of anything insane like this. The way they handled Prepaid to Current Accounts was absolutely seamless https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payments-work/ https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payment...
- tachion 8y ago
  This has nothing to do with microservices and everything to do with lack of skill, proper engineering and chaos in IT departments of almost all banks. You can write and deploy (don't forget about the fact developers aren't skilled in ops at all and that's a common cause for these things) good software in microservices and monolith architectures, you just need to know what are you doing. Monzo/Mondo had issues on their own, and in fact they have very frequent issues with their payment processors.
  - philcrump 8y ago
    They had very frequent issues with their payment processors. The current account (now the only account as the prepaid beta programme has ended) runs on their internally developed payment processor, and has so far had minimal issues. More information here: https://monzo.com/blog/2018/04/04/ending-prepaid/ https://monzo.com/blog/2018/04/04/ending-prepaid/
- gaius 8y ago
  You know “microservices” was invented by IBM in the 1960s on mainframes, right?
- djhworld 8y ago
  Starling bank do this too, and have more features than Monzo
- leejo 8y ago
  > This kind of thing gives me confidence in "challenger banks" like Monzo who are building their systems in-house and I hope with a lot more testing than this. I know someone who interviewed at Monzo and said the interview process and technical test (write a web crawler in Go) was generally a pretty good process. However they "noped!" out of there when the interviewer said that they don't do unit/automated testing, and it's the developer's responsibility to make sure their code works. This was a while ago, so hopefully things have changed or hopefully the interviewer was not telling the truth (perhaps looking for a reaction?). However if it is the case then they're about as "modern" a bank as all the rest of these legacy banks, or at least will be in a few years time when their technical debt starts to catch up with them.
weavie 8y ago
"Do not request a bean from a BeanFactory in a destroy method implementation!" User friendly error message of the year! [https://twitter.com/thejackthomson_/status/988564354512687104/photo/1 https://twitter.com/thejackthomson_/status/98856435451268710...]
- IncRnd 8y ago
  Jack! Did you really sell the cow for beans??
dsabanin 8y ago
Is this a Mr. Robot episode?
- codeulike 8y ago
  I think it might be the ending of Fight Club
richardhod 8y ago
Ironically, before they merged in the 90s, Lloyds had a terrible computer system, and TSB was market-leading in major bank IT customer systems, at least from a customer POV. Then they didn't learn and continue with TSB's IT systems and management, but became more lloyds instead. Sadly, this new TSB is likely only in name anything like the old one.
kown223 8y ago
I'm surprised no one mentioned the solution, buy bitcoin..
- akerro 8y ago
  HEY LET ME JUST LOGIN TO MY TSB TO MAKE THE TRAnsfer...
jasonsync 8y ago
bank IT is a systemic risk waiting to happen The same thing happened with Tangerine.ca when they launched their new website late last year. Tangerine is owned by one of Canada's "big 5" banks, Scotiabank. For at least a week after the launch, users were experiencing login errors, timeouts, incorrect balances and features not working. Backend code exceptions would bubble up to the surface. It was ugly. But even with the backend issues ironed out weeks later, users continue to lament about the horrible new UI, which over 6 months later, is still a UX nightmare on desktop web browsers. Reading the 400+ Facebook comments on their "launch day post (they since removed the post from the timeline) is quite an eye opener. Apparently they did minimal testing and disregarded all user feedback during the beta phase. The comments are still accessible from the post photo here: https://www.facebook.com/TangerineBank/photos/pb.467988996640009.-2207520000.1507258210./1225878164184418/?type=3 https://www.facebook.com/TangerineBank/photos/pb.46798899664...
sorokod 8y ago
The mess is big enough for the post-mortem to be made public. Should be interesting.
- gadders 8y ago
  I would imagine the UK regulators would launch an enquiry and publish a report like they did when NatWest managed to break all their overnight batches: http://www.fca.org.uk/your-fca/documents/final-notices/2014/rbs-natwest-ulster http://www.fca.org.uk/your-fca/documents/final-notices/2014/...
djhworld 8y ago
The CEO says he's given a deadline of Saturday for IBM to fix the problem, that seems a pretty tight deadline. Are they using IBM technologies for their stack?
- onion2k 8y ago
  It'd be a bit unfair to say that to IBM if they're not.
merricksb 8y ago
This is from 3 days ago, same day as another thorough discussion about the topic on HN: https://news.ycombinator.com/item?id=16910947 https://news.ycombinator.com/item?id=16910947
- tim333 8y ago
  This kind of #3 in the HN TSB screw up series - 'TSB gave me access to someone's £35,000' (bbc.co.uk) - TSB Bank: Botched upgrade has left customers unable to access their accounts - this - Warning signs for TSB's IT meltdown were clear a year ago, according to insider
jamiethompson 8y ago
I'm a Lloyds customer, and I'm not suggesting that this is in any way related but my current account today showed a planned 0% overdraft limit of £2 shy of £10bn.

97 points by pm24601 8y ago

Spooky23 8y ago

This is one reason that I refuse to give up on paper statements mailed to my home.

kozhevnikov 8y ago
I download all PDFs and back them up as any other important data. I fail to see how paper statements solve anything besides collecting dust.
- sillysaurus3 8y ago
  The solution is for the bank to just email you the actual PDF as an attachment. EDIT: Nope, I'm wrong. That would involve sending sensitive data over unencrypted channels, which is a no-go. Personally I wouldn't care and I'd want that, but that's irrelevant.
  - therealidiot 8y ago
    Perhaps PDFs that have been encrypted with some pre-agreed password? I receive payslips as encrypted PDFs by email
  - tgragnato 8y ago
    > That would involve sending sensitive data over unencrypted channels, which is a no-go. Why do you think that? My inbox is plenty of (unsigned) PDFs coming from mx<n°>.<mybank>.eu ... Headers show they're relayed from <redacted>.internal.<mybank>.eu (private IP included), that spf authentication is ok, but dkim is not even provided.
    - Silhouette 8y ago
      You're talking about authentication and integrity: can you confirm that the statement really came from your bank and has not been modified? The GP was talking about encryption: can anyone else involved with any system your email passed through read your intimate financial details?
      - tgragnato 8y ago
        S/MIME no, PGP no, TLS yes. TLS means in transit interception is not easy and SPF that the address is not spoofed, but without DKIM I'm not able to verify that the content is not forged. As such, for example, my email provider is able to read those emails, and probably to modify them without me noticing. If something doesn't grant confidentiality, data integrity and authentication, it's not encryption. I don't think TLS alone is enough to qualify an email as an "encrypted channel".
        Silhouette 8y ago
        In general, you can't control how an email will travel from the original sender's system to your own, or from your own to its final recipient. Such is the nature of SMTP. A lot of modern email systems will be encrypting their connections on various hops, but it's not something you can guarantee for mail that leaves your own organisation, and even if all of the hops were encrypting the data in motion, there are still plenty of vulnerabilities while the data is at rest anywhere along the way. In short, banks sending out PDF statements by email would be dangerous, as sillysaurus3 originally implied.
        tgragnato 8y ago
        It's dangerous, but happened anyway in my case. And without the mitigations that are already available. I'm not trying to argue that it's ok, I'm saying that in my experience bank employees use email for confidential communications.
        8y ago
- codeulike 8y ago
  We know that PDFs are editable, but everyone seems to act like they're not, at some point thats going to turn into a mini-crisis.
  - kozhevnikov 8y ago
    Everything is editable in a hex editor, but PDFs can be certificate signed allowing for document authenticity and integrity verification.
    - pbhjpbhj 8y ago
      And of course the banks sign their PDFs ...
      - kozhevnikov 8y ago
        That's not PDF's fault, just like you can't blame Git for developers not signing their commits.
  - 21 8y ago
    Paper is also pretty editable or creatable. That's why paper money has so many anti-edit features today.
    - codeulike 8y ago
      It would take a lot more effort to edit a paper bank statement compared to a PDF one
      - aninhumer 8y ago
        Not really? The paper statement is usually just a printout of the PDF.
        IncRnd 8y ago
        And, the purpose of a hard copy is to show the PDF contents from a specific time, whereas the online PDF may have been altered several times.
- tomp 8y ago
  I tried doing that, but then I realized that I can't even select-copy-paste the data from PDF, after which I decided to continue with paper statements... there's no real advantage to PDF if I can't even export the data!
- Spooky23 8y ago
  That works too. It’s lower effort for me to take an envelope and put it in a “2018 statements” folder than to go download statements. It’s also easier to throw out that folder in 2020 than to sort out digital documents.
davidgrenier 8y ago
You could be downloading any kind of snapshot or the online statement and back it up somewhere.

krona 8y ago

As is usually the case, the owner outsourced to IBM last year. What happened next was all but inevitable.

88 8y ago
The CEO brought IBM in this week to clean up the mess. It doesn’t appear they were involved in the original migration.
- krona 8y ago
  The big support contacts started rolling in years ago. I know, because I was there.
s0l1dsnak3123 8y ago
TSB's parent company had appointed an offshore 3rd party to implement the work. The CEO of TSB had hired IBM because he did not trust the offshore company's ability to deliver anymore. I agree it's likely going to cost even more in technical debt and/or setbacks if they bring in fresh eyes at this point!

21 8y ago

This is one of the reasons I have 4 bank accounts in 2 different countries, and also cash at home. I can't understand people with one account, one card, and 10 pounds in their pockets. See also recent Swedish fears that their cashless society is very vulnerable to a digital attack.

adav 8y ago
Lots of working people don't feel they have the luxury to take time away from their busy lives in order to plan their finances in any great detail. I feel bad for the those now on TSB who were originally Lloyds customers even though, like you, I have several accounts and cards spread across different banks.
Silhouette 8y ago
I haven't gone quite as far as that, but I agree with the principle. As much as reasonably possible, I keep facilities like personal banking, personal credit cards (always at least two), any mortgage or other secured loan, and any business financial services with different organisations that aren't part of the same group. There's often small print in bank terms and conditions about being allowed to grab money from anything of yours they have access to if there is a problem with anything else, and frankly I don't trust them not to abuse that, particularly if they've already made a mistake that caused a serious problem in the first place. Also, the FSCS compensation limits tend to be per person per firm, so dividing assets can sometimes increase the amount covered if anything really bad happens to the financial firms. And yes, I do also have some emergency cash hidden in safe places. I don't get crazy paranoid about it, but it's a substantial amount, certainly enough to fill my tank and buy essential provisions for a while. I would be concerned about becoming a truly cashless society as well, for this among other reasons.

sbarre 8y ago

I feel bad for the rank-and-file developers (the ones without decision-making power) who are stuck trying to fix all this in what are probably insanely bad working conditions right now. Yeah yeah "get a different job" or "you made the mess, you fix it" but massive failures like this aren't orchestrated by the front-line developers, they happen because people many steps removed from the work make bad executive decisions. A deadline was probably set long ago, and no one adjusted it when unexpected problems and complexities arose.

joelhaasnoot 8y ago
Sure, this definitely is true, but when your website or app is throwing an IndexOutOfBoundsException left and a InvalidBeanException right, your code quality also has issues. Security auditors (good or bad as they may be) would likely have your head for any leaked exception, for all they know the application may also be leaking other info.
- edf13 8y ago
  My guess is the code just wasn't ready for production... someone higher up was screaming they didn't care and a deadline is a deadline so push it out.
  - EnderMB 8y ago
    Alongside this, it's all down to risk. I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great. For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on. A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick. The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.

onion2k 8y ago

This article is 3 days old. Customers are still having issues, so it's 8 days so far.

planetjones 8y ago

British Banks have been playing "fast and loose" with their IT systems for a long time now. TSB won't be the last time we see such a catastrophe. It is not the first in recent memory either. "Fast and Loose" may mean rushing changes through with unrealistic timelines, outsourcing the software development or/and infrastructure management to the cheapest location possible, sacking onshore experts to save money, etc. The UK Government is fully complicit, as their support for RBS (owned by the Government themselves) during their ruthless cost-cutting showed. Banks are of systemic importance. People suffer if they can't make payments on time or can't access money that's needed for something essential. But the systemic importance is not reflected in the legislation: we have a situation where Banks outsource their IT to Indian service companies and do their upmost to move the "heavy lifting" of software development to the cheapest location they can find, while the social networks and search engines are recruiting the top talent.

jackweirdy 8y ago
Indeed. Banks are technology companies. British high street banks either don't understand or refuse to believe that they are technology companies.
- gaius 8y ago
  Like the Civil Service, the senior ranks are selected by where their parents sent them to school, and how well they learned to conjugate Latin verbs while they were there. Technical skill is not only viewed with contempt it is actually a bar to promotion. Investment banking is actually a lot saner - Thatcher’s reforms in the 80s went a long way towards destroying the Old Boys Network in that industry niche.
collyw 8y ago
I worked for one of the biggest names in the business in ~ 2007 and was fairly shocked at how poor quality the IT systems were (and i wasn't particularly experienced then). Probably the worst job I have had. Crap work (might as well have been working in a factory for the creativity it and genuine problem solving that it involved) aggressive style management seemed to be what got promoted. And high turnover - I don't think anyone in my team lasted much over a year. The thing is they had the money to throw at the problem. Just keep on hiring young devs at a bit above market rates. Th name would look good on your CV.

xedarius 8y ago

I am a TSB customers and the service has been insane. Here's some of the highlights - Login screen regularly throws a message from the backend saying 'java.lang.NullPointerException' (so now I know what the backend written in) - My balance was totalled incorrectly, which made me appear to be very wealthy. - Payments had gone missing from my statement (how do I know they were missing, one of them was my pay) - When I did login I tried to transfer some funds and was met with 'Payment successful' followed by a java.lang.IndexOutOfBoundsException. Had the payment worked? Who knows - They have a fundamental scaling issue. I've never seen anything like it. I do not get the impression this will be fixed quickly.

3pt14159 8y ago
Why don’t you just bank elsewhere?
- gaius 8y ago
  Because it’s a chicken and egg situation
- sbarre 8y ago
  If I was a TSB customer, the second this was all resolved I would move 100% of my business away from them and never look back. I'm sure there are plenty of people waiting to do just that.
- swarnie_ 8y ago
  British banks are developing a bit a of a habit for massive IT failures. Moving is fine but someone else will fail next month.
  - twic 8y ago
    Although, as the article mentions, TSB are owned by a Spanish bank, Sabadell, and it was the move to Sabadell's platform, developed in Spain, that caused the trouble!
- xedarius 8y ago

cesarb 8y ago

I've been wondering: if this issue isn't solved soon, how long before we see a run on that bank?

21 8y ago
Jokes on you, you can't do a run on a bank where you can't login or take any money out. More seriously, I would definitely move my money somewhere else if I had an account there. Not because I would worry that it will disappear (government insurance & all that), but just because this looks like the kind of problem which will take a long time to fix, and which could strike back.
- sbarre 8y ago
  I can only imagine the bad actors that must be hammering TSB's websites and interfaces as we speak, trying to find and exploit holes...
  - spydum 8y ago
    Exactly.. you have clearly under-tested code that is in debug mode printing out all sorts of detailed error messages and systems team that is overloaded with errors and problems to triage. Talk about a ripe target. That is stuff of nightmares
    - 21 8y ago
      Even worse, there are reports of people seeing accounts which don't belong to them.
- PeterisP 8y ago
  You definitely can do a run on a bank where you can't log in - you come to the office (or mail a registered letter) with written instructions to transfer all your money elsewhere and close the accounts, and legally they have a quite limited time to comply, I believe one or two business days at most, until various EU consumer protection laws for financial products kick in. Furthermore, there's a lot of process for switching banks that can be done by the customer through the bank they're switching to. While your savings may be stuck in TSC for some days, you don't need to go to TSC at all if you'd choose to switch your wages and recurring bill payments to somewhere else, you just go to the new bank.

kennydude 8y ago

This kind of thing gives me confidence in "challenger banks" like Monzo who are building their systems in-house and I hope with a lot more testing than this. As far as I know Monzo is using microservices (a bank is probably big enough for that to be the right choice) to try and limit the damage of anything insane like this. The way they handled Prepaid to Current Accounts was absolutely seamless https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payments-work/ https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payment...

tachion 8y ago
This has nothing to do with microservices and everything to do with lack of skill, proper engineering and chaos in IT departments of almost all banks. You can write and deploy (don't forget about the fact developers aren't skilled in ops at all and that's a common cause for these things) good software in microservices and monolith architectures, you just need to know what are you doing. Monzo/Mondo had issues on their own, and in fact they have very frequent issues with their payment processors.
- philcrump 8y ago
  They had very frequent issues with their payment processors. The current account (now the only account as the prepaid beta programme has ended) runs on their internally developed payment processor, and has so far had minimal issues. More information here: https://monzo.com/blog/2018/04/04/ending-prepaid/ https://monzo.com/blog/2018/04/04/ending-prepaid/
gaius 8y ago
You know “microservices” was invented by IBM in the 1960s on mainframes, right?
djhworld 8y ago
Starling bank do this too, and have more features than Monzo
leejo 8y ago
> This kind of thing gives me confidence in "challenger banks" like Monzo who are building their systems in-house and I hope with a lot more testing than this. I know someone who interviewed at Monzo and said the interview process and technical test (write a web crawler in Go) was generally a pretty good process. However they "noped!" out of there when the interviewer said that they don't do unit/automated testing, and it's the developer's responsibility to make sure their code works. This was a while ago, so hopefully things have changed or hopefully the interviewer was not telling the truth (perhaps looking for a reaction?). However if it is the case then they're about as "modern" a bank as all the rest of these legacy banks, or at least will be in a few years time when their technical debt starts to catch up with them.

weavie 8y ago

"Do not request a bean from a BeanFactory in a destroy method implementation!" User friendly error message of the year! [https://twitter.com/thejackthomson_/status/988564354512687104/photo/1 https://twitter.com/thejackthomson_/status/98856435451268710...]

IncRnd 8y ago
Jack! Did you really sell the cow for beans??

dsabanin 8y ago

Is this a Mr. Robot episode?

codeulike 8y ago
I think it might be the ending of Fight Club

richardhod 8y ago

Ironically, before they merged in the 90s, Lloyds had a terrible computer system, and TSB was market-leading in major bank IT customer systems, at least from a customer POV. Then they didn't learn and continue with TSB's IT systems and management, but became more lloyds instead. Sadly, this new TSB is likely only in name anything like the old one.

kown223 8y ago

I'm surprised no one mentioned the solution, buy bitcoin..

akerro 8y ago
HEY LET ME JUST LOGIN TO MY TSB TO MAKE THE TRAnsfer...

jasonsync 8y ago

bank IT is a systemic risk waiting to happen The same thing happened with Tangerine.ca when they launched their new website late last year. Tangerine is owned by one of Canada's "big 5" banks, Scotiabank. For at least a week after the launch, users were experiencing login errors, timeouts, incorrect balances and features not working. Backend code exceptions would bubble up to the surface. It was ugly. But even with the backend issues ironed out weeks later, users continue to lament about the horrible new UI, which over 6 months later, is still a UX nightmare on desktop web browsers. Reading the 400+ Facebook comments on their "launch day post (they since removed the post from the timeline) is quite an eye opener. Apparently they did minimal testing and disregarded all user feedback during the beta phase. The comments are still accessible from the post photo here: https://www.facebook.com/TangerineBank/photos/pb.467988996640009.-2207520000.1507258210./1225878164184418/?type=3 https://www.facebook.com/TangerineBank/photos/pb.46798899664...

sorokod 8y ago

The mess is big enough for the post-mortem to be made public. Should be interesting.

gadders 8y ago
I would imagine the UK regulators would launch an enquiry and publish a report like they did when NatWest managed to break all their overnight batches: http://www.fca.org.uk/your-fca/documents/final-notices/2014/rbs-natwest-ulster http://www.fca.org.uk/your-fca/documents/final-notices/2014/...

djhworld 8y ago

The CEO says he's given a deadline of Saturday for IBM to fix the problem, that seems a pretty tight deadline. Are they using IBM technologies for their stack?

onion2k 8y ago
It'd be a bit unfair to say that to IBM if they're not.

merricksb 8y ago

This is from 3 days ago, same day as another thorough discussion about the topic on HN: https://news.ycombinator.com/item?id=16910947 https://news.ycombinator.com/item?id=16910947

tim333 8y ago
This kind of #3 in the HN TSB screw up series - 'TSB gave me access to someone's £35,000' (bbc.co.uk) - TSB Bank: Botched upgrade has left customers unable to access their accounts - this - Warning signs for TSB's IT meltdown were clear a year ago, according to insider

jamiethompson 8y ago

I'm a Lloyds customer, and I'm not suggesting that this is in any way related but my current account today showed a planned 0% overdraft limit of £2 shy of £10bn.