7 ms·
TSB Bank: Botched upgrade has left customers unable to access their accounts
- maxehmookau 9y agoIronically I tried to close my TSB account on Saturday morning because I'd just switched to Monzo Bank and was told they couldn't close my account due to the issues they were having. Still no access to my old account this morning...
- jdefelice 9y agoHave you switched to Monzo as your full-time bank? I use it for everyday spending but I not sure I want to put my wages into it yet.
- sofaofthedamned 9y agoI tried Monzo and they're awesome, but ended up ultimately moving to Starling. Reason being they have a full FSCS license, so your balance is guaranteed by the government up to around £85k. Apart from that they're both much the same, and both leagues better than a normal bank. It's amazing buying stuff or getting cash out and the notification comes through on the phone before the ATM tells you the cash is coming! Another good thing both have - you can switch off cards instantly. I thought i'd lost my Starling card when out last week, instead of cancelling it I just temporarily halted it until it was found.
- maxehmookau 9y agoMonzo now has a full FSCS licence.
- sofaofthedamned 9y agoGood to know, thank you. What I do love about Monzo is their approach to development - they blog everything, even the outages. Good to see the process involved in making a new bank from scratch. I'm keeping the account as a scratch account but have already moved the direct debits from my old Halifax account to Starling.
- maxehmookau 9y agoAgreed. Watching them discuss in detail in their blog about how they use Kubernetes has been really interesting.
- sofaofthedamned 9y agoYeah. As somebody who was learning Kubernetes whilst using it to deploy a data science platform it was very useful to see the failure cases :D
- dankohn1 9y agoMonzo's head of engineering will be keynoting at KubeCon + CloudNativeCon in Copenhagen next week, and talking about past outages: https://kccnceu18.sched.com/event/Dsan/keynote-anatomy-of-a-production-kubernetes-outage-oliver-beattie-head-of-engineering-monzo-bank https://kccnceu18.sched.com/event/Dsan/keynote-anatomy-of-a-...
- maxehmookau 9y agoThat sounds great. Gutted I won't be there, hope it's recorded!
- dankohn1 9y agoYes, we'll have it on the CNCF YouTube channel a few days later. https://www.youtube.com/c/cloudnativefdn https://www.youtube.com/c/cloudnativefdn
- maxehmookau 9y agoAmazing, thanks!
- maxehmookau 9y agoYep. My employer pays directly in to them and has for 6 months now. Zero complaints and top notch customer service.
- Rjevski 9y agoI’ve been using the Monzo Current Account since August 17’ (back when it was in closed beta) as my main account and it’s never failed me.
- katpas 9y agoAlso using it full-time, moved over my direct debits and it's been great. Ironically we went with TSB over a challenger business bank thinking they might be more reliable... Will have to find a branch this afternoon so we can pay our team as online is down
- HenryBemis 9y agoMonzo!! I was wondering when they will become a full bank. I remember they were offering prepaid cards. I am shocked that in the UK you cannot get an SMS alert when making a purchase or cannot see "fast/live" when you make an expense. I am most definitely joining them (but keeping my "main" bank) As for the TSB, it would be interesting to see what went wrong and how they fixed it.
- danpalmer 9y ago> I am most definitely joining them (but keeping my "main" bank) This seems to be a very common pattern, and I wonder if they are having issues because of it. I was a Monzo early adopter and even I haven't moved over entirely. I think a lot of users _liked_ the prepaid cards, to help them budget spending money, etc. I suspect that a lot of people haven't upgraded to a full bank account with them as a result.
- londons_explore 9y agoThey forced everyone to upgrade or leave a few weeks ago.
- benbristow 9y agoMonzo has a new 'pots' function on it. Basically you can put money in your account into separately named isolated areas in your account that can't be spent using your card normally. In layman's terms as many saving accounts as you want with more flexibility. You can still kind of use it as a prepaid card by just topping it up from your bank when you want. It's just better as you can now use Android Pay etc. and have proper banking functions like having an actual sort number/account number, direct debits, standing orders etc.
- Jonnax 9y agoAndroid pay has a similar functionality depending if the card supports it. My Amex is linked and I get a notification of all purchases as they go through regardless of whether I use my phone or card.
- sorokod 9y agoSerious question - are you comfortable with Monzo being on-line only? If it fails because of a failed upgrade, hacker attack, AWS not being available, etc... what is your plan B ?
- dx034 9y agoWhy didn't they roll back the update as soon as that was discovered? Would've caused another downtime of a few hours but better than the current chaos.
- tonyedgecombe 9y agoI'm not sure this is an option, this is all about the separation from Lloyds bank.
- isp 9y agoI have no inside information for TSB, but I have worked on other large migrations before at $DAYJOB. Based on the original planned 50-hour outage, I suspect that TSB chose a "big bang" migration: all-or-nothing, no rollback possible. These are technically easier (so cheaper) to develop, but far far riskier than taking a phased approach. I have successfully argued against such approaches in the past, due to the high risk of catastrophic failure if it goes wrong (i.e., exactly what has happened to TSB). I regard TSB's failure as an in-the-making textbook example of "how not to do it".
- mcroft 9y agoOften, these kinds of migrations are the only option due to previous cost cutting/money saving decisions - likely due to pressure from the business and nothing to do with any technical reason. I remember one migration which involved lots of internal services which were all tightly coupled, meaning updates across the whole backend, not just the part that needed it and no way to do a phased rollout. It was made worse by the fact that the update was to the platform and that depended on a DB upgrade, but that upgrade was incompatible with the old version which meant everything had to be done at once. They had no disaster recovery plan. They got it done, but I'm sure a lot of people involved went grey early thanks to that nightmare. Of course the business had no idea of the utter mess that caused all of this and continued to make harmful decisions in the name of saving money and further underfunding the IT department.
- isp 9y ago
- errozero 9y agoI logged in to my TSB account yesterday and found angular placeholders and bits of un-rendered html all over the place. The previous version was all server side rendered and worked perfectly.
- akerro 9y ago>. The previous version was all server side rendered and worked perfectly. Really? I'm with TSB since 2011 and it never worked nice for me. When I login I always get Greetings, ???NAME??
- ams6110 9y agoMove fast and break things....
- arachnophobe 9y agoThe app-before-this-one (before the server went down) was a reasonably competent bit of kit, basically a lift and shift of Lloyds, which when retired late last year became something with flashy animation on everything (animated log in place holder, 2FA selectors floating in from off screen - WHY!) but less functionality.
- errozero 9y agoThis article from 3 years ago predicted this situation: Sabadell warned about ‘high risks’ in IT integration of TSB https://www.ft.com/content/c5157c1e-20ab-11e5-aa5a-398b2169cf79 https://www.ft.com/content/c5157c1e-20ab-11e5-aa5a-398b2169c...
- trollied 9y agoThe amusing thing is, they forgot to cancel the automated press release that's boasting about a successful go-live: http://prensa.bancsabadell.com/en/News/2018/04/banco-sabadell-successfully-completes-tsb-technology-migration http://prensa.bancsabadell.com/en/News/2018/04/banco-sabadel...
- isaack 9y agoTaken down now Backup 1: http://archive.is/WnwiB http://archive.is/WnwiB Backup 2: http://webcache.googleusercontent.com/search?q=cache%3Ahttp%3A%2F%2Fprensa.bancsabadell.com%2Fen%2FNews%2F2018%2F04%2Fbanco-sabadell-successfully-completes-tsb-technology-migration&rlz=1CALEAJ_enUS730US730&oq=cache%3Ahttp%3A%2F%2Fprensa.bancsabadell.com%2Fen%2FNews%2F2018%2F04%2Fbanco-sabadell-successfully-completes-tsb-technology-migration&aqs=chrome..69i57j69i58.965j0j4&sourceid=chrome&ie=UTF-8 http://webcache.googleusercontent.com/search?q=cache%3Ahttp%...
- sorokod 9y agoThe teams that have participated in migration have proven to have the skills and knowledge required, which they have significantly developed and honed during the course of this project. Banco Sabadell’s proposal to be an agent of change has become evident, with an advanced technological proposition, not only in the domestic market but also on an international scale. With this project Banco Sabadell has created a competitive and value-added platform for the TSB franchise. Funny and poignant
- isp 9y agoCross-post from https://news.ycombinator.com/item?id=16911321 https://news.ycombinator.com/item?id=16911321 : This appears to be an IT failure comparable to the 2012 and 2015 IT failures at the Royal Bank of Scotland. These involved multi-day / multi-week full/partial outages, and multi-million pound fines. FT: https://www.ft.com/content/9178ef60-46d9-11e8-8ee8-cae73aab7... https://www.ft.com/content/9178ef60-46d9-11e8-8ee8-cae73aab7.... Background: TSB is a UK bank. They had a planned 50-hour maintenance window over the weekend (Fri 16:00 - Sun 18:00), to move away from Lloyds's systems (following the "un-merger" in 2013-2014). This went wrong, in what is likely to become a textbook example of an IT failure. From a letter from Nicky Morgan, chair of the UK Parliament's Treasury committee, to TSB CEO Paul Pester: > The reports of unauthorised transactions, access to other customers’ accounts, and failures of in-branch services have all the hallmarks of an IT meltdown. This is yet another addition to the litany of failures of banking IT systems. Potentially millions of customers could be affected by uncertainty and disruption. Some interesting screenshots of people seeing debugging information in the front-end: https://twitter.com/bcleeve/status/988333991676796928 https://twitter.com/bcleeve/status/988333991676796928 https://twitter.com/apphancer/status/988698182170357760 https://twitter.com/apphancer/status/988698182170357760 https://twitter.com/thejackthomson_/status/988564354512687104 https://twitter.com/thejackthomson_/status/98856435451268710... Ironically, TSB's parent company issued a press release (in error?) claiming that the migration has been a "success"! http://press.bancsabadell.com/en/News/2018/04/banco-sabadell-successfully-completes-tsb-technology-migration http://press.bancsabadell.com/en/News/2018/04/banco-sabadell...
- cup-of-tea 9y agoThe IT systems of these companies are held together loosely with gaffa tape. This kind of thing could happen at any time. This is what happens when you deliver your project due to pressure from non technical people even though you know it's not properly finished. Why do people not listen to the professionals when it comes to IT? In my current job I have repeatedly told non technical superiors that reverse engineering an outdated API is not a proper solution and could fall apart at any time. They do not care. It doesn't even register. Do civil engineers have this problem if they realise there might be a structural weakness?
- c12 9y agoAs far as I know civil engineers can go to prison if the construction they took part in collapses - no such provision is afforded to software engineers.
- FLUX-YOU 9y agoBesides simply quitting, what can a civil engineer do to blow the whistle on a company that is cutting corners for a bridge?
- danpalmer 9y agoThey can "blow the whistle". This could involve going to the press, to a standards body in the country, to the government, to their union or industry representative body. In the UK there are laws protecting whistleblowers who take actions like this.
- scottfr 9y agoIn the US civil engineers must go through a multi-year certification process that involves a significant amount of on the job work in addition to standardized examination before they are labelled a Professional Engineer (PE). As a PE, they can face significant legal repercussions for errors in work they approve. The benefit to the PE is that many Civil Engineering projects are legally required to have a PE design/approve them. The PE literally has a physical stamp they use to stamp blueprints and designs when they approve it. Work won't proceed without that stamp. So, if a PE refused to approve a bridge design, presumably it would be very difficult to find another PE who would approve it (especially given the record of the first PE rejecting the plans) and the work would not proceed.
- swarnie_ 9y agoThe banks owners LLoyds PLC have an earnings call 26/04/18. Will be interesting to see if this incident is brought up. EDIT: Sorry correction, looks like they sold the stake in 2015.
- jamiethompson 9y agoIt's got so bad that when my partner went into our local branch they were handing out cash and writing it down in a little book because none of their in branch systems were working either.
- dazc 9y agoI'm actually old enough to remember when that was normal. You could also phone your branch and the security was basically someone who recognised your voice.