5 ms·

RBS software issues likely to continue into third week

17 points by lucaspiller 14y ago

EwanToo 14y ago
I can see this going down in IT folklore, "Remember to do test your backups and roll back plans, or you might end up like Ulster Bank"
- sausagefeet 14y ago
  I think it would work better with some kind fo rhyming scheme.
- omh 14y ago
  Or perhaps: "Don't outsource all of your IT ops team or you might end up like RBS/Ulster Bank" If you're an IT guy then that one is perhaps a more important battle to win.
  - d4nt 14y ago
    In theory a separate company who's core competency is running IT systems should be less likely to mess this up, not more so. However, retail banks are rapidly becoming sort of API hosting companies for the processing of financial transactions. Their core competency IS running IT systems. If none of RBS's physical branches opened tomorrow, the UK would probably tick along just fine, with some inconvenience caused to people who deal mostly in cash. But fail to process payments for 24 hours, and the whole UK has problems. If this had gone any longer than it did, it's not hard to envisage a run on the bank and a domino effect to other banks. It's really quite scary how badly we depend on these IT systems, waaay more than Google or AWS.
    - omh 14y ago
      In theory a separate company who's core competency is running IT systems should be less likely to mess this up, not more so. That's true. Rather than "Don't outsource" the warning should be "Don't fire all the existing staff who understand your systems".
      - edwinnathaniel 14y ago
        They didn't. They fired half of the team.
- d4nt 14y ago
  I work with some people who used to work for RBS IT, they tell me that the department was as bureaucratic as they come (presumably to reduce risk) but once all the hoops had been jumped through, it came down to single team who would put code into production, and they were often quite poor at it. When I think about the scale of this mess, and the hundreds of millions that RBS is going to loose in compensation and lost customers, I hope that RBS (and in house corporate IT in general) starts to appreciate the need for automated deployment and fault tolerant IT systems rather than yet more process as the answer (I'm looking at you ITIL).
  - webwanderings 14y ago
    ITIL is not a problem, in fact it is a good thing that has happened to IT. However, the problem is the non-IT people running the ITIL. They end up sidelining the developers, or the developers badmouthing the processes, which causes all sorts of separation and issues. When people follow processes appropriately, things work out well and on time.
    - acdha 14y ago
      ITIL is a problem. It's extremely rare that an organization has problems because nobody's ever thought about how to do things better rather than having broken internal structure or incentives – ITIL can't help with either of those but the huge amounts of time and money it soaks up will make it much harder to actually do anything about them.
- rmc 14y ago
  "to do a NatWest" (Unfortuntely it slightly rhymes with 'best', but meh)
  - pasbesoin 14y ago
    I couldn't help thinking of the backronym: "Really Bad Shit".
Irishsteve 14y ago
I'm a customer with these guys. The problem started on the 21st of June, the only transactions on my account are small ATM withdrawals. No card payments or more importantly no salary! Tried to pay the rent. Didn't work. Tried to pay the phone, didn't work, broadband? Nope didn't work either. From what I understand its some old mainframe-ish style legacy infrastructure that broke down. Ulster bank downsized anyone who was working in that area a year ago and shipped it all out of the country.
- luxpir 14y ago
  The Register had some decent coverage of what went wrong [1]. If this does prove to be the root cause, it just highlights the costs of cutting costs irresponsibly. For customers, as has been said elsewhere, second bank accounts and cash buffers can be cheap insurance against the incompetence of others. [1] http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_offshoring_software_glitch/ http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_off...
  - pasbesoin 14y ago
    Boy, some of that described experience is pretty familiar.
sausagefeet 14y ago
Thanks online news source for just reposting twitter comments without adding anything. Would have been nice to actually contact a poster to get more details or like...do more work than search for a hashtag.
jacktams 14y ago
Its really not an IT problem, the upside to this for RBS is they can keep a very tight grip of their deposit to loan ratios. They aren't bothered about the customers, its about keeping the bank above the waterline. http://www.reuters.com/finance/stocks/incomeStatement?stmtType=BAL&symbol=RBS.L http://www.reuters.com/finance/stocks/incomeStatement?stmtTy... Just look at how very close the Deposit vs. Loan values are, and also consider the timeline of this whole debacle. RBS downgraded, more loans than deposits, suddenly a "IT problem" that lets them even out the odds a bit, so they can afford to borrow to cover the remainder, and gain much needed liquidity. http://hat4uk.wordpress.com/2012/06/25/rbsulster-glitch-slog-survey-of-users-plus-analysis-of-the-numbers-raise-further-doubts/ http://hat4uk.wordpress.com/2012/06/25/rbsulster-glitch-slog...
- arethuza 14y ago
  Is a Loan-to-Deposit ratio above 100% really so unusual that they would engineer an IT failure to hide it? Some banks seem to be operating which much higher values: http://www.zerohedge.com/news/what-europes-loan-deposit-ratios-look http://www.zerohedge.com/news/what-europes-loan-deposit-rati... [Having said that, nothing would surprise me about RBS]
- urbiops 14y ago
  Given that both payments in and payments out seen to have been affected, this isn't really likely, not to mention that the fallout from events like these in terms of press coverage and regulatory interest would outweigh any benefit from such an act. Lastly, organisation of such an event would require way too many people to stay secret.
- anextio 14y ago
  No. There will be an Irish Government investigation into this. They would never get away with manufacturing it.
hopeless 14y ago
I'm beginning to wonder if Ulster Bank have reached the event horizon beyond which they don't have the computing power to recover from this.
- Irishsteve 14y ago
  Interesting. I wonder what would happen in this instance. They have admitted losing some transaction information forever. Wonder what the financial impact will be

17 points by lucaspiller 14y ago

I can see this going down in IT folklore, "Remember to do test your backups and roll back plans, or you might end up like Ulster Bank"

sausagefeet 14y ago
I think it would work better with some kind fo rhyming scheme.
omh 14y ago
Or perhaps: "Don't outsource all of your IT ops team or you might end up like RBS/Ulster Bank" If you're an IT guy then that one is perhaps a more important battle to win.
- d4nt 14y ago
  In theory a separate company who's core competency is running IT systems should be less likely to mess this up, not more so. However, retail banks are rapidly becoming sort of API hosting companies for the processing of financial transactions. Their core competency IS running IT systems. If none of RBS's physical branches opened tomorrow, the UK would probably tick along just fine, with some inconvenience caused to people who deal mostly in cash. But fail to process payments for 24 hours, and the whole UK has problems. If this had gone any longer than it did, it's not hard to envisage a run on the bank and a domino effect to other banks. It's really quite scary how badly we depend on these IT systems, waaay more than Google or AWS.
  - omh 14y ago
    In theory a separate company who's core competency is running IT systems should be less likely to mess this up, not more so. That's true. Rather than "Don't outsource" the warning should be "Don't fire all the existing staff who understand your systems".
    - edwinnathaniel 14y ago
      They didn't. They fired half of the team.
d4nt 14y ago
I work with some people who used to work for RBS IT, they tell me that the department was as bureaucratic as they come (presumably to reduce risk) but once all the hoops had been jumped through, it came down to single team who would put code into production, and they were often quite poor at it. When I think about the scale of this mess, and the hundreds of millions that RBS is going to loose in compensation and lost customers, I hope that RBS (and in house corporate IT in general) starts to appreciate the need for automated deployment and fault tolerant IT systems rather than yet more process as the answer (I'm looking at you ITIL).
- webwanderings 14y ago
  ITIL is not a problem, in fact it is a good thing that has happened to IT. However, the problem is the non-IT people running the ITIL. They end up sidelining the developers, or the developers badmouthing the processes, which causes all sorts of separation and issues. When people follow processes appropriately, things work out well and on time.
  - acdha 14y ago
    ITIL is a problem. It's extremely rare that an organization has problems because nobody's ever thought about how to do things better rather than having broken internal structure or incentives – ITIL can't help with either of those but the huge amounts of time and money it soaks up will make it much harder to actually do anything about them.
rmc 14y ago
"to do a NatWest" (Unfortuntely it slightly rhymes with 'best', but meh)
- pasbesoin 14y ago
  I couldn't help thinking of the backronym: "Really Bad Shit".

Irishsteve 14y ago

I'm a customer with these guys. The problem started on the 21st of June, the only transactions on my account are small ATM withdrawals. No card payments or more importantly no salary! Tried to pay the rent. Didn't work. Tried to pay the phone, didn't work, broadband? Nope didn't work either. From what I understand its some old mainframe-ish style legacy infrastructure that broke down. Ulster bank downsized anyone who was working in that area a year ago and shipped it all out of the country.

luxpir 14y ago
The Register had some decent coverage of what went wrong [1]. If this does prove to be the root cause, it just highlights the costs of cutting costs irresponsibly. For customers, as has been said elsewhere, second bank accounts and cash buffers can be cheap insurance against the incompetence of others. [1] http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_offshoring_software_glitch/ http://www.theregister.co.uk/2012/06/28/rbs_job_cuts_and_off...
- pasbesoin 14y ago
  Boy, some of that described experience is pretty familiar.

sausagefeet 14y ago

Thanks online news source for just reposting twitter comments without adding anything. Would have been nice to actually contact a poster to get more details or like...do more work than search for a hashtag.

jacktams 14y ago

Its really not an IT problem, the upside to this for RBS is they can keep a very tight grip of their deposit to loan ratios. They aren't bothered about the customers, its about keeping the bank above the waterline. http://www.reuters.com/finance/stocks/incomeStatement?stmtType=BAL&symbol=RBS.L http://www.reuters.com/finance/stocks/incomeStatement?stmtTy... Just look at how very close the Deposit vs. Loan values are, and also consider the timeline of this whole debacle. RBS downgraded, more loans than deposits, suddenly a "IT problem" that lets them even out the odds a bit, so they can afford to borrow to cover the remainder, and gain much needed liquidity. http://hat4uk.wordpress.com/2012/06/25/rbsulster-glitch-slog-survey-of-users-plus-analysis-of-the-numbers-raise-further-doubts/ http://hat4uk.wordpress.com/2012/06/25/rbsulster-glitch-slog...

arethuza 14y ago
Is a Loan-to-Deposit ratio above 100% really so unusual that they would engineer an IT failure to hide it? Some banks seem to be operating which much higher values: http://www.zerohedge.com/news/what-europes-loan-deposit-ratios-look http://www.zerohedge.com/news/what-europes-loan-deposit-rati... [Having said that, nothing would surprise me about RBS]
urbiops 14y ago
Given that both payments in and payments out seen to have been affected, this isn't really likely, not to mention that the fallout from events like these in terms of press coverage and regulatory interest would outweigh any benefit from such an act. Lastly, organisation of such an event would require way too many people to stay secret.
anextio 14y ago
No. There will be an Irish Government investigation into this. They would never get away with manufacturing it.

hopeless 14y ago

I'm beginning to wonder if Ulster Bank have reached the event horizon beyond which they don't have the computing power to recover from this.

Irishsteve 14y ago
Interesting. I wonder what would happen in this instance. They have admitted losing some transaction information forever. Wonder what the financial impact will be