4 ms·

Some more information about the tech behind the failings here: http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_went_wrong/ http://www.theregister.co.uk

Some more information about the tech behind the failings here: http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_went_wrong/ http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_wen... TLDR: Looks like a failure in their CA-7 batch processing systems, which compounded so they weren't able to back out. Lots of the experts in RBS's implementation of CA-7 have been made redundant, and been replaced with offshore-expertise I'm not going to question the proficiency of the offshore talent, however, I'm sure most on HN would agree that there's great value with legacy systems in not getting rid of the people who have been working on them for the last 20 years, and know all the idiosyncrasies and foibles. Lots of the UK banks still use highly complex decades old main-frame based systems which interact with hundreds of other systems. It feels to me like the systems have got so complex, that its hard to see a way of replacing or rewriting and simplifying them.

timrobinson 14y ago
Specifically, it looks like a manual error during a deployment: http://www.guardian.co.uk/technology/2012/jun/25/how-natwest-it-meltdown http://www.guardian.co.uk/technology/2012/jun/25/how-natwest... "It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"
- Zenst 14y ago
  The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet. Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on about TAX evasion, this too me makes no sence and is why I don't run banks :|.
  - Isofarro 14y ago
    IT is seen as an overhead, and as something that doesn't contribute directly to profitability. When compared to dealers, dealers bring in the profit, IT slurps it away.
    - Zenst 14y ago
      Very true sadly. Don't see many outsourced HR and accounts sections though. Sad part is that when it goes wrong it does effect the balance sheet. Even sadder is how they internaly cost IT and do it wrong. Remove the IT and see how many people/time is needed to do the same job and that is the true cost/potentual impact of IT. Sadly though that is never done and only comes to light when things fail and then they blame IT and not the effects of seagul managment, budget cuts etc.
      - gadders 14y ago
        Oh man. Please, please, please outsource HR. Preferably to Mars. Biggest waste of space in the whole company.
    - sbarlster 14y ago
      Interesting that a retail bank whose core business is to process transactions takes the decision to outsource that capability. And then is exposed as unable to monitor/control the outsourced operation. Some similarities to the now defunct Railtrack whose remit was to maintain the UK railways decided that it was a good idea to outsource all of its engineering capability. It turned out that Railtrack did not have the ability to manage, monitor or access the state of work carried out by contractors. Some details and links in... http://en.wikipedia.org/wiki/Railtrack#Founding http://en.wikipedia.org/wiki/Railtrack#Founding A good lesson. Accessing which are the core functions of your company. And the ability to execute them without failure.
  - arethuza 14y ago
    "who made bad deals and lost money" Quite an understatement that, the €71bn RBS acquisition of part of ABN Amro often makes it into the lists of "worst business deals ever": http://www.independent.co.uk/news/business/analysis-and-features/was-abn-the-worst-takeover-deal-ever-1451520.html http://www.independent.co.uk/news/business/analysis-and-feat...
  - timrobinson 14y ago
    If it's true that this is a manual error, then I have two questions: 1) Why did it take until the early hours of Wednesday to find out? Why not run a manual check after the change went in? 2) If they found out on Wednesday morning, why did it take until Friday - after three failed batch runs - to fix it?
    - ovi256 14y ago
      To both your questions, the answer is lack of expertise at hand. People who knew the smells and idiosyncrasies of the systems were not there anymore. A good engineer (as I'm sure there are plenty in the new offshore teams) can fix mostly anything, he'll just take some time if he never did it before. Now they learned how long those delays were. Maybe in their next business case for off-shoring, they'll plug in realistic numbers for the risk of downtime and its cost.
kahawe 14y ago
> there's great value with legacy systems in not getting rid of the people who have been working on them for the last 20 years, and know all the idiosyncrasies and foibles. As someone who unfortunately has been in touch with CA-7 and who comes from a strong Unix/Linux background I can tell you I wouldn't want to touch that thing with a 30ft pole and on top of that these batches are usually extremely complicated in terms of dependencies and a ton of history and lessons-learned go into building those chains and the whole system is so complex yet so critical, we have 1 or 2 guys doing graveyard shifts for nothing but watching CA7 do its thing because if anything fails it could also shut down vital parts of the bank. (e.g. because literally some plain text files didn't get copied from A to B, in 2012...) So not only is there possible great value in NOT getting rid of people who basically grew up with the system and know it inside out but when you do get rid of them it should be more than crystal clear to even the most disconnected manager that this is some really serious stuff you are outsourcing and just blindly trusting the friendly sales guy's promises is not going to cut it... it is almost like outsourcing a custom in-house software to an off-shore "expert" and hoping they can just instantly pick things up where your people left it off. Now it is time to really take a very close look at those SLAs that the project team just wanted approved and signed asap....

timrobinson 14y ago

Specifically, it looks like a manual error during a deployment: http://www.guardian.co.uk/technology/2012/jun/25/how-natwest-it-meltdown http://www.guardian.co.uk/technology/2012/jun/25/how-natwest... "It seems whoever made the update to CA-7 managed to delete or corrupt the files which hold the schedule for the overnight jobs, so they did not run, or ran incorrectly"

Zenst 14y ago
The aspect that they didn't export the job queue prior to update is something that cross's the border of neglegence with both feet. Still can't understand why a bank that had people who made bad deals and lost money causing the bank to partialy fail are kept inplace and paid bonus's and the IT people who did there job well are replaced by cheap labour external to the country and this is at a time when there going on about TAX evasion, this too me makes no sence and is why I don't run banks :|.
- Isofarro 14y ago
  IT is seen as an overhead, and as something that doesn't contribute directly to profitability. When compared to dealers, dealers bring in the profit, IT slurps it away.
  - Zenst 14y ago
    Very true sadly. Don't see many outsourced HR and accounts sections though. Sad part is that when it goes wrong it does effect the balance sheet. Even sadder is how they internaly cost IT and do it wrong. Remove the IT and see how many people/time is needed to do the same job and that is the true cost/potentual impact of IT. Sadly though that is never done and only comes to light when things fail and then they blame IT and not the effects of seagul managment, budget cuts etc.
    - gadders 14y ago
      Oh man. Please, please, please outsource HR. Preferably to Mars. Biggest waste of space in the whole company.
  - sbarlster 14y ago
    Interesting that a retail bank whose core business is to process transactions takes the decision to outsource that capability. And then is exposed as unable to monitor/control the outsourced operation. Some similarities to the now defunct Railtrack whose remit was to maintain the UK railways decided that it was a good idea to outsource all of its engineering capability. It turned out that Railtrack did not have the ability to manage, monitor or access the state of work carried out by contractors. Some details and links in... http://en.wikipedia.org/wiki/Railtrack#Founding http://en.wikipedia.org/wiki/Railtrack#Founding A good lesson. Accessing which are the core functions of your company. And the ability to execute them without failure.
- arethuza 14y ago
  "who made bad deals and lost money" Quite an understatement that, the €71bn RBS acquisition of part of ABN Amro often makes it into the lists of "worst business deals ever": http://www.independent.co.uk/news/business/analysis-and-features/was-abn-the-worst-takeover-deal-ever-1451520.html http://www.independent.co.uk/news/business/analysis-and-feat...
- timrobinson 14y ago
  If it's true that this is a manual error, then I have two questions: 1) Why did it take until the early hours of Wednesday to find out? Why not run a manual check after the change went in? 2) If they found out on Wednesday morning, why did it take until Friday - after three failed batch runs - to fix it?
  - ovi256 14y ago
    To both your questions, the answer is lack of expertise at hand. People who knew the smells and idiosyncrasies of the systems were not there anymore. A good engineer (as I'm sure there are plenty in the new offshore teams) can fix mostly anything, he'll just take some time if he never did it before. Now they learned how long those delays were. Maybe in their next business case for off-shoring, they'll plug in realistic numbers for the risk of downtime and its cost.

kahawe 14y ago

> there's great value with legacy systems in not getting rid of the people who have been working on them for the last 20 years, and know all the idiosyncrasies and foibles. As someone who unfortunately has been in touch with CA-7 and who comes from a strong Unix/Linux background I can tell you I wouldn't want to touch that thing with a 30ft pole and on top of that these batches are usually extremely complicated in terms of dependencies and a ton of history and lessons-learned go into building those chains and the whole system is so complex yet so critical, we have 1 or 2 guys doing graveyard shifts for nothing but watching CA7 do its thing because if anything fails it could also shut down vital parts of the bank. (e.g. because literally some plain text files didn't get copied from A to B, in 2012...) So not only is there possible great value in NOT getting rid of people who basically grew up with the system and know it inside out but when you do get rid of them it should be more than crystal clear to even the most disconnected manager that this is some really serious stuff you are outsourcing and just blindly trusting the friendly sales guy's promises is not going to cut it... it is almost like outsourcing a custom in-house software to an off-shore "expert" and hoping they can just instantly pick things up where your people left it off. Now it is time to really take a very close look at those SLAs that the project team just wanted approved and signed asap....