17 ms·

I feel bad for the rank-and-file developers (the ones without decision-making power) who are stuck trying to fix all this in what are probably insanely bad work

by sbarre 9y ago

I feel bad for the rank-and-file developers (the ones without decision-making power) who are stuck trying to fix all this in what are probably insanely bad working conditions right now. Yeah yeah "get a different job" or "you made the mess, you fix it" but massive failures like this aren't orchestrated by the front-line developers, they happen because people many steps removed from the work make bad executive decisions. A deadline was probably set long ago, and no one adjusted it when unexpected problems and complexities arose.

joelhaasnoot 9y ago
Sure, this definitely is true, but when your website or app is throwing an IndexOutOfBoundsException left and a InvalidBeanException right, your code quality also has issues. Security auditors (good or bad as they may be) would likely have your head for any leaked exception, for all they know the application may also be leaking other info.
- edf13 9y ago
  My guess is the code just wasn't ready for production... someone higher up was screaming they didn't care and a deadline is a deadline so push it out.
  - EnderMB 9y ago
    Alongside this, it's all down to risk. I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great. For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on. A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick. The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.
    - ownagefool 9y ago
      I've worked with the UK banks. From experience they're typically large outsourcing firms that contract their work out to the large consultancies that are more interested in appearance and winning bun fights than delivering quality. I just left one of these orgs about a month ago where the program CTO of the bank (title inflation) wanted to hire me direct because they believe they need to have a core competency that isn't aligned with external political agendas, but largely the other senior managers just see having their own techs as a risk because they can't outsource the blame. That and directly answering questions or questioning process upsets a lot of people because they have fiedoms. Like someone asks you for all your IP address you have on cloud and what server they're associated with. They have no ability to handle the dynamic nature of these changing but their job is to write them down so you need to fall inline sorta deal...
- Shivetya 9y ago
  Signs of a QA team that only unit tests or only is allowed to unit test to a change management system that is too easily bypassed. so while I can too place some blame on developers it only takes one rogue and a bad change management system to get code out that should have stayed home
- sbarre 9y ago
  I agree with you entirely, but even then someone somewhere overruled those security auditors, right? And if this job was falling behind, they may have contracted additional developers who came in to do work in a hurry, which can lead to an even worse and misaligned codebase, etc..
PeterisP 9y ago
It seems possible that the deadline could not be adjusted - they were running on infrastructure that was not their own but rented for a limited time from their old owner, who is now their competitor (and has no interest in their success). As the time agreed during the restructuring runs out, they likely don't have the option of continuing as before for some time more, either they switch to something new, no matter how bad it is, or shut down.
- sbarre 9y ago
  Meh, I don't believe this.. A deal can always be made.. The old owner surely had a price (because everyone has a price) that could have been met to keep the existing systems up and running. TSB just chose to screw their customers, and make them pay for the problems, rather than spend the money to pay for the problems themselves.
sevensor 9y ago
I expect that, when the dust settles, we'll see an insider account of how this all went wrong, including inexperienced developers, management ignoring numerous warnings, and an intractable disaster of a codebase. Edit: Turns out I missed the most important factor. According to https://news.ycombinator.com/item?id=16939848 https://news.ycombinator.com/item?id=16939848, they had brought in IBM.

by sbarre 9y ago

joelhaasnoot 9y ago

Sure, this definitely is true, but when your website or app is throwing an IndexOutOfBoundsException left and a InvalidBeanException right, your code quality also has issues. Security auditors (good or bad as they may be) would likely have your head for any leaked exception, for all they know the application may also be leaking other info.

edf13 9y ago
My guess is the code just wasn't ready for production... someone higher up was screaming they didn't care and a deadline is a deadline so push it out.
- EnderMB 9y ago
  Alongside this, it's all down to risk. I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great. For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on. A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick. The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.
  - ownagefool 9y ago
    I've worked with the UK banks. From experience they're typically large outsourcing firms that contract their work out to the large consultancies that are more interested in appearance and winning bun fights than delivering quality. I just left one of these orgs about a month ago where the program CTO of the bank (title inflation) wanted to hire me direct because they believe they need to have a core competency that isn't aligned with external political agendas, but largely the other senior managers just see having their own techs as a risk because they can't outsource the blame. That and directly answering questions or questioning process upsets a lot of people because they have fiedoms. Like someone asks you for all your IP address you have on cloud and what server they're associated with. They have no ability to handle the dynamic nature of these changing but their job is to write them down so you need to fall inline sorta deal...
Shivetya 9y ago
Signs of a QA team that only unit tests or only is allowed to unit test to a change management system that is too easily bypassed. so while I can too place some blame on developers it only takes one rogue and a bad change management system to get code out that should have stayed home
sbarre 9y ago
I agree with you entirely, but even then someone somewhere overruled those security auditors, right? And if this job was falling behind, they may have contracted additional developers who came in to do work in a hurry, which can lead to an even worse and misaligned codebase, etc..

PeterisP 9y ago

It seems possible that the deadline could not be adjusted - they were running on infrastructure that was not their own but rented for a limited time from their old owner, who is now their competitor (and has no interest in their success). As the time agreed during the restructuring runs out, they likely don't have the option of continuing as before for some time more, either they switch to something new, no matter how bad it is, or shut down.

sbarre 9y ago
Meh, I don't believe this.. A deal can always be made.. The old owner surely had a price (because everyone has a price) that could have been met to keep the existing systems up and running. TSB just chose to screw their customers, and make them pay for the problems, rather than spend the money to pay for the problems themselves.

sevensor 9y ago

I expect that, when the dust settles, we'll see an insider account of how this all went wrong, including inexperienced developers, management ignoring numerous warnings, and an intractable disaster of a codebase. Edit: Turns out I missed the most important factor. According to https://news.ycombinator.com/item?id=16939848 https://news.ycombinator.com/item?id=16939848, they had brought in IBM.