10 ms·
The IT 'cowboys' didn't ride into sunset. They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastruc
by romwell 8y ago
The IT 'cowboys' didn't ride into sunset.
They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems.
FTA:
>One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages.
>In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated.
Also FTA:
>Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis.
>The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million).
So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job.
The 'problem' is entirely self-made.
- repolfx 8y agoSelf made ... by both the banks and the programmers. As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer. Perhaps they are cowboy developers in more ways than one.
- Ensorceled 8y agoRight. Those developers who were given days and months to document their work deliberately did nothing instead. Wait, no, I've literally NEVER heard of management asking developers to delay the next release to document their work.
- bitwize 8y agoCouldn't they just find the time to document while they're developing? Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.
- bigmanwalter 8y agoNot really. Code needs to solidify before you document it. Documentation is best saved for after a major release.
- Ensorceled 8y agoExactly. I'm always surprised when people think that it's easy to find time for documentation and that you only skipped that step because you are lazy and/or unprofessional.
- repolfx 8y agoHonestly, I know that's the conventional wisdom, but I've never once heard an engineer go to their manager and say, "I'd like to spend some time documenting this important process you asked me to implement" and being told "no, we don't have time to document! just keep it all in your head!" I'm sure this literally never happens. What I do see quite frequently is developers who don't like writing documentation finding ways to not do it or justify doing a poor job.
- arethuza 8y agoOnly time I've encountered a desperate fixation by a CIO on documenting applications was when the relevant development team was about to be laid off. Fortunately I had already resigned at that point.
- vetinari 8y agoIf you want to blame someone, you can always say that he didn't document his stuff enough. The problem is, that there is no standard, what's "enough", so it is easy to move the goal posts. Combine that with fast paced environment and putting out fires constantly, and you have a recipe for disaster.
- lawlessone 8y ago>If you want to blame someone, you can always say that he didn't document his stuff enough. This, he could have documented it perfectly well in a time when everyone knew COBOL. But that documentation would all be useless to me today with my knowledge of Java and Kotlin, but understandable to someone else from is generation that works with COBOL.
- setr 8y agoTbf, it only really makes sense for the individual programmer to properly document code if there’s a decent chance they’ll forget (eg some obscure part of the codebase), or if you expect a reasonable chance that someone will have to read the code without you. Otherwise, its useful, but not necessary: you just go to the guru (perhaps yourself); his living body is the documentation. If you’re thinking that only you’ll ever read it, or you’ll be working on this codebase for the next 30 years, it doesn’t really matter. Peoper Documentation is a cultural activity required for teams, especially those with shifting workers, as is common now. And should really be enforced by the managers, not the programmers thenselves. Its significantly less necessary when you can expect to spend most of your career on a single codebase, a workstyle these cobol programmers presumably came from.
- TomMarius 8y agoWhat happens when a bus hits the guru?
- zoggenhoff 8y agoGuru meditation error.
- sekh60 8y agoYour DR planning better include a necromancer.
- llcoolv 8y agoyeah. this is how it had been done before the invention of writing. time has proven it not to be a very efficient method.
- tcbawo 8y agoI try to design and name interfaces clearly and comment code for myself. If it's been a few weeks since I've looked at something it helps me snap back. Also, I would never approach a codebase as if I will be working on it for 30 years. Job security comes from marketability, not obscurity.
- oldandtired 8y agoYou, obviously, have no idea how hard it is to write decent documentation for any system. Especially when it has been in existence for many years. Good documentation is hard and anyone who thinks it is not does not have a clue as to what is needed for the purposes of maintaining any piece of software. It is a very rare event to see documentation of a quality that explains all the interrelationships, reasons for the choice, etc, etc, etc. I have in the past been required to provide all the documentation for projects I have worked on, functional and technical specifications, user guides and cheat sheets, detailed design documents and all of these take considerable time to write well. I can't say I am expert in this area because I am not. But I have done a lot better than most of the projects that I have had to interface to over the decades. Documentation is a low priority for many management teams, they do not see the use in the kind of details that are needed. They do not give it sufficient support to do the job well. I have seen and worked with people who have the skill to write the kind of documentation that is actually needed and they are a rare breed, well worth any money you pay them. When companies complain about "cowboys", they often do not see that it is they who are the "cowboys".
- repolfx 8y agoYou, obviously, have no idea how hard it is to write decent documentation for any system. Especially when it has been in existence for many years. I've been programming for decades and my current project is frequently praised by other developers who work with it (it's a framework) for the quality of its documentation. So, in fact, I do have such an idea. It's hard work. We do it anyway because we're professionals. The fact that people frequently don't do it doesn't change this. One technique I use is to keep documentation in the same repository as the code, and force people to write docs via the code review process. That is, a proposed change doesn't pass review until it also updates the documentation files. People want to get their code merged, so they know they'll have to take docs seriously or else it'll get picked up in review. Now maybe back in the mainframe days there was no culture of code review, but these are banks we're talking about. They excel at process. So it seems kind of unlikely they'd not want things documented, if management had understood what needed to be documented in the first place. But ultimately it was only the employees who knew that.
- sharemywin 8y agoHow many new devs are willing to even touch a COBOL program. documentation or not.
- ams6110 8y agoFor enough money, I'd say many. COBOL isn't sexy, but it's not difficult either. And it's demonstrated a lot more staying power than most other languages.
- diggernet 8y agoDarn right! I'd cheerfully learn Cobol for a job with the right parameters. I once took an RPG course in order to apply for one (though sadly did not get it).
- toast0 8y agoThe only way to have really quality documentation is to have a lot of people reading it and providing feedback, over time. It also doesn't hurt when a quality technical writer actually writes or edits the documentation in consultation with the people who designed or implemented the system.
- netcan 8y agoThe bank and "IT Cowboy's" technology teams have probably spent years on "Ayne Rand's Train." Projects headed for failure. Everyone is concerned with avoiding/pinning blame instead of building software that works. Deep knowledge of contracts instead of deep knowledge of code. These are common pathologies. I agree though. This is incompetence at technology from strategy to the details. It's important to understand how much technology gets built this way. Usually these projects are not outright, public failures.
- geofft 8y agoThe problem is that these decisions are rational - you get personally rewarded for getting yourself promoted or putting neat projects on your resume (which will increase your salary in your next job), but not for helping the business long-term. Avoiding blame is far more personally valuable, in terms of reward for time, than building good things. If you take "I swear by my life and my love of it that I will never live for the sake of another man, nor ask another man to live for mine" seriously, this is the end result. Why should any IC or low-level manager at Taggart Transcontinental live for the sake of the Taggarts? But how can the railroad succeed without ICs and low-level managers? If you want to fix this, don't ask people to live altruistically; instead, fix their incentives. Giving people cool projects such that they feel personally fulfilled when they succeed whether or not the business recognizes it is one option, but not particularly reliable or scalable.
- netcan 8y agoIm not buying her interpretation of why the train scene happens either, but I do think the scene was kind of a brilliant metaphor for this sort of thing. Whenever you have a systemically pathological situation like big, enterprise software projects often get into... it's probably an incentive problem. Tldr agreed
- Ensorceled 8y agoAnd, this shit has been going on for 20 years, my uncle (now in his 80's) made bank Cobol contracting and subcontracting EXACTLY the same way in the run up to Y2K: hiring developers "early retired".
- crdb 8y ago> a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job "a bank" is technically incorrect. The decision was a perfectly rational prisoner's dilemma defection from one executive (cut costs by firing the "expensive employees" and hiring "cheaper and newer"), who went on to be promoted on that basis, and it was the next generation of executives who had to deal with the mess and "hired IBM" so they could absolve responsibility of a situation with no visible upside if successful and tons of visible downside if not.
- noobermin 8y agoYou added a wrinkle that seems to partially explain it. Moreover, it does not and should absolve the relevant parties of their responsibility.
- crdb 8y agoThe ironic thing about your reply is that it is crafted in exactly the type of management speak that executives used to convince stakeholders in both occasions. "absolve relevant parties of their responsibilities" - what relevant parties? What responsibilities? It sounds great and responsible, giving a good impression of dismissal of the point made, but the wording is so general as to be impossible to act upon - just as with "a bank". I'm sure standardised checklists were ticked, committees reviewed the decisions and came to the conclusion that it was best, the board was satisfied with the new direction taken by the CTO's organisation, and the shareholders happy with the cost reduction at the time as described by the board. To paraphrase Thatcher, what colour trousers do the relevant parties wear? You see this defection pattern everywhere. For example, the never ending stream of useless new features in most Google and some Microsoft products which coincidentally get slower and buggier every year, because you can get promoted for the visible addition of a feature as a PM, but it is much harder to prove that getting the Gmail app load time down by 70% has stopped the feeding of your best power users to the competition (Calendar is too easy a target for criticism). The classic defector on the tech side is the "ninja rockstar developer" whose prolific output of code and visually attractive but functionally superficial features is matched only by the amount of frustration generated after they are promoted somewhere else or changes jobs if it doesn't happen fast enough, and the rest of the team has to deal with the technical debt and new, unrealistic user expectations. Security is another good candidate for shortcuts, since there is no visible upside to good security (maybe you were just not attacked) and tons of downside for taking responsibility since you make yourself a visible scapegoat for blame when a crisis does happen. And note that none of these examples are in "move fast and break things" i.e. "do illegal things and hope they become legal ex-post" territory! I personally take great pains to understand these dynamics so that I can avoid them in my own company, which is a matter of survival.
- crispyambulance 8y ago> The 'problem' is entirely self-made. And richly deserved. The finance sector can afford it, for now. "Technical debt is your job." -- is what I once heard some PHB say. Not from a finance person but the attitude, I think, is the same.
- FussyZeus 8y agoWhich is a bad attitude in general, but an especially bad one for an industry known to use exceedingly old tech. In all honesty: why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable?
- Rubikan 8y ago> In all honesty: why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable? It is not un-replacable in itself, I'd imagine it is just a huge undertaking and many higher ups are afraid to do it and risk something going wrong.
- bauerd 8y agoNo one who's in a position to decide about COBOL's replacement has any strong incentives to do so. You don't want to be the one exec who made that risky move and crushed half their bank's tech foundation in the process.
- theandrewbailey 8y ago"If it's not broken, don't fix it" is a saying for a reason. This might be it.
- Marazan 8y agoIt's not the language, it is the system that was built. You can't just throw it away.
- yxhuvud 8y agoBanks were among the first to computerize their systems, and switching systems cost a lot of money, regardless of industry. Hence they have a lot of old systems.
- snaky 8y agoThe problem? I wonder whether it was the same VP who got promotion for decision to cut corners a couple of years back, and another promotion for decision to fix it by contractors.
- malgregator 8y agoIt's actually common OPEX, CAPEX optimization. Executives are paid better when doing this, but is's definitely bad for a long term
- nickonline 8y agoI was on that project as a grad. They spent a billion dollars, over 5 years, with a staff of 1,000 strong to replace old software. I would be surprised if even 20 of the 1,000 were touching COBOL.
- AtlasBarfed 8y agoI'm guessing it was an Accenture project based on the outtakes. That's what Accenture does: armies of cheap people for huge contracts. Flip a coin on the outcome. Not that IBM or any of the other top-level consultancies/services behemoths are any good, but Andersen Consulting/Accenture basically invented that approach.
- Gokenstein 8y agoIt takes 980 MBAs to reach the level of schmoozing required to secure the contract.
- onethree 8y agoWhile most of your points are valid, you've misunderstood the last quote. CBA didn't fire people and then need to pay a billion dollars to fix something. They specifcally did migrate to a modern system - they completely replaced their entire legacy core banking system with a brand new one.
- gota 8y agoPardon my ignorance but FTA means 'from the article', yes?
- arminiusreturns 8y agoIt isn't just banks either. I'm a greybeard sysadmin type who has seen the insides of hundreds of companies, and for some reason in the late 2000s it seemed like everyone from law firms to insurance companies decided to fire their IT teams and then pay 3x the money for 1/16 the service from contractor/msp-types... and they wonder why they wallow in tech debt... I'd they even knew what it is, they did fire all the people telling them about it after all!
- nasalgoat 8y agoCAPEX vs OPEX. Operational expenses looks better on the balance sheet (ie. renting versus owning).
- closeparen 8y agoYou don’t own employees. MSP and Cloud are separate concepts. The typical MSP has a bunch of technicians who drive around to client sites and tend to the servers in their closets.