24 ms·

Banks scramble to fix old systems as IT 'cowboys' ride into sunset (2017)

145 points by carlchenet 8y ago

bryanrasmussen 8y ago
didn't we just have this article a few days ago and the consensus was you don't actually get that much money for knowing Cobol?
- isostatic 8y ago
  https://news.ycombinator.com/item?id=18110054 https://news.ycombinator.com/item?id=18110054
- Forge36 8y ago
  I don't think this is a case of knowing Cobol. I think it's a case of company's hiring the experts who were let go. Person A has X years experience with company Y. Person is let go. Mission critical infrastructure starts to fail. Person A is now a contractor with X years experience on that exact system. Sure they can hire a new guy for cheaper, however that appears to be what led up to A being let go. They suddenly need experience and A is willing at a much higher rate.
mikorym 8y ago
I think this needs a "(2017)" string in the title. I also remember seeing this on HN (probably in 2017 then).
tannhaeuser 8y ago
It's cheap to portray COBOL programmers as cowboys, but COBOL and the ecosystem around it is actually one of the most organized and fit-for-purpose programming languages around even if it isn't modern by any means. Time will tell if banks and insurances will fare better with J2EE-based systems (meaning classic Java-based enterprise stacks such as EJBs and portlets running under shudder Websfear, which count as new-fangled compared to COBOL in these circles). I could imagine classic J2EE maintenance is being neglected as career path for millenials, just as COBOL is/was for the generation before. J2EE exposes much more escape hatches into system programming and rope to hang yourself, and at the same time will be around for a long time to come, since Java has been the go-to language for almost all big-time eCommerce and FinTech projects at banks/insurances since 2000. Edit: a-ok, "cowboys" is part of name of that guy's consulting business
- wiz21c 8y ago
  I don't know for COBOL, but for JEE, we've been able to migrate out of weblogic (to jboss) and then from oracle to postgresql (on code which is 10 years old). So even if JEE might be hard to maintain, it allows some flexibility in the base tools you choose. (my view of COBOL is that once you're into a vendor, it's pretty hard to leave to another one because COBOL is close to the metal, I may be totally wrong) but in the long run, the problem is also the business logic. What disappears is the knowledge of the hacks that were applied at the business logic level. And this is much harder than to recover from the code. So I guess one has to make the difference between business logic maintenance and the actual COBOL maintenance. But having rewrote some huge COBOL app in Java, I can say that Java is better structured, at a framework complexity cost.
  - repolfx 8y ago
    Better structured, easier to debug, and the tooling exists to incrementally refactor in a relatively safe way. There are also migration paths to better worlds. IntelliJ can auto-rewrite Java into Kotlin, preserving comments and formatting along the way, and even detecting common Java idioms that have better Kotlin equivalents and rewriting them too. So, not only can you migrate to better app containers, database engines, upgrade your garbage collector and compilers etc, but also even switch to a much more modern language nearly pain free.
    - tcbawo 8y ago
      Regardless of whether Kotlin is considered a better world than Java, I'm skeptical that this sort of automatic translation adds significant value.
      - einarfd 8y ago
        I've been using Intellij's Java to Kotlin conversion a bit lately, and the code it spits out is fairly decent Kotlin, and with a pass through on nullability and switching out Java types with native Kotlin types afterwards. It gets just as good as it would get if I hand translated my self, but it takes a lot less time. I suspect that part of the reason that this works well, is that Java and Kotlin isn't that dissimilar. Doing automated conversion between Java and Scheme for example, would be a lot harder to get right.
romwell 8y ago
The IT 'cowboys' didn't ride into sunset. They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems. FTA: >One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages. >In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated. Also FTA: >Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis. >The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million). So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job. The 'problem' is entirely self-made.
- repolfx 8y ago
  Self made ... by both the banks and the programmers. As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer. Perhaps they are cowboy developers in more ways than one.
  - Ensorceled 8y ago
    Right. Those developers who were given days and months to document their work deliberately did nothing instead. Wait, no, I've literally NEVER heard of management asking developers to delay the next release to document their work.
    - bitwize 8y ago
      Couldn't they just find the time to document while they're developing? Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.
      - 8y ago
pbadenski 8y ago
100$ an hour? It doesn't sound like a serious expense at all.
- LeonM 8y ago
  I guess because good programmers are not good in business. If a bank or any big corp is in trouble with legacy software and they approach you to fix it, charge them 500/h or more. That is still cheaper than the time they need to find someone else.
  - corobo 8y ago
    Charge them a weekly rate and 500/h is probably still undervaluing yourself [1] https://www.kalzumeus.com/2015/05/01/talking-about-money/#consulting-rates https://www.kalzumeus.com/2015/05/01/talking-about-money/#co...
- quickthrower2 8y ago
  There is not a shortage of Cobol programmers.
  - 3rdAccount 8y ago
    No, but Cobol + bank experience is different. There's the whole business terminology and overview you need to have to actually understand and fix the code.
arethuza 8y ago
I watched "The Bank That Almost Broke Britain" about the RBS debacle and it seems a bit rich for bankers to be calling anyone cowboys: https://www.bbc.co.uk/programmes/b0bmbhzb https://www.bbc.co.uk/programmes/b0bmbhzb
xte 8y ago
I use this news as a ignition for another point: in the past for business reasons we have evolved concept of "platforms" witch means independent software layers build like a trailer to carry on other software on top. Before we have another concept, those from LispM, the "system" as a single entity of well_integrated stuff. We have many example of those two way of thinking today: on "platform" side we have snap, flatpack, appimage, lx[cd]/docker, ... on system side we have Emacs, NixOS, GuixSD, ... Well, for years the "platform" model seems to be the most reasonable, today seems ancient MIT&c hackers ware right, "system" approach is better. Simple to manage, for good software, do not hide bad practice, force collaboration etc. On datacenters today and not from today we do substantially the same, in the past datacenters was a big collection of independent computers, now their are substantially all "a single computer" (even before The datacenter as a Computer by Google), made of many well_integrated components. My poor English may not help, but I hope I have been clear up there, if so, what you think?
- 3rdAccount 8y ago
  Are you just saying the old approach is better than docker containers everywhere? I think each probably has a place.
taneq 8y ago
Using "cowboys" as a pejorative term ignores the fact that cowboys are fine in their natural environment: The wild west, where self-sufficiency, resilience, adaptability and creativity are more important than engineering processes, documentation, and bureaucracy. A cowboy is a loose cannon in the big city, but a "proper software engineer" is useless in the wild west. These banks have just realised that there are a few ranches left.
- sdrothrock 8y ago
  > Using "cowboys" as a pejorative term It's not being used as a perjorative term. It's talking about a company called Cobol Cowboys, itself inspired by Space Cowboys, about a bunch of old guys called back into service to help with an emergency. Charmingly enough, their motto is "not our first rodeo."
  - Wohlf 8y ago
    They're also from Texas, making cowboys even more appropriate.
  - Jaruzel 8y ago
    The motto is also spoken by the 'cowboys' in 'Space Cowboys' as well.
- paulcole 8y ago
  What specifically in the article made you think "cowboy" was being used as a pejorative? After the title the first use refers to the name of a company founded by the person mentioned in the first sentence of the article.
- watwut 8y ago
  Not sure why you are bringing mythology into this.
zoggenhoff 8y ago
Another submission about cobol. https://news.ycombinator.com/item?id=14202585 https://news.ycombinator.com/item?id=14202585 https://news.ycombinator.com/item?id=17979417 https://news.ycombinator.com/item?id=17979417 https://news.ycombinator.com/item?id=18110054 https://news.ycombinator.com/item?id=18110054
polskibus 8y ago
[2017]
RickJWagner 8y ago
I started in COBOL back in the 90s. When I was working in that shop, one of my older co-workers told me about a guy he knew. That guy left his service/repair gig at IBM to start his own shop, he knew how to fix punch-card machines. For the decade I worked there, my buddy told me that his pal made a very good living repairing those machines, even though they were long out of fashion and 'there was no market'. Seems the same is true for software, too.
- ams6110 8y ago
  I worked in an IBM shop in the early 1990s, at the time they had a big chart on the wall titled "Punch Card Elimination Project." It was one of those big thermometer charts like United Way drives use to document how close they are to their contribution goal. As I recall it was at about 70%.
- techopoly 8y ago
  Fashion is such a driver for IT study. Just try to find a good book on modern AIX -- they aren't really out there. But AIX is still very much in use at the enterprise level. Computer trends and actual use don't always match up. Which can be a boon for those who realize they can specialize in unfashionable technology.
ataturk 8y ago
This is so stupid. The asshole companies had every opportunity to fix and modernize their crap they just chose to ride the investments beyond any reasonable timeframe. Let them go out of business.
mxuribe 8y ago
One little side comment: within one of the photos shown as part of this article, they show a photo of Grace Hopper, but no mention nor credit of COBOL's roots to her, nor her contributions to COBOL. So, the past was only made up of cowboys, but not cowgirls, eh? Tsk tsk.
_pmf_ 8y ago
This meme has been around for at least 15 years. Banks will be fine, since they have unlimited resources.
bsg75 8y ago
> was brought in as a contractor to the same bank to fix issues management had not anticipated And continue to ignore. This is not a new issue, but the same story repeating over and over. Management purely via cost control is a terrible approach.
SeanFerree 8y ago
Nice article! Great info
cafard 8y ago
Is it by any chance COBOL season at HN? This is about the third COBOL article in a week.
thanatropism 8y ago
> The risk is “not so much that an individual may have retired,” Andrew Starrs, group technology officer at consulting firm Accenture PLC, said. “He may have expired, so there is no option to get him or her to come back.” Is "expired" a standard English synonym for pining for the fjords?
- Balero 8y ago
  I think in this case it means "died".
- bklaasen 8y ago
  It is. "He expired on the couch" is fairly common. It has the connotation of a quiet death. One does not expire while solo climbing or BASE jumping.
tmaly 8y ago
> But COBOL veterans say it takes more than just knowing the language itself. COBOL-based systems vary widely and original programmers rarely wrote handbooks, making trouble-shooting difficult for others. There never seems to be time for documentation, specifications, and clean code
ovrkil 8y ago
This article really hit home for me because this is exactly what my current career consists of. I am coverting old AS400s running RPG to QNX servers running C. And like the Cobol Cowboy's say the financial institutions will pay what ever you demand. I owe all this work to a college professor that encouraged me to thoroughly learn RPG for this very reason. Thank you Professor Her.

145 points by carlchenet 8y ago

didn't we just have this article a few days ago and the consensus was you don't actually get that much money for knowing Cobol?

isostatic 8y ago
https://news.ycombinator.com/item?id=18110054 https://news.ycombinator.com/item?id=18110054
Forge36 8y ago
I don't think this is a case of knowing Cobol. I think it's a case of company's hiring the experts who were let go. Person A has X years experience with company Y. Person is let go. Mission critical infrastructure starts to fail. Person A is now a contractor with X years experience on that exact system. Sure they can hire a new guy for cheaper, however that appears to be what led up to A being let go. They suddenly need experience and A is willing at a much higher rate.

mikorym 8y ago

I think this needs a "(2017)" string in the title. I also remember seeing this on HN (probably in 2017 then).

tannhaeuser 8y ago

It's cheap to portray COBOL programmers as cowboys, but COBOL and the ecosystem around it is actually one of the most organized and fit-for-purpose programming languages around even if it isn't modern by any means. Time will tell if banks and insurances will fare better with J2EE-based systems (meaning classic Java-based enterprise stacks such as EJBs and portlets running under shudder Websfear, which count as new-fangled compared to COBOL in these circles). I could imagine classic J2EE maintenance is being neglected as career path for millenials, just as COBOL is/was for the generation before. J2EE exposes much more escape hatches into system programming and rope to hang yourself, and at the same time will be around for a long time to come, since Java has been the go-to language for almost all big-time eCommerce and FinTech projects at banks/insurances since 2000. Edit: a-ok, "cowboys" is part of name of that guy's consulting business

wiz21c 8y ago
I don't know for COBOL, but for JEE, we've been able to migrate out of weblogic (to jboss) and then from oracle to postgresql (on code which is 10 years old). So even if JEE might be hard to maintain, it allows some flexibility in the base tools you choose. (my view of COBOL is that once you're into a vendor, it's pretty hard to leave to another one because COBOL is close to the metal, I may be totally wrong) but in the long run, the problem is also the business logic. What disappears is the knowledge of the hacks that were applied at the business logic level. And this is much harder than to recover from the code. So I guess one has to make the difference between business logic maintenance and the actual COBOL maintenance. But having rewrote some huge COBOL app in Java, I can say that Java is better structured, at a framework complexity cost.
- repolfx 8y ago
  Better structured, easier to debug, and the tooling exists to incrementally refactor in a relatively safe way. There are also migration paths to better worlds. IntelliJ can auto-rewrite Java into Kotlin, preserving comments and formatting along the way, and even detecting common Java idioms that have better Kotlin equivalents and rewriting them too. So, not only can you migrate to better app containers, database engines, upgrade your garbage collector and compilers etc, but also even switch to a much more modern language nearly pain free.
  - tcbawo 8y ago
    Regardless of whether Kotlin is considered a better world than Java, I'm skeptical that this sort of automatic translation adds significant value.
    - einarfd 8y ago
      I've been using Intellij's Java to Kotlin conversion a bit lately, and the code it spits out is fairly decent Kotlin, and with a pass through on nullability and switching out Java types with native Kotlin types afterwards. It gets just as good as it would get if I hand translated my self, but it takes a lot less time. I suspect that part of the reason that this works well, is that Java and Kotlin isn't that dissimilar. Doing automated conversion between Java and Scheme for example, would be a lot harder to get right.

romwell 8y ago

The IT 'cowboys' didn't ride into sunset. They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems. FTA: >One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages. >In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated. Also FTA: >Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis. >The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million). So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job. The 'problem' is entirely self-made.

repolfx 8y ago
Self made ... by both the banks and the programmers. As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer. Perhaps they are cowboy developers in more ways than one.
- Ensorceled 8y ago
  Right. Those developers who were given days and months to document their work deliberately did nothing instead. Wait, no, I've literally NEVER heard of management asking developers to delay the next release to document their work.
  - bitwize 8y ago
    Couldn't they just find the time to document while they're developing? Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.
    - 8y ago

pbadenski 8y ago

100$ an hour? It doesn't sound like a serious expense at all.

LeonM 8y ago
I guess because good programmers are not good in business. If a bank or any big corp is in trouble with legacy software and they approach you to fix it, charge them 500/h or more. That is still cheaper than the time they need to find someone else.
- corobo 8y ago
  Charge them a weekly rate and 500/h is probably still undervaluing yourself [1] https://www.kalzumeus.com/2015/05/01/talking-about-money/#consulting-rates https://www.kalzumeus.com/2015/05/01/talking-about-money/#co...
quickthrower2 8y ago
There is not a shortage of Cobol programmers.
- 3rdAccount 8y ago
  No, but Cobol + bank experience is different. There's the whole business terminology and overview you need to have to actually understand and fix the code.

arethuza 8y ago

I watched "The Bank That Almost Broke Britain" about the RBS debacle and it seems a bit rich for bankers to be calling anyone cowboys: https://www.bbc.co.uk/programmes/b0bmbhzb https://www.bbc.co.uk/programmes/b0bmbhzb

xte 8y ago

I use this news as a ignition for another point: in the past for business reasons we have evolved concept of "platforms" witch means independent software layers build like a trailer to carry on other software on top. Before we have another concept, those from LispM, the "system" as a single entity of well_integrated stuff. We have many example of those two way of thinking today: on "platform" side we have snap, flatpack, appimage, lx[cd]/docker, ... on system side we have Emacs, NixOS, GuixSD, ... Well, for years the "platform" model seems to be the most reasonable, today seems ancient MIT&c hackers ware right, "system" approach is better. Simple to manage, for good software, do not hide bad practice, force collaboration etc. On datacenters today and not from today we do substantially the same, in the past datacenters was a big collection of independent computers, now their are substantially all "a single computer" (even before The datacenter as a Computer by Google), made of many well_integrated components. My poor English may not help, but I hope I have been clear up there, if so, what you think?

3rdAccount 8y ago
Are you just saying the old approach is better than docker containers everywhere? I think each probably has a place.

taneq 8y ago

Using "cowboys" as a pejorative term ignores the fact that cowboys are fine in their natural environment: The wild west, where self-sufficiency, resilience, adaptability and creativity are more important than engineering processes, documentation, and bureaucracy. A cowboy is a loose cannon in the big city, but a "proper software engineer" is useless in the wild west. These banks have just realised that there are a few ranches left.

sdrothrock 8y ago
> Using "cowboys" as a pejorative term It's not being used as a perjorative term. It's talking about a company called Cobol Cowboys, itself inspired by Space Cowboys, about a bunch of old guys called back into service to help with an emergency. Charmingly enough, their motto is "not our first rodeo."
- Wohlf 8y ago
  They're also from Texas, making cowboys even more appropriate.
- Jaruzel 8y ago
  The motto is also spoken by the 'cowboys' in 'Space Cowboys' as well.
paulcole 8y ago
What specifically in the article made you think "cowboy" was being used as a pejorative? After the title the first use refers to the name of a company founded by the person mentioned in the first sentence of the article.
watwut 8y ago
Not sure why you are bringing mythology into this.

zoggenhoff 8y ago

Another submission about cobol. https://news.ycombinator.com/item?id=14202585 https://news.ycombinator.com/item?id=14202585 https://news.ycombinator.com/item?id=17979417 https://news.ycombinator.com/item?id=17979417 https://news.ycombinator.com/item?id=18110054 https://news.ycombinator.com/item?id=18110054

polskibus 8y ago

[2017]

RickJWagner 8y ago

I started in COBOL back in the 90s. When I was working in that shop, one of my older co-workers told me about a guy he knew. That guy left his service/repair gig at IBM to start his own shop, he knew how to fix punch-card machines. For the decade I worked there, my buddy told me that his pal made a very good living repairing those machines, even though they were long out of fashion and 'there was no market'. Seems the same is true for software, too.

ams6110 8y ago
I worked in an IBM shop in the early 1990s, at the time they had a big chart on the wall titled "Punch Card Elimination Project." It was one of those big thermometer charts like United Way drives use to document how close they are to their contribution goal. As I recall it was at about 70%.
techopoly 8y ago
Fashion is such a driver for IT study. Just try to find a good book on modern AIX -- they aren't really out there. But AIX is still very much in use at the enterprise level. Computer trends and actual use don't always match up. Which can be a boon for those who realize they can specialize in unfashionable technology.

ataturk 8y ago

This is so stupid. The asshole companies had every opportunity to fix and modernize their crap they just chose to ride the investments beyond any reasonable timeframe. Let them go out of business.

mxuribe 8y ago

One little side comment: within one of the photos shown as part of this article, they show a photo of Grace Hopper, but no mention nor credit of COBOL's roots to her, nor her contributions to COBOL. So, the past was only made up of cowboys, but not cowgirls, eh? Tsk tsk.

_pmf_ 8y ago

This meme has been around for at least 15 years. Banks will be fine, since they have unlimited resources.

bsg75 8y ago

> was brought in as a contractor to the same bank to fix issues management had not anticipated And continue to ignore. This is not a new issue, but the same story repeating over and over. Management purely via cost control is a terrible approach.

SeanFerree 8y ago

Nice article! Great info

cafard 8y ago

Is it by any chance COBOL season at HN? This is about the third COBOL article in a week.

thanatropism 8y ago

> The risk is “not so much that an individual may have retired,” Andrew Starrs, group technology officer at consulting firm Accenture PLC, said. “He may have expired, so there is no option to get him or her to come back.” Is "expired" a standard English synonym for pining for the fjords?

Balero 8y ago
I think in this case it means "died".
bklaasen 8y ago
It is. "He expired on the couch" is fairly common. It has the connotation of a quiet death. One does not expire while solo climbing or BASE jumping.

tmaly 8y ago

> But COBOL veterans say it takes more than just knowing the language itself. COBOL-based systems vary widely and original programmers rarely wrote handbooks, making trouble-shooting difficult for others. There never seems to be time for documentation, specifications, and clean code

ovrkil 8y ago

This article really hit home for me because this is exactly what my current career consists of. I am coverting old AS400s running RPG to QNX servers running C. And like the Cobol Cowboy's say the financial institutions will pay what ever you demand. I owe all this work to a college professor that encouraged me to thoroughly learn RPG for this very reason. Thank you Professor Her.