11 ms·

The governor of NJ went live on TV to request COBOL programmers

59 points by saadalem 7y ago

COBOL is operating for a number of practical reasons in the US. Universities stopped training because it was not—glamorous. Now NJ needs 100s of programmers—TODAY.

Tepix 7y ago
URL? What does he need them for?
okareaman 7y ago
link: https://www.youtube.com/watch?v=nlJqlWSPJeI https://www.youtube.com/watch?v=nlJqlWSPJeI
- choward 7y ago
  Cobalt? Haha. I don't think anyone on earth has experience in that. It better pay well.
  - xxpor 7y ago
    Some child miners in DRC might disagree :(
steven77723 7y ago
Discussed at: https://news.ycombinator.com/item?id=22782097 https://news.ycombinator.com/item?id=22782097
tomohawk 7y ago
Must not be willing to pay enough or be hard to work with. At the right price, 100 people should not be hard to find. Of course, they probably would not be in this position if they were willing to put money into the system or keep people on retainer at least.
- dreary_dugong 7y ago
  I recommend taking a look at the other discussions linked. Unfortunately the situation is more nuanced then "just pay them more."
  - renewiltord 7y ago
    I tried reading the other threads but they are not very information-dense. It's mostly people arguing about whether governments are good employers. Would you mind sharing a quick summary if you've synthesized the position?
sharkweek 7y ago
My dad, when I was in elementary school, started a small commercial janitorial business that he runs to this day, even though he’s mostly retired. He always told me how it was helpful to have an unsexy but useful skill set, and basically do the work that nobody else wanted to do to ensure a long career.
- papeda 7y ago
  Curious, how does a small commercial janitorial business survive for so long? My naive understanding as a non-business-owner is that small entities can survive or even do well if they offer a product that's hard or not worth it for big entities to replicate at scale. Some examples might be art, or a nice feeling of "buying local" from people you know and like, or a niche product whose market is just not big enough for giants to bother with. A janitorial business doesn't seem to be any of those things. So it seems like an area where a big company would dominate (because, say, a lot of 1000 janitorial carts is way cheaper than a lot of 10). But, no?
  - 1123581321 7y ago
    I don’t know why^, but the largest janitorial/cleaning companies that serve ordinary businesses^^ are franchise companies. Local independents just have to compete against branded local companies. ^I suspect it’s because janitors have access so trust of the individuals involved matters more than cost or having access to a pool of interchangeable employees. ^^There are large contractors that serve really large facilities, but that business isn’t necessary to win to be viable.
  - lumpsti 7y ago
    I guess like with many job. You can put a little heart into it and care about the customers. This is something that is hard to replicate in scale for big cleaning company. Not many cleaning companies do that. And those who do are loved deeply by their customers.
  - ridaj 7y ago
    There aren't very meaningful benefits to scale in that business. It's labor intensive, and no critical central infrastructure is needed. Capital costs (those janitorial carts) and supplies are a lot smaller than labor costs.
amelius 7y ago
How hard would it be to automatically convert that COBOL to e.g. C++ or Rust? I suspect not that hard, since COBOL does not even have a heap afaik.
- AndrewDucker 7y ago
  And then you have a bunch of code that's even harder to understand than the original, working in ways you don't expect from the language it's now in. And not working with the normal build or debug tools on the mainframe it runs on.
- bpicolo 7y ago
  Presumably, hard: https://aws.amazon.com/blogs/apn/automated-refactoring-of-a-u-s-department-of-defense-mainframe-to-aws/ https://aws.amazon.com/blogs/apn/automated-refactoring-of-a-...
- downerending 7y ago
  A high-fidelity translation between almost any two serious programming languages sound like a very difficult task. The first 95% might indeed be quite easy, but the last 0.1% would often turn out to be extremely difficult. And if the software "matters", you have to achieve 100%.
- ape4 7y ago
  Lots of things don't have a 1:1 match. For example Cobol has a PICTURE clause for describing a record. It uses 1 character symbols like in fscanf() but there are different options. Like binary coded decimal.
- retrac 7y ago
  It can be automatically translated. But then all you've done is convert a bunch of IF A < B THEN GO TO STEPTWO into if (a < b) { goto steptwo; } And with a lot of extra unreadable stuff to support things COBOL does but the target language doesn't. The biggest problem with COBOL codebases is that COBOL (at least back in the day) doesn't offer much in the way of structured programming or standard libraries. Horrifying spaghetti code with one-off local solutions for common tasks is the norm, especially on some ancient endlessly patched code.
Blackthorn 7y ago
> Universities stopped training because it was not—glamorous. It also doesn't pay very well.
- Bnshsysjab 7y ago
  If it’s anything like legacy public service code in my country it’s not just that, it’s also dealing with terrible organisational culture, d-grade devs, c-grade contractors and corrupt management that let it get that way (as millions are dumped on the contractors, as it turns out). Some days I wish an ultra virus would take out computers as we know it so we’d be forced to start again with what we know now, namely not creating these monolithic nasty monsters.
  - chx 7y ago
    Yes and no remote work as far as I am aware of these things. It just sucks. If it wouldn't suck, I would've went into it: I like retro and COBOL is not a hard language to learn, really, it is not.
  - grogenaut 7y ago
    Cool we'd have a bunch of new monsters based on a super old version of nodejs instead.
    - Bnshsysjab 7y ago
      I’m not saying you can’t write a monster in node (nor would I recommend writing complex apps in it) but modern systems are somewhat less monolithic, if a component is garbage rebuild it, run unit/integration tests and push to business to verify. You can’t do that when your application is such a black box even the devs don’t truly know how it works, which often seems to be the case with legacy cobol.
      - grogenaut 7y ago
        Cool you missed "actually understand the legacy system" and went right to the grand rewrite into microservices. If you can't be bothered to understand one service from 30 years ago, how is someone 30 years from now going to understand your 45 service masterpiece let alone run the out of date kube cluster you deployed it to.
27182818284 7y ago
In the YouTube clip, the URL http://covid19.nj.gov/ http://covid19.nj.gov/ but there is still as of yet no mention of COBOL as far as I can see. If anyone has the actual links that include the job description, etc, I'd be curious.
- macintux 7y ago
  I don’t have a direct link for you, but this topic has popped up several times in the past few days and you might find something more substantive. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=cobol&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- blasket 7y ago
  8 seconds in he says "we need Cobalt programmers" and repeats at 27 seconds https://www.youtube.com/watch?v=HSVgHlSTPYQ https://www.youtube.com/watch?v=HSVgHlSTPYQ
- blasket 7y ago
  https://www.youtube.com/watch?v=HSVgHlSTPYQ https://www.youtube.com/watch?v=HSVgHlSTPYQ 8 seconds in he mentionds "Cobalt" & also at 27s
Waterluvian 7y ago
Generally speaking how long would it take to mobilize any competent senior software developer as a COBOL developer? And could you imagine the DPA being invoked to govern what kind of _software_ output a company produces? I know I'm being far fetched but I'm thinking about the hypothetical of the government just saying, "Apple, you're doing government payment software now. Here's your contact. Get to work."
- gregjor 7y ago
  I learned COBOL from the (then) standard text by McCracken back in the early 1980s. It’s not hard, a week or two for someone with programming experience. The bigger problem is domain expertise (or lack of), and coming up to speed on the standards and idioms used in projects like this. Requirements and documentation likely out of date.
ellius 7y ago
I'm a younger engineer, so I say this with a degree of humility, but is trying to modify these monstrous legacy systems on the fly during a worldwide catastrophe really the most responsible or intelligent solution? Is there no way to authorize some kind of emergency system built with modern tooling to offload the system burden and then process it later through the system of record? If I were trying to deal with this problem, my approach would be the following: - Organize the legislature(s) to pass some kind of emergency act that gives protection around PCI compliance and that sort of thing to new engineering teams - Hire business and technical experts, fast - Build a system on AWS that can handle massive scale and start offloading the most important business functionality to that system (things like registering for aid, fraud prevention) - when things begin to return to normal, start processing requests through the original system, deal with problems, whatever I know all of this is not easy. Legal and compliance stuff is tough, and I know the domain itself must not exactly be simple. But this is an emergency, and it calls for emergency measures. Trying to hack your way through an ancient code base on the fly seems like it might be doomed to failure.
- gregjor 7y ago
  I think you underestimate the scope and scale of systems like this. And maybe overestimate the success rate of rewrites, made worse when the developers have no domain experience and a rushed schedule. Under the best conditions rewrites fail more often than not. Maintaining legacy code isn’t always easy, but it has a big advantage over a rewrite: it works and satisfies requirements to some non-zero percentage, whereas imaginary code built with “modern tooling” does not work or meet any requirements. COBOL and old business systems aren’t that hard to understand.
  - chberry 7y ago
    I completely agree. Rewriting a system on the fly is absolutely going to fail. There are no requirements and without the domain knowledge, there is no way that anything could be completed correctly in any reasonable time. This is a project that should have been scheduled years ago. Some level of technical debt will exist in every organization. The problem is that NJ never had prioritized this work and now they have an emergency and they are looking for volunteers. They should absolutely should be paying for this work.
blululu 7y ago
>>needs 100's of programmers - TODAY If you haven't done it already, please read Fred Brook's timeless classic 'The Mythic Man-Month' (1975) https://en.wikipedia.org/wiki/The_Mythical_Man-Month https://en.wikipedia.org/wiki/The_Mythical_Man-Month Nine women can't make a baby in one month - classic wisdom that should be understood by anyone trying to organize and plan (like a governor).

59 points by saadalem 7y ago

COBOL is operating for a number of practical reasons in the US. Universities stopped training because it was not—glamorous. Now NJ needs 100s of programmers—TODAY.

Tepix 7y ago

URL? What does he need them for?

okareaman 7y ago

link: https://www.youtube.com/watch?v=nlJqlWSPJeI https://www.youtube.com/watch?v=nlJqlWSPJeI

choward 7y ago
Cobalt? Haha. I don't think anyone on earth has experience in that. It better pay well.
- xxpor 7y ago
  Some child miners in DRC might disagree :(

steven77723 7y ago

Discussed at: https://news.ycombinator.com/item?id=22782097 https://news.ycombinator.com/item?id=22782097

tomohawk 7y ago

Must not be willing to pay enough or be hard to work with. At the right price, 100 people should not be hard to find. Of course, they probably would not be in this position if they were willing to put money into the system or keep people on retainer at least.

dreary_dugong 7y ago
I recommend taking a look at the other discussions linked. Unfortunately the situation is more nuanced then "just pay them more."
- renewiltord 7y ago
  I tried reading the other threads but they are not very information-dense. It's mostly people arguing about whether governments are good employers. Would you mind sharing a quick summary if you've synthesized the position?

sharkweek 7y ago

My dad, when I was in elementary school, started a small commercial janitorial business that he runs to this day, even though he’s mostly retired. He always told me how it was helpful to have an unsexy but useful skill set, and basically do the work that nobody else wanted to do to ensure a long career.

papeda 7y ago
Curious, how does a small commercial janitorial business survive for so long? My naive understanding as a non-business-owner is that small entities can survive or even do well if they offer a product that's hard or not worth it for big entities to replicate at scale. Some examples might be art, or a nice feeling of "buying local" from people you know and like, or a niche product whose market is just not big enough for giants to bother with. A janitorial business doesn't seem to be any of those things. So it seems like an area where a big company would dominate (because, say, a lot of 1000 janitorial carts is way cheaper than a lot of 10). But, no?
- 1123581321 7y ago
  I don’t know why^, but the largest janitorial/cleaning companies that serve ordinary businesses^^ are franchise companies. Local independents just have to compete against branded local companies. ^I suspect it’s because janitors have access so trust of the individuals involved matters more than cost or having access to a pool of interchangeable employees. ^^There are large contractors that serve really large facilities, but that business isn’t necessary to win to be viable.
- lumpsti 7y ago
  I guess like with many job. You can put a little heart into it and care about the customers. This is something that is hard to replicate in scale for big cleaning company. Not many cleaning companies do that. And those who do are loved deeply by their customers.
- ridaj 7y ago
  There aren't very meaningful benefits to scale in that business. It's labor intensive, and no critical central infrastructure is needed. Capital costs (those janitorial carts) and supplies are a lot smaller than labor costs.

amelius 7y ago

How hard would it be to automatically convert that COBOL to e.g. C++ or Rust? I suspect not that hard, since COBOL does not even have a heap afaik.

AndrewDucker 7y ago
And then you have a bunch of code that's even harder to understand than the original, working in ways you don't expect from the language it's now in. And not working with the normal build or debug tools on the mainframe it runs on.
bpicolo 7y ago
Presumably, hard: https://aws.amazon.com/blogs/apn/automated-refactoring-of-a-u-s-department-of-defense-mainframe-to-aws/ https://aws.amazon.com/blogs/apn/automated-refactoring-of-a-...
downerending 7y ago
A high-fidelity translation between almost any two serious programming languages sound like a very difficult task. The first 95% might indeed be quite easy, but the last 0.1% would often turn out to be extremely difficult. And if the software "matters", you have to achieve 100%.
ape4 7y ago
Lots of things don't have a 1:1 match. For example Cobol has a PICTURE clause for describing a record. It uses 1 character symbols like in fscanf() but there are different options. Like binary coded decimal.
retrac 7y ago
It can be automatically translated. But then all you've done is convert a bunch of IF A < B THEN GO TO STEPTWO into if (a < b) { goto steptwo; } And with a lot of extra unreadable stuff to support things COBOL does but the target language doesn't. The biggest problem with COBOL codebases is that COBOL (at least back in the day) doesn't offer much in the way of structured programming or standard libraries. Horrifying spaghetti code with one-off local solutions for common tasks is the norm, especially on some ancient endlessly patched code.

Blackthorn 7y ago

> Universities stopped training because it was not—glamorous. It also doesn't pay very well.

Bnshsysjab 7y ago
If it’s anything like legacy public service code in my country it’s not just that, it’s also dealing with terrible organisational culture, d-grade devs, c-grade contractors and corrupt management that let it get that way (as millions are dumped on the contractors, as it turns out). Some days I wish an ultra virus would take out computers as we know it so we’d be forced to start again with what we know now, namely not creating these monolithic nasty monsters.
- chx 7y ago
  Yes and no remote work as far as I am aware of these things. It just sucks. If it wouldn't suck, I would've went into it: I like retro and COBOL is not a hard language to learn, really, it is not.
- grogenaut 7y ago
  Cool we'd have a bunch of new monsters based on a super old version of nodejs instead.
  - Bnshsysjab 7y ago
    I’m not saying you can’t write a monster in node (nor would I recommend writing complex apps in it) but modern systems are somewhat less monolithic, if a component is garbage rebuild it, run unit/integration tests and push to business to verify. You can’t do that when your application is such a black box even the devs don’t truly know how it works, which often seems to be the case with legacy cobol.
    - grogenaut 7y ago
      Cool you missed "actually understand the legacy system" and went right to the grand rewrite into microservices. If you can't be bothered to understand one service from 30 years ago, how is someone 30 years from now going to understand your 45 service masterpiece let alone run the out of date kube cluster you deployed it to.

27182818284 7y ago

In the YouTube clip, the URL http://covid19.nj.gov/ http://covid19.nj.gov/ but there is still as of yet no mention of COBOL as far as I can see. If anyone has the actual links that include the job description, etc, I'd be curious.

macintux 7y ago
I don’t have a direct link for you, but this topic has popped up several times in the past few days and you might find something more substantive. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=cobol&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
blasket 7y ago
8 seconds in he says "we need Cobalt programmers" and repeats at 27 seconds https://www.youtube.com/watch?v=HSVgHlSTPYQ https://www.youtube.com/watch?v=HSVgHlSTPYQ
blasket 7y ago
https://www.youtube.com/watch?v=HSVgHlSTPYQ https://www.youtube.com/watch?v=HSVgHlSTPYQ 8 seconds in he mentionds "Cobalt" & also at 27s

Waterluvian 7y ago

Generally speaking how long would it take to mobilize any competent senior software developer as a COBOL developer? And could you imagine the DPA being invoked to govern what kind of _software_ output a company produces? I know I'm being far fetched but I'm thinking about the hypothetical of the government just saying, "Apple, you're doing government payment software now. Here's your contact. Get to work."

gregjor 7y ago
I learned COBOL from the (then) standard text by McCracken back in the early 1980s. It’s not hard, a week or two for someone with programming experience. The bigger problem is domain expertise (or lack of), and coming up to speed on the standards and idioms used in projects like this. Requirements and documentation likely out of date.

ellius 7y ago

I'm a younger engineer, so I say this with a degree of humility, but is trying to modify these monstrous legacy systems on the fly during a worldwide catastrophe really the most responsible or intelligent solution? Is there no way to authorize some kind of emergency system built with modern tooling to offload the system burden and then process it later through the system of record? If I were trying to deal with this problem, my approach would be the following: - Organize the legislature(s) to pass some kind of emergency act that gives protection around PCI compliance and that sort of thing to new engineering teams - Hire business and technical experts, fast - Build a system on AWS that can handle massive scale and start offloading the most important business functionality to that system (things like registering for aid, fraud prevention) - when things begin to return to normal, start processing requests through the original system, deal with problems, whatever I know all of this is not easy. Legal and compliance stuff is tough, and I know the domain itself must not exactly be simple. But this is an emergency, and it calls for emergency measures. Trying to hack your way through an ancient code base on the fly seems like it might be doomed to failure.

gregjor 7y ago
I think you underestimate the scope and scale of systems like this. And maybe overestimate the success rate of rewrites, made worse when the developers have no domain experience and a rushed schedule. Under the best conditions rewrites fail more often than not. Maintaining legacy code isn’t always easy, but it has a big advantage over a rewrite: it works and satisfies requirements to some non-zero percentage, whereas imaginary code built with “modern tooling” does not work or meet any requirements. COBOL and old business systems aren’t that hard to understand.
- chberry 7y ago
  I completely agree. Rewriting a system on the fly is absolutely going to fail. There are no requirements and without the domain knowledge, there is no way that anything could be completed correctly in any reasonable time. This is a project that should have been scheduled years ago. Some level of technical debt will exist in every organization. The problem is that NJ never had prioritized this work and now they have an emergency and they are looking for volunteers. They should absolutely should be paying for this work.

blululu 7y ago

>>needs 100's of programmers - TODAY If you haven't done it already, please read Fred Brook's timeless classic 'The Mythic Man-Month' (1975) https://en.wikipedia.org/wiki/The_Mythical_Man-Month https://en.wikipedia.org/wiki/The_Mythical_Man-Month Nine women can't make a baby in one month - classic wisdom that should be understood by anyone trying to organize and plan (like a governor).