14 ms·

Been working on accounting systems in RPG and COBOL since ~1992. I also know C/X86ASM/Pascal/Delphi/VB/Fortran. Never bothered with C++ that much; played with

by sanguy 10y ago

Been working on accounting systems in RPG and COBOL since ~1992. I also know C/X86ASM/Pascal/Delphi/VB/Fortran. Never bothered with C++ that much; played with Java a bit but Oracle irritates my bowels so moved away from that. As mentioned in the article it's good work; but it is also not easy work. You tend to go through cycles of being pushed out to brought back under extreme emergency at any costs to get stuff working. Only for the cycle to repeat. Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" - I can't even keep track of how many "rewrites" I've seen in my life fail because of this. We are the dinosaur club; but it's a club that pays extremely well (high 6 figures a year without working too hard if you are talented and have a good client base and reputation), but like fossil fuel one day it will all be gone ;)

movedx 10y ago
I smiled at your last comment. You're clearly a charming individual :-) Out of interest, what's the solution here? With your experiences in mind, what are the banks doing wrong from a technological perspective? I get the impression banks should almost start again on the side, building a completely new bank they then (manually?) move customers over to, or just use to take in new customers whilst waiting for the previous ones to die out. I guess if they built a whole new system, wouldn't it then be a "simple" case of doing a money transfer into the new system?
- UweSchmidt 10y ago
  You can't wait for customers to "die out" as their mortgages will go on for decades. You definitely need a customer's old data in the new system. Now you have to maintain and develop two systems - bugs, banking regulation require constant work. Often the clock is ticking for old systems and asking vendors to provide and maintain ancient tools can be expensive or impossible. You can't "manually" move customers over, you need to "migrate" lots and lots of data - both systems will have huge, completely different database schemas.
  - joncrocks 10y ago
    While mortgages do go on for decades (typically), people will frequently re-mortgage based on new interest rates. e.g. in the UK it's typical to sign up for 2-10 year deals, with the interest increasing after these offer periods, meaning people will likely switch to another deal. So there will be a certain amount or organic churn.
    - twic 10y ago
      Not enough churn, though. I worked on mortgages for a UK high street bank a while ago, and they were still running the mortgage systems for every bank and building society they'd taken over in their decades of growth, plus a few different attempts at building one system to rule them all.
  - sedachv 10y ago
    For an example of how this system switchover looks like, First Bank in the United States just did a whole-system switchover in March, at least for their business customers. They handle my wife's business' banking, which means I do all the paperwork, and it definitely was a pain. Basically it was like signing up for a whole new bank account. They copied over something like only the last 6 months of transactions and some payroll info, but things like automatic bill payments and everything else (even basic online login) had to be set up again.
- tajen 10y ago
  French banks are actually doing that! SG/Société Générale, a brick and mortar bank that costs a hundred dollars a year, built Boursorama, an e-bank that's free and that gives you $200 if you subscribe. Advantage: SG has too many manned agencies and can't justify preemptively mass-firing them yet – they first need the customer base to dwindle. Better IT, usage of Internet techniques at the core, and drastically simplifying the product line is an intended consequence. Same happens with other French banks. As a personal PoV, banks' manned agencies provide bad service (from 20x delays to plain mistakes), so I won't cry for them.
  - user5994461 10y ago
    Boursorama is not a bank. It's a website to allow people to play on the financial markets ("bourse" = exchange). I think Americans would call that a sort of dealing account. Obviously, a dealing account has to hold funds and handle transfers. That's not any close to a consumer bank though.
    - tajen 10y ago
      Boursorama Banque is a bank, as in, a normal cheque account, IBAN, VISA card, loans, term deposits and no trading account. Real normal bank. The main Boursorama website is another department, it's not really clear on their website.
    - gbersac 10y ago
      It is now also a bank. I own one Boursorama bank and I have account credit card...
    - diegoprzl 10y ago
      Boursorama also owns an Spanish "digital bank" called Self Bank. I have an account there.
Joeri 10y ago
> Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" Exactly this is why rewrites fail. The challenge of a rewrite is not in mapping the core architecture and core use case, it's mapping all the edge cases and covering all the end user needs. You need people intimately familiar with the old system to make sure the new system does all the weird stuff which nobody understood but had good reasons that was in the corners of the old system's code. IMHO the best way to approach a rewrite is to make a blended team of experts of the old system and experts of the new technology, and you put a manager in charge with excellent people skills who can get them to work together.
- Bombthecat 10y ago
  What a ton of people forget : you also need to implement bugs or edge cases the same as in the old software. More often than not the next developer using the old system worked around it. But never documented it as a bug (or wrong format, it featur! Not a bug!)
- jerguismi 10y ago
  What about not doing a rewrite? What about instead refactoring, documenting etc the old system.
  - kpil 10y ago
    Because COBOL is the PHP of the 60s, and mainframes are slow and expensive. Also, too many of the talents are stuck in a blocked I/O mindset somehow. Some are wizards though, writing assembly and making raspberry pi sized systems blazingly fast. OK a couple of raspberry pi:s
    - tyingq 10y ago
      As a counterpoint, many very well funded and talent rich organizations have failed to retire what TPF mainframes do every day for airlines, banks, and credit card companies. Cobol isn't involved, but those slow mainframes are.
acchow 10y ago
> high 6 figures a year without working too hard Wow, like $800-900k?
- Johnny_Brahms 10y ago
  It's not unheard of. But you must understand that these are people that have a pretty decent job security in an often huge codebase that has been aquiring cruft for 30 years in language most people don't understand. I would like to be able to claim that the code is good, but some of the people writing it in the first place were not programmers. It is not strictly bad, but some parts are just mine fields. A bit related: I spent 2 years of my life on a C codebase written by mathematicians that would much rather have written it in APL. I don't know if you have seen C written as APL, but that is 2 years of my life I will never get back. I left for a lesser paid, more fun job. Upon leaving the manager offered me a 60% pay raise (yup, I should have had higher demands, but at least I proved myself :) ). Unless you enjoy torturing yourself, legacy COBOL programming is not very rewaeding, and at least in Sweden, most of it is slowly being moved to other languages.
  - UweSchmidt 10y ago
    Best story I've heard is one of the old programmers commenting his COBOL code in latin. It was a different time...
    - endgame 10y ago
      Malus est?
    - H4CK3RM4N 10y ago
      I feel like that's a play for job security more than anything else.
      - eru 10y ago
        And different times: these days with Google Translate Latin doesn't add much to your job security.
gambler 10y ago
> Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" So true, even for more recent stuff. It's so absurd I always wanted to make a web comic about this. Companies keep ignoring advice of their own developers, and then eventually hire some "technology expert" who is going to implement the same tech their existing staff recommend years ago. Except, of course, the expert has no idea about business processes and user needs, so you end up with a long and expensive train-wreck that results in something barely better than what you had before.
- collyw 10y ago
  Our company does the opposite. The tech staff recommend against buying bloated enterprise software. They buy it anyway and tell us to make it work. It takes 5 times longer then building something simple and custom and is generally not fun to work with.
jmknoll 10y ago
As a developer who recently turned thirty, I think about this a lot. I like working in startups, but its not going to fly forever. I'm already the old guy in the room. Would learning 'ancient' technology be a good career move? These systems aren't going anywhere, right? But the people who know how to maintain them are. Which means that maintaining these system, which already pays well, will pay even better in the future? Or is there something I'm missing here.
- z3t4 10y ago
  it the cost becomes too high or rather if there are uncertainties they will just replace them.
- twic 10y ago
  Just hang around - the technologies you learnt when you were 20 will be ancient soon enough.
  - u02sgb 10y ago
    Yeah, but my impression is they're so ephemeral they might not have a big market share.
    - pc86 10y ago
      Wait you mean Rust and Go aren't going to be around 30 years from now?!
      - beat 10y ago
        Go probably will be, but it's still a niche. Rust, the jury is still hearing evidence.
- throwaway7645 10y ago
  You're missing the part where they're being aggressively replaced by Java at every turn and will eventually cease to exist and also these jobs are temporary consulting gigs more popular with retired experts going back to the same bank that fired them years previously. If you want to be well employed, learn Java and move to the American south where a nice house with a 1/2 acre backyard sets you back less $130k - $200k and they can't find enough qualified IT people. No it isn't a startup or Google, but the pay compared to the average income of $30k lets you live far better than San Francisco as long as you don't mind less things to do in town.
Shivetya 10y ago
We have most platforms from IBM, z (Mainframe), iSeries (AS/400), p (AIX), plus about everything windows. Moving off core systems (z/i/p) isn't simple mostly because of the amount of data combined with all the custom applications. every attempt to move the apps off the z don't come to fruition because of scope and the fact it just works. the i is increasing its load to pick up from the z and getting a good dose of webfacing and rest services. these are dead simple to implement and many reuse all the existing code. talent wise, it really hasn't been much of a challenge in Atlanta/Dallas to find those who can support what is needed. the language isn't the biggest road block for many mainframe systems, its the file systems that can trip people. the i is pure DB2 so anyone versed in SQL can use it and RPG looks more like Pascal these days than that three column stuff people normally associate with it. the one thing many here don't understand is just how many companies are invested in mainframe, i, and large p series systems. don't scoff at these platforms. with modern tools they can be webfaced just fine and the advantage their default coding languages have is they are business math oriented and simple. with modern features and file systems it just comes down to, what is management comfortable with and does it serve the companies needs.
- DoofusOfDeath 10y ago
  > Moving off core systems (z/i/p) isn't simple mostly because of the amount of data combined with all the custom applications. I wonder if someone could get away with selling a z/i/p emulator, akin to Wine. (Or perhaps more appropos, Mame, since the machine architecture would differ as well.)
  - eru 10y ago
    Aren't the modern mainframes something like emulators / hypervisors of the old ones already? (IBM still sells you new mainframes.)
  - msh 10y ago
    IBM would never license you the OS.
  - forgottenpass 10y ago
throwaway_374 10y ago
I think the question everyone is failing to ask is: what's the interview process like? Do they put you through 4 rounds of whiteboard coding bullshit followed by a coding assessment followed by a pair programming session? If anything, I'd argue the interview process for such a nice role should be the most rigorous because you can't rely on auto-complete-by-stackoverflow.
- SmellTheGlove 10y ago
  Doubtful. I'd bet a lot of the people who can do this work wouldn't be interested in a whiteboard interview. If you're fishing in a small pond for people that can maintain your legacy code, there's going to be a lot less bullshit in the process. Whiteboard interviews have their place, I'm sure. This isn't one of them.
  - ExactoKnight 10y ago
    I'm pretty sure with these kinds of systems, if you can repeat about 10 nouns relevant to working within a mainframe environment like this, you're hired.
    - SmellTheGlove 10y ago
      I've considered it, seriously even. I could go off, contract for large hourlies, probably get paid a lot more than I do now as a more junior exec with a lot less stress (and when I first moved into management, I had the same thoughts). If you have the skills and reputation, this is an area where you write your own ticket. If I were closer to retirement age instead of in my mid-30's, I think I'd be doing it.
caseymarquis 10y ago
Sounds like you should start an "enterprise expert" consultancy that knows what it's doing. Seems like a good opportunity.
- bernardlunn 10y ago
  great idea. Or an expert network as many are self employed.
- ExactoKnight 10y ago
  Go for it, if you like death marches that will just fail. The real trick to software consultancy nirvana is to find the big whales paying big money for what are just kiddie apps.
eru 10y ago
I guess software is so ingrained into companies these days---and banks were particularly early adopters, that the best chance of a rewrite of a company's software is to start a new company. A new company whose processes for example don't have all these interdependent edge cases (yet). The domain will start pushing edge cases on the new company too sooner or later, but that knowledge still has a chance to diffuse---and lots of newer companies these days are probably better at avoiding small bus factors.
bluedino 10y ago
>> Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" Too often the old guys simply re-create the old system using the old methods in new software, and you end up with all the same problems as before. In a perfect world, the two would work together.

by sanguy 10y ago

movedx 10y ago

I smiled at your last comment. You're clearly a charming individual :-) Out of interest, what's the solution here? With your experiences in mind, what are the banks doing wrong from a technological perspective? I get the impression banks should almost start again on the side, building a completely new bank they then (manually?) move customers over to, or just use to take in new customers whilst waiting for the previous ones to die out. I guess if they built a whole new system, wouldn't it then be a "simple" case of doing a money transfer into the new system?

UweSchmidt 10y ago
You can't wait for customers to "die out" as their mortgages will go on for decades. You definitely need a customer's old data in the new system. Now you have to maintain and develop two systems - bugs, banking regulation require constant work. Often the clock is ticking for old systems and asking vendors to provide and maintain ancient tools can be expensive or impossible. You can't "manually" move customers over, you need to "migrate" lots and lots of data - both systems will have huge, completely different database schemas.
- joncrocks 10y ago
  While mortgages do go on for decades (typically), people will frequently re-mortgage based on new interest rates. e.g. in the UK it's typical to sign up for 2-10 year deals, with the interest increasing after these offer periods, meaning people will likely switch to another deal. So there will be a certain amount or organic churn.
  - twic 10y ago
    Not enough churn, though. I worked on mortgages for a UK high street bank a while ago, and they were still running the mortgage systems for every bank and building society they'd taken over in their decades of growth, plus a few different attempts at building one system to rule them all.
- sedachv 10y ago
  For an example of how this system switchover looks like, First Bank in the United States just did a whole-system switchover in March, at least for their business customers. They handle my wife's business' banking, which means I do all the paperwork, and it definitely was a pain. Basically it was like signing up for a whole new bank account. They copied over something like only the last 6 months of transactions and some payroll info, but things like automatic bill payments and everything else (even basic online login) had to be set up again.
tajen 10y ago
French banks are actually doing that! SG/Société Générale, a brick and mortar bank that costs a hundred dollars a year, built Boursorama, an e-bank that's free and that gives you $200 if you subscribe. Advantage: SG has too many manned agencies and can't justify preemptively mass-firing them yet – they first need the customer base to dwindle. Better IT, usage of Internet techniques at the core, and drastically simplifying the product line is an intended consequence. Same happens with other French banks. As a personal PoV, banks' manned agencies provide bad service (from 20x delays to plain mistakes), so I won't cry for them.
- user5994461 10y ago
  Boursorama is not a bank. It's a website to allow people to play on the financial markets ("bourse" = exchange). I think Americans would call that a sort of dealing account. Obviously, a dealing account has to hold funds and handle transfers. That's not any close to a consumer bank though.
  - tajen 10y ago
    Boursorama Banque is a bank, as in, a normal cheque account, IBAN, VISA card, loans, term deposits and no trading account. Real normal bank. The main Boursorama website is another department, it's not really clear on their website.
  - gbersac 10y ago
    It is now also a bank. I own one Boursorama bank and I have account credit card...
  - diegoprzl 10y ago
    Boursorama also owns an Spanish "digital bank" called Self Bank. I have an account there.

Joeri 10y ago

> Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" Exactly this is why rewrites fail. The challenge of a rewrite is not in mapping the core architecture and core use case, it's mapping all the edge cases and covering all the end user needs. You need people intimately familiar with the old system to make sure the new system does all the weird stuff which nobody understood but had good reasons that was in the corners of the old system's code. IMHO the best way to approach a rewrite is to make a blended team of experts of the old system and experts of the new technology, and you put a manager in charge with excellent people skills who can get them to work together.

Bombthecat 10y ago
What a ton of people forget : you also need to implement bugs or edge cases the same as in the old software. More often than not the next developer using the old system worked around it. But never documented it as a bug (or wrong format, it featur! Not a bug!)
jerguismi 10y ago
What about not doing a rewrite? What about instead refactoring, documenting etc the old system.
- kpil 10y ago
  Because COBOL is the PHP of the 60s, and mainframes are slow and expensive. Also, too many of the talents are stuck in a blocked I/O mindset somehow. Some are wizards though, writing assembly and making raspberry pi sized systems blazingly fast. OK a couple of raspberry pi:s
  - tyingq 10y ago
    As a counterpoint, many very well funded and talent rich organizations have failed to retire what TPF mainframes do every day for airlines, banks, and credit card companies. Cobol isn't involved, but those slow mainframes are.

acchow 10y ago

> high 6 figures a year without working too hard Wow, like $800-900k?

Johnny_Brahms 10y ago
It's not unheard of. But you must understand that these are people that have a pretty decent job security in an often huge codebase that has been aquiring cruft for 30 years in language most people don't understand. I would like to be able to claim that the code is good, but some of the people writing it in the first place were not programmers. It is not strictly bad, but some parts are just mine fields. A bit related: I spent 2 years of my life on a C codebase written by mathematicians that would much rather have written it in APL. I don't know if you have seen C written as APL, but that is 2 years of my life I will never get back. I left for a lesser paid, more fun job. Upon leaving the manager offered me a 60% pay raise (yup, I should have had higher demands, but at least I proved myself :) ). Unless you enjoy torturing yourself, legacy COBOL programming is not very rewaeding, and at least in Sweden, most of it is slowly being moved to other languages.
- UweSchmidt 10y ago
  Best story I've heard is one of the old programmers commenting his COBOL code in latin. It was a different time...
  - endgame 10y ago
    Malus est?
  - H4CK3RM4N 10y ago
    I feel like that's a play for job security more than anything else.
    - eru 10y ago
      And different times: these days with Google Translate Latin doesn't add much to your job security.

gambler 10y ago

> Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" So true, even for more recent stuff. It's so absurd I always wanted to make a web comic about this. Companies keep ignoring advice of their own developers, and then eventually hire some "technology expert" who is going to implement the same tech their existing staff recommend years ago. Except, of course, the expert has no idea about business processes and user needs, so you end up with a long and expensive train-wreck that results in something barely better than what you had before.

collyw 10y ago
Our company does the opposite. The tech staff recommend against buying bloated enterprise software. They buy it anyway and tell us to make it work. It takes 5 times longer then building something simple and custom and is generally not fun to work with.

jmknoll 10y ago

As a developer who recently turned thirty, I think about this a lot. I like working in startups, but its not going to fly forever. I'm already the old guy in the room. Would learning 'ancient' technology be a good career move? These systems aren't going anywhere, right? But the people who know how to maintain them are. Which means that maintaining these system, which already pays well, will pay even better in the future? Or is there something I'm missing here.

z3t4 10y ago
it the cost becomes too high or rather if there are uncertainties they will just replace them.
twic 10y ago
Just hang around - the technologies you learnt when you were 20 will be ancient soon enough.
- u02sgb 10y ago
  Yeah, but my impression is they're so ephemeral they might not have a big market share.
  - pc86 10y ago
    Wait you mean Rust and Go aren't going to be around 30 years from now?!
    - beat 10y ago
      Go probably will be, but it's still a niche. Rust, the jury is still hearing evidence.
throwaway7645 10y ago
You're missing the part where they're being aggressively replaced by Java at every turn and will eventually cease to exist and also these jobs are temporary consulting gigs more popular with retired experts going back to the same bank that fired them years previously. If you want to be well employed, learn Java and move to the American south where a nice house with a 1/2 acre backyard sets you back less $130k - $200k and they can't find enough qualified IT people. No it isn't a startup or Google, but the pay compared to the average income of $30k lets you live far better than San Francisco as long as you don't mind less things to do in town.

Shivetya 10y ago

We have most platforms from IBM, z (Mainframe), iSeries (AS/400), p (AIX), plus about everything windows. Moving off core systems (z/i/p) isn't simple mostly because of the amount of data combined with all the custom applications. every attempt to move the apps off the z don't come to fruition because of scope and the fact it just works. the i is increasing its load to pick up from the z and getting a good dose of webfacing and rest services. these are dead simple to implement and many reuse all the existing code. talent wise, it really hasn't been much of a challenge in Atlanta/Dallas to find those who can support what is needed. the language isn't the biggest road block for many mainframe systems, its the file systems that can trip people. the i is pure DB2 so anyone versed in SQL can use it and RPG looks more like Pascal these days than that three column stuff people normally associate with it. the one thing many here don't understand is just how many companies are invested in mainframe, i, and large p series systems. don't scoff at these platforms. with modern tools they can be webfaced just fine and the advantage their default coding languages have is they are business math oriented and simple. with modern features and file systems it just comes down to, what is management comfortable with and does it serve the companies needs.

DoofusOfDeath 10y ago
> Moving off core systems (z/i/p) isn't simple mostly because of the amount of data combined with all the custom applications. I wonder if someone could get away with selling a z/i/p emulator, akin to Wine. (Or perhaps more appropos, Mame, since the machine architecture would differ as well.)
- eru 10y ago
  Aren't the modern mainframes something like emulators / hypervisors of the old ones already? (IBM still sells you new mainframes.)
- msh 10y ago
  IBM would never license you the OS.
- forgottenpass 10y ago

throwaway_374 10y ago

I think the question everyone is failing to ask is: what's the interview process like? Do they put you through 4 rounds of whiteboard coding bullshit followed by a coding assessment followed by a pair programming session? If anything, I'd argue the interview process for such a nice role should be the most rigorous because you can't rely on auto-complete-by-stackoverflow.

SmellTheGlove 10y ago
Doubtful. I'd bet a lot of the people who can do this work wouldn't be interested in a whiteboard interview. If you're fishing in a small pond for people that can maintain your legacy code, there's going to be a lot less bullshit in the process. Whiteboard interviews have their place, I'm sure. This isn't one of them.
- ExactoKnight 10y ago
  I'm pretty sure with these kinds of systems, if you can repeat about 10 nouns relevant to working within a mainframe environment like this, you're hired.
  - SmellTheGlove 10y ago
    I've considered it, seriously even. I could go off, contract for large hourlies, probably get paid a lot more than I do now as a more junior exec with a lot less stress (and when I first moved into management, I had the same thoughts). If you have the skills and reputation, this is an area where you write your own ticket. If I were closer to retirement age instead of in my mid-30's, I think I'd be doing it.

caseymarquis 10y ago

Sounds like you should start an "enterprise expert" consultancy that knows what it's doing. Seems like a good opportunity.

bernardlunn 10y ago
great idea. Or an expert network as many are self employed.
ExactoKnight 10y ago
Go for it, if you like death marches that will just fail. The real trick to software consultancy nirvana is to find the big whales paying big money for what are just kiddie apps.

eru 10y ago

I guess software is so ingrained into companies these days---and banks were particularly early adopters, that the best chance of a rewrite of a company's software is to start a new company. A new company whose processes for example don't have all these interdependent edge cases (yet). The domain will start pushing edge cases on the new company too sooner or later, but that knowledge still has a chance to diffuse---and lots of newer companies these days are probably better at avoiding small bus factors.

bluedino 10y ago

>> Companies never think of the old guys as the ones to implement the new system - that's a job for the "enterprise experts" Too often the old guys simply re-create the old system using the old methods in new software, and you end up with all the same problems as before. In a perfect world, the two would work together.