7 ms·

Don't ever work on a COBOL modernization project. It is a career killer with little to no upside. COBOL systems and the people directly using them are Chernob

by Dowwie 6y ago

Don't ever work on a COBOL modernization project. It is a career killer with little to no upside. COBOL systems and the people directly using them are Chernobyl. Both government regulators and operations personnel have made careers out of trusting black boxes that no one fully understands anymore. Regulatory specifications are ambiguous and disorganized. Imagine working with someone who knows that you are trying to shed light on the technology that they were responsible for, yet do not fully understand. How cooperative do you think these people will be with your efforts? How do you think the managers responsible for this work survive the politics of the project's inevitable failure? Blame rolls downhill. Modernization projects require strong leadership and collaboration across many teams. That's a huge problem because such an environment very likely does not exist in the world today. No one understands how the algorithms work or even how they should work! Forget how challenging it is to work with COBOL, which someone could eventually figure out. Whatever you've written has to be tested, and you don't have a trusted source of business logic from which to verify how it ought to work. You only have the black box. If you've been given this task, start interviewing at other companies. You were given a suicide mission by people who are well aware of that. My experience with this was with regulatory margin trading systems supporting a multi-billion dollar market.

qz2 6y ago
There’s a lot of that out there. I have avoided COBOL but I have seen stuff written in VB6/COM which was also in the same state. Eventually the one guy who knows anything about it leaves or dies. In the name of “progress” someone wraps it in something else and sticks a web API around it but everyone is afraid to touch that bit of VB code. Now a third party calls this and wonders why it takes 10 seconds for an HTTP API call to run and they can’t be issued in parallel or it throws a 500 error. Then the third party asks to send 20,000 of these a day and then finger pointing and stuff occurs at management level between both companies. I have been on both sides of this and it sucks. I came up with Rug Driven Development as a title. All the risks and future problems are swept under the rug quickly while maintaining a happy smile and pretending everything is just fine and dandy.
- sdoering 6y ago
  > I came up with Rug Driven Development I so need to steal this term. Not exactly for what you are describing even if I come across this on a nearly weekly basis in so many different industries. Be it automotive, be it finance, be it what ever. But - and this is were I will probably use it in the future - I come across it in agency work. You win a shiny new project proposal and need to quickly show something for your corporate "partners" to shine before their management. To make them look good for their yearly appraisals or what not. You already know that you will never fix the underlying shit you built, because your contract will be a new pitch three years down the line and it isn't clear if you will win it again. So let the next people deal with the tech debt - but guess what: They will just do the same. So debt will pile on and on. So "Rug Driven Development" will be my goto term going forward.
- puritanicdev 6y ago
  > Rug Driven Development I'll steal this :) Encountered a few of those during my career as well as if you can't see a problems they can't hurt you...
  - stjohnswarts 6y ago
    I have just straight up cleaned out my desk and turned in my notice once upon discovering something similar when delving deeply into the innards of an old embedded hardware stack.
- heresie-dabord 6y ago
  "Rug-Driven Development" is a combination Delegation of Responsibility, Explosives Engineering, and Musical Chairs. ^_^
  - qz2 6y ago
    Nailed it :)
- yourapostasy 6y ago
  Rug Driven Development causes the majority of technical debt that I see in most of my clients, and it happens at the small scale, too. Just a couple weeks ago, a junior engineer was faced with a key process that segfaulted about once a minute. No worries though, they reasoned; the process immediately respawned and because it managed a stateless procedure, nothing is lost. Except the 1 out ~30,000 times it didn't respawn. No worries though, they reasoned; they put in a request to the monitoring team to look for the absence of the process for longer than three minutes and auto-spawn it. Ta da! Problem solved! They let this go on for about six years before I arrived and saw it. It took me pointing out that there is a decent chance this is caused by the process relying upon something in the OS kernel that isn't quite aligned to the kernel documentation but is good enough, and the immediate respawn behavior might rely upon that in turn. If the kernel ever is "fixed", then this audit-compliance-related process suddenly stops and doesn't respawn, or worse, won't start at all, becoming top of mind with all the senior management. It is always cheaper to fix problems in the small before they become problems no one can ignore. The urge to sweep problems under the rug and move on is very powerful within our industry. Until you've done the same yourself and been bitten so many times, your scar tissue twitches every time you see the same pattern again. These days, I treat code problems like I treat cleaning up messes while cooking: I clean up as I go along. My scar tissue thanks me now. There is a delicate balance between addressing these problems in the small and bikeshedding, though. BTW, I performed an strace that revealed a SIGKILL just pops in without anyone or any known process issuing it. The application developers suspect something in the OS, so we're now engaged with the application support team, OS support team, and our own internal OS support team to trace it down and beat it into submission.
wdb 6y ago
£1000/day to work on a COBOL project ain't bad though
- Dowwie 6y ago
  It depends on whether you are paid regardless of the outcomes beyond your control.
  - user5994461 6y ago
    Assume standard contracting in the UK. It's always a daily rate, nothing to do with outcomes.
- stjohnswarts 6y ago
  I doubt if you're going to get that much if you haven't got a decade plus of experience with COBOL systems.
  - wdb 6y ago
    Helps when you still have your dad's notes when he wrote the system ;)
- lumost 6y ago
  It sounds like the majority of that rate is for knowing how to navigate an organization still using cobol. A fortune 50 using such a system as its backbone may be less cost sensitive than a FANG with only a small number of engineers responsible for holding the whole thing up.

by Dowwie 6y ago

qz2 6y ago

There’s a lot of that out there. I have avoided COBOL but I have seen stuff written in VB6/COM which was also in the same state. Eventually the one guy who knows anything about it leaves or dies. In the name of “progress” someone wraps it in something else and sticks a web API around it but everyone is afraid to touch that bit of VB code. Now a third party calls this and wonders why it takes 10 seconds for an HTTP API call to run and they can’t be issued in parallel or it throws a 500 error. Then the third party asks to send 20,000 of these a day and then finger pointing and stuff occurs at management level between both companies. I have been on both sides of this and it sucks. I came up with Rug Driven Development as a title. All the risks and future problems are swept under the rug quickly while maintaining a happy smile and pretending everything is just fine and dandy.

sdoering 6y ago
> I came up with Rug Driven Development I so need to steal this term. Not exactly for what you are describing even if I come across this on a nearly weekly basis in so many different industries. Be it automotive, be it finance, be it what ever. But - and this is were I will probably use it in the future - I come across it in agency work. You win a shiny new project proposal and need to quickly show something for your corporate "partners" to shine before their management. To make them look good for their yearly appraisals or what not. You already know that you will never fix the underlying shit you built, because your contract will be a new pitch three years down the line and it isn't clear if you will win it again. So let the next people deal with the tech debt - but guess what: They will just do the same. So debt will pile on and on. So "Rug Driven Development" will be my goto term going forward.
puritanicdev 6y ago
> Rug Driven Development I'll steal this :) Encountered a few of those during my career as well as if you can't see a problems they can't hurt you...
- stjohnswarts 6y ago
  I have just straight up cleaned out my desk and turned in my notice once upon discovering something similar when delving deeply into the innards of an old embedded hardware stack.
heresie-dabord 6y ago
"Rug-Driven Development" is a combination Delegation of Responsibility, Explosives Engineering, and Musical Chairs. ^_^
- qz2 6y ago
  Nailed it :)
yourapostasy 6y ago
Rug Driven Development causes the majority of technical debt that I see in most of my clients, and it happens at the small scale, too. Just a couple weeks ago, a junior engineer was faced with a key process that segfaulted about once a minute. No worries though, they reasoned; the process immediately respawned and because it managed a stateless procedure, nothing is lost. Except the 1 out ~30,000 times it didn't respawn. No worries though, they reasoned; they put in a request to the monitoring team to look for the absence of the process for longer than three minutes and auto-spawn it. Ta da! Problem solved! They let this go on for about six years before I arrived and saw it. It took me pointing out that there is a decent chance this is caused by the process relying upon something in the OS kernel that isn't quite aligned to the kernel documentation but is good enough, and the immediate respawn behavior might rely upon that in turn. If the kernel ever is "fixed", then this audit-compliance-related process suddenly stops and doesn't respawn, or worse, won't start at all, becoming top of mind with all the senior management. It is always cheaper to fix problems in the small before they become problems no one can ignore. The urge to sweep problems under the rug and move on is very powerful within our industry. Until you've done the same yourself and been bitten so many times, your scar tissue twitches every time you see the same pattern again. These days, I treat code problems like I treat cleaning up messes while cooking: I clean up as I go along. My scar tissue thanks me now. There is a delicate balance between addressing these problems in the small and bikeshedding, though. BTW, I performed an strace that revealed a SIGKILL just pops in without anyone or any known process issuing it. The application developers suspect something in the OS, so we're now engaged with the application support team, OS support team, and our own internal OS support team to trace it down and beat it into submission.

wdb 6y ago

£1000/day to work on a COBOL project ain't bad though

Dowwie 6y ago
It depends on whether you are paid regardless of the outcomes beyond your control.
- user5994461 6y ago
  Assume standard contracting in the UK. It's always a daily rate, nothing to do with outcomes.
stjohnswarts 6y ago
I doubt if you're going to get that much if you haven't got a decade plus of experience with COBOL systems.
- wdb 6y ago
  Helps when you still have your dad's notes when he wrote the system ;)
lumost 6y ago
It sounds like the majority of that rate is for knowing how to navigate an organization still using cobol. A fortune 50 using such a system as its backbone may be less cost sensitive than a FANG with only a small number of engineers responsible for holding the whole thing up.