3 ms·

I briefly worked in the (Swedish) public sector, was on a team tasked to replace an old system written partially in Cobol and partially in some IBM-tech nobody

I briefly worked in the (Swedish) public sector, was on a team tasked to replace an old system written partially in Cobol and partially in some IBM-tech nobody understood anymore. The system was huge and poorly understood, and a lot of people used it to do most of their work. Extremely critical system for the very fabric of society. There had been several previous attempts at replacing it. They'd sort of fizzled out along the way, so they were stuck with several half-finished iterations of replacements that were in used in parallel with the old system. This was it though. Hundreds of people. Micro services. Modern JS frameworks. C4 architecture and hexagonal design and cloud native and every goddamn buzzword that's appeared in a programming blog the last 10 years. Just get it all in there! Everything the cool start-up kids were using needed to be apart of this. It wasn't clear exactly how it was going to better, but it sure was gonna be modern. I didn't see the project through but I could see where it was heading clear as day. It was only a year in and maintenance was already becoming a problem. Shit was falling apart. It was a nightmare to debug. Nobody knew how the heck all the services quite communicated. I'll eat my shoe if they don't spend the next 25 years fixing leaky pipes with this microservice mess bolted onto the previous iterations as an even more grotesque programming-cronenberg.

xorcist 4y ago
This is a telling example of the innovation budget idea! If you're going to replace a creaky old legacy system, never ever do it with the latest technology. Do it with tried and boring technology. Otherwise soon enough you will have two problems.
- marginalia_nu 4y ago
  I think part of the problem was they needed to pitch the project to the government to get it funded. Probably easier to sell the idea of the Homer Simpson Model S[1] than what they actually needed. That, and everyone with any sort of deciding power were technology-illiterate bureaucrats who waste no opportunity to try to make us somehow use AI to somehow enhance this system in some fashion, largely so they could announce to the world that their system use AI somehow. [1] https://imgur.com/97lhWVR https://imgur.com/97lhWVR
- hkpack 4y ago
  I think the problem is not with the latest technology, the problem is with the inappropriate one. Have they chose golang, or (God forbid) Rust, they will most likely be fine. There are choices, which are very dangerous, as they increase complexity dramatically, like micro-services, or "event-driven" architectures, or Kubernetes. Add all three and good luck lifting project off the ground.
thatwasunusual 4y ago
I had a similar experience in the Norwegian (hello, neighbour!) public sector, and it was _also_ a rewrite from a partly COBOL-based solution. The difference is that we succeeded, which is - if you're Norwegian - almost unheard of. :) The whole thing was rebuilt using new tech; Java for the most part, cloud-ish hosting (our "own" bare metal servers and OpenShift via a Norwegian company), and this stack was never a problem. In fact, not much was a problem, because: The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on. Without those tests - written by programmers who probably are long dead - the rewrite would never have succeeded, or at least not in the time frame given, and at least not on budget. Since then, whenever our consulting company is involved in anything that has to do with existing software, we ask the client about how their test coverage is; * None, but you will give us $1 million per hour? Nah. Doesn't matter. Find someone else. * Lots, and you'll pay us $1,000 per hour? Tell us more. So: create (unit) tests for _everything_. Fixed a bug? Create a test for it. Changed some behaviour? Create a test for it. Removed a feature? Create a test for it. EDIT: Final stack looked like this: * Self-ish-hosted servers (non-Norwegian cloud providers was not an alternative because of privacy issues). * Java as main programming language on the backend, but many services was written in C#. * Frontend didn't have to be fancy, so some simple JS-stuff there; think auto-complete, no fancy DOM-handling or anything like that. Application as a whole was very server-centric. From what I've heard, they are now trying to get away from Java, and move more and more over to C#. I just hope they keep the unit tests. :)
- kqr 4y ago
  This is such a good approach. A former colleague of mine built an entire scaffolding of integration tests surrounding a test-free system that needed to have a large fraction rewritten. Mocking clients, collaborators and everything. The tests were only ever used once for that rewrite (the customer terminated the contract shortly after the rewrite was complete, and the entire codebase was thrown into the rubbish bin) but I still think it was worth it. The tests didn't even run that many times because most of them passed at the first attempt but I suspect the effort of enumerating cases had a strong impact on the successful design of the rewrite. Spend the first four hours sharpening the axe, and so on.
TristanBall 4y ago
One of the things I've had my nose rubbed in several times is just how very effective some of of those "legacy" languages - like Cobol or the various 4GL's - are for writing business systems apps. It clicked for me when I started thinking of them as DSL's, where the "domain" is "business stuff" ( or equivalent, government data processing etc ) Of course they have limitations - but they're also often part of a full platform and very opinionated as to app and data model, and within those constraints programmers can be crazy productive. So many of these rewrites fail, and I think a big part of it is people don't fully realise the ramifications of the fact that they're replacing both a platform and a system built on that platform. Sure, there's probably a wizzbang new architecture, which well intentioned smart people will be able to prove improves on that platform in every way - with reams of documentation about the choices made and the way things work... but they're still choices, and sometime they'll get misunderstood, or ignored, or subverted by people who think they know better or who just need a short cut - and it's often big enough project that staff turnover is actually a significant factor too, so opinions may shift. But many of the architecture/data model decisions are just built in to those older languages. The architecture isn't subject to the same drift pressures because they're not choices, and that makes the system less complicated overall, and especially over time. Of course limitations of the various language implementations, as well as the very human distaste of working within a constrained system is probably a big part of the reason that these legacy languages effectively died. But I think we lost something valuable, without many of us ever realising.

xorcist 4y ago

This is a telling example of the innovation budget idea! If you're going to replace a creaky old legacy system, never ever do it with the latest technology. Do it with tried and boring technology. Otherwise soon enough you will have two problems.

marginalia_nu 4y ago
I think part of the problem was they needed to pitch the project to the government to get it funded. Probably easier to sell the idea of the Homer Simpson Model S[1] than what they actually needed. That, and everyone with any sort of deciding power were technology-illiterate bureaucrats who waste no opportunity to try to make us somehow use AI to somehow enhance this system in some fashion, largely so they could announce to the world that their system use AI somehow. [1] https://imgur.com/97lhWVR https://imgur.com/97lhWVR
hkpack 4y ago
I think the problem is not with the latest technology, the problem is with the inappropriate one. Have they chose golang, or (God forbid) Rust, they will most likely be fine. There are choices, which are very dangerous, as they increase complexity dramatically, like micro-services, or "event-driven" architectures, or Kubernetes. Add all three and good luck lifting project off the ground.

thatwasunusual 4y ago

I had a similar experience in the Norwegian (hello, neighbour!) public sector, and it was _also_ a rewrite from a partly COBOL-based solution. The difference is that we succeeded, which is - if you're Norwegian - almost unheard of. :) The whole thing was rebuilt using new tech; Java for the most part, cloud-ish hosting (our "own" bare metal servers and OpenShift via a Norwegian company), and this stack was never a problem. In fact, not much was a problem, because: The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on. Without those tests - written by programmers who probably are long dead - the rewrite would never have succeeded, or at least not in the time frame given, and at least not on budget. Since then, whenever our consulting company is involved in anything that has to do with existing software, we ask the client about how their test coverage is; * None, but you will give us $1 million per hour? Nah. Doesn't matter. Find someone else. * Lots, and you'll pay us $1,000 per hour? Tell us more. So: create (unit) tests for _everything_. Fixed a bug? Create a test for it. Changed some behaviour? Create a test for it. Removed a feature? Create a test for it. EDIT: Final stack looked like this: * Self-ish-hosted servers (non-Norwegian cloud providers was not an alternative because of privacy issues). * Java as main programming language on the backend, but many services was written in C#. * Frontend didn't have to be fancy, so some simple JS-stuff there; think auto-complete, no fancy DOM-handling or anything like that. Application as a whole was very server-centric. From what I've heard, they are now trying to get away from Java, and move more and more over to C#. I just hope they keep the unit tests. :)

kqr 4y ago
This is such a good approach. A former colleague of mine built an entire scaffolding of integration tests surrounding a test-free system that needed to have a large fraction rewritten. Mocking clients, collaborators and everything. The tests were only ever used once for that rewrite (the customer terminated the contract shortly after the rewrite was complete, and the entire codebase was thrown into the rubbish bin) but I still think it was worth it. The tests didn't even run that many times because most of them passed at the first attempt but I suspect the effort of enumerating cases had a strong impact on the successful design of the rewrite. Spend the first four hours sharpening the axe, and so on.

TristanBall 4y ago

One of the things I've had my nose rubbed in several times is just how very effective some of of those "legacy" languages - like Cobol or the various 4GL's - are for writing business systems apps. It clicked for me when I started thinking of them as DSL's, where the "domain" is "business stuff" ( or equivalent, government data processing etc ) Of course they have limitations - but they're also often part of a full platform and very opinionated as to app and data model, and within those constraints programmers can be crazy productive. So many of these rewrites fail, and I think a big part of it is people don't fully realise the ramifications of the fact that they're replacing both a platform and a system built on that platform. Sure, there's probably a wizzbang new architecture, which well intentioned smart people will be able to prove improves on that platform in every way - with reams of documentation about the choices made and the way things work... but they're still choices, and sometime they'll get misunderstood, or ignored, or subverted by people who think they know better or who just need a short cut - and it's often big enough project that staff turnover is actually a significant factor too, so opinions may shift. But many of the architecture/data model decisions are just built in to those older languages. The architecture isn't subject to the same drift pressures because they're not choices, and that makes the system less complicated overall, and especially over time. Of course limitations of the various language implementations, as well as the very human distaste of working within a constrained system is probably a big part of the reason that these legacy languages effectively died. But I think we lost something valuable, without many of us ever realising.