10 ms·
The World of Legacy IT Systems
- readingnews 6y agoAs a systems admin of some 30 years, I have read this over and over, and even seen it firsthand. I think the real root cause of (at least in my case, the ones I have seen) of this problem is management. Leaders do not want to spend now. Leaders do not want to make a progressive plan to keep up with technology and move ahead. It costs money, and is very difficult to talk to upper-upper management about. Telling the boss to tell his boss to spend money on something "that works now" is very difficult. Few want to invest in the future when it costs now and is working now (unless its the stock market).
- mysterydip 6y agoI think the problem is people think of hardware (and software) as a system, rather than a system-of-systems. It works now, because the exact combination of hardware, software, and peripherals available today works. Things you need to interface with use standards that your hardware is equipped with. As soon as that changes, your system needs to upgrade or change to continue functioning. Backwards compatibility only lasts so long, and is different for each interface (hardware and software).
- jrott 6y agoThis is a huge part of the problem. The amount of risk that exists in legacy systems is actually pretty huge because of the amount of interdependencies that exist. It's the sort of thing that people who operate those systems know but that knowledge doesn't seem to spread beyond that.
- WrtCdEvrydy 6y agoTrust me, I have had to do this myself :( Needed to increase the NodeJS version from 0.16 to 4.0 because a new shiny thing needed it. I knew it was a major version change but the dependencies didn't exist for the newer node versions... so rewrite something that isn't yours and maintain it forever... or do nothing.
- tabtab 6y agoIt should be weighed like any non-trivial investment. Present an estimated yearly cost curve for keeping the system versus overhauling it: two curves. Upper managers will probably relate to that better than techno-speak. Typically a replacement system has a nearer-term spike in costs for development, but is cheaper in the longer run than keeping the old one, because the old one will need gradually more expensive duct-tape, chicken-wire, and harder-to-find technicians willing to support something obsolete. One advantage of the older system is that the costs are more predictable, because it probably works "good enough" at the time and will likely just gradually "degenerate". An overhauled system could be great or a grand failure. An exception is if a vendor stops supporting a key legacy component and there is not enough demand for 3rd-party tweakers to take up the slack. Similar can happen if a security hole is found in a key legacy component.
- WrtCdEvrydy 6y ago> Present an estimated yearly cost curve for keeping the system versus overhauling it That's the problem, most organizations can't track this. At a workplace, we replaced a legacy item with something new without doing this analysis... but after that, the move accelerated when it became aware making new features on "New vs Old" was a 2 hour vs 2 week endeavor. The new system covers 70% of the traffic and will slowly be going all the way to 100% of the traffic over the next 1-2 years.
- adrianN 6y agoCompanies can't track costs that stem from systemic inefficiencies. Say your programmers still use CVS for version control and you want to convince management to switch to Git. The CVS server doesn't cost much, a git server wouldn't cost much either. There would be a large cost for changing all your tooling and processes that rely on the CVS model and in training your programmers to use the new system. Why would you do that? How do you quantify the benefits of atomic commits or simple branching?
- nix23 6y agoThat example is too simplistic, it's called legacy systems and not legacy software. If you upgrade your tool-chain you include git. Really no one is interested in upgrading 'just' git, make a package out of it (new compiler, new CI, new testing, new version controll..fuzz etc) call the project "sustainable nextgen tool-chain...short SuNeTuCha" and sell it to the management like that.
- RickJWagner 6y agoIt's almost always cheaper (and safer) to stay with what you've got. Right up until unsupported software or hardware makes it untenable.
- NomDePlum 6y agoToo often there is a push to replace what is not actually understood in the first place.
- alacombe 6y agoWhich might be a good thing if the loss of knowledge came from employee turnover. If the architect / author leaves, the visions leaves too. Next step is monkey patching, fast forward a couple years, a decade or two, more turnover, and you end up with a indescribable mess. In this case the replacement is all about re-gaining the knowledge.
- thatguymike 6y agoI know that large business systems are complicated, and migrating old data is much much harder. But a persistent part of me thinks... is this really so hard? The IRS core system basically requires a decision tree, a calculator app and several bindersful of business logic. They need to process about 350 million people each year, which is a big but hardly unmanageable Spark job. Isn't this just a question of getting a team of good experienced Silicon Valley engineers to build something that just works? The Obamacare system is a great example of this. Almost half a billion dollars spent on corporate contractors, then a team of like 10 experienced engineers came in and fixed the damn thing.
- downvoteme1 6y agoI think this comment highlights your inexperience in enterprise it systems. There challenges facing enterprise IT are not due to there being no good engineers - because all of them are working in Silicon Valley. It is a people problem. Every department, every group, entry business unit is its own little kingdom that is fiercely defended by its workers. They cling on to it because that’s all they have done for the last 20 years. They know that if this system gets modernized, then their job is toast. Their job is vapor ware and they know it. Having worked with many state and local governments, I see that these entities are never in favor of a business process redesign that simplified things, unless the consultants carve out a special exception for their little Access database or somehow save their pre historic C# ASP.Net application.
- jacquesm 6y agoEvery year absolutely mind bogglingly large sums of money are spent on redeveloping existing applications of that level of complexity. These all have a very predictable ending: utter failure, and the legacy stuff ends up living to fight another day. It is a special kind of blindness: We think that just because our tools and machines are so much more powerful that rebuilding a complex legacy application is now childsplay. From a UI perspective you are most likely right. But it's the couple of million lines of business logic that will be your Waterloo. In a nutshell: forget about it. A bit longer: it is possible but only by a very experienced crew who has nothing else to do under competent leadership and who can chart an incremental path between where they are now and where they want to be. In 5 years or so. A big bang redevelopment is DOA. Every time.
- jacquesm 6y agoLegacy is a special form of good: It means something actually works. Good software outlives its original programmers, great software outlives its original programmers and is still maintainable. You can only get to that level if you make a commitment up front that this software is going to stand the test of time, update your systems, don't get side tracked by fluff and eye candy and concentrate on what matters: the core business logic, at the expense of almost everything else. Do that and you stand a chance of avoiding the legacy trap. A year of complacency and you have a year of catching up to do and you'll never get that year back unless you want to freeze development.
- drieddust 6y agoAbsolutely I have seen MGMT wearing "innovation" blinders only to realize nothing fucking was wrong in the first place. Before declaring something legacy, a fair evaluation of the future business requirement and fitment of current solution is a must. 12 Years back I worked with an org who had an AS400 which they considered legacy. System is still running and serves the business need perfectly well. All plans to replace just wasted money. Another company decided to implement a new centralized ERP without realizing that company's highly federated businesses which can also be considered a conglomerate of very small independent businesses doesn't need it and won't use it. Million went down the gutter before higher ups realized the sad fact and fired the CIO. IT world is literred up with sorry tales like these.
- johngalt 6y agoI've tackled legacy systems my entire career, and there is a certain art to untying the knot of dependencies, procedures, and expectations. Not surprised at all that most projects to modernize end up as smoldering craters rather than functioning systems. The legacy system trap is built on three pillars: 1. Maintenance costs are always lower than upgrade costs. It's even cheaper to pay for neither. Even if you do spend money aggressively to maintain it; every feature added, operator hired, workaround published, becomes a growing suit of armor protecting the system from replacement. 2. Management incentives are rarely aligned with the planning and sacrifices it takes to unwind technical debt. Everyone wants to cut the ribbon on the shiny new big-data/AI/ML/buzzword project, but no one wants to take responsibility for integrating it with the rusting hulks creaking along actually running day to day operations. 3. The software industry can shift paradigms enough that you end up looking to buy a new/faster horse for your buggy. And nothing fits your use case, or even offers a path out. All three in combination mean you get businesses spending obscene amounts of money on technology they can't use. It is quite satisfying to successfully maneuver an organization out of the trap. It's like solving a complex puzzle over months. Where hundreds or thousands of real people will be better off because of your team's effort.
- waheoo 6y ago> Not surprised at all that most projects to modernize end up as smoldering craters rather than functioning systems. This is because people think rewriting is a solution. It very never is. It can help, rarely, but it's never the solution, it just does a good job of hiding what the solution was. If you can't map out what is really broken in your legacy system and map out a way to refactor it to what you want then a rewrite will always fail.
- jffhn 6y ago>people think rewriting is a solution. It very never is. I often see this idea formulated, in this forum or in articles. >what is really broken in your legacy system. Software can be arbitrarily broken (think of only copy-pasting code from thedailywtf): http://www.cs.utexas.edu/users/dsb/CS373S/Images/softwaredesign.jpg http://www.cs.utexas.edu/users/dsb/CS373S/Images/softwaredes... I knew a legacy system where everything was broken. There were no proper data model and structures, so every computation involved multiple nested loops, making everything much slower than needed. Everything was modifiable as if it was public, so it was hard to keep things consistent. Allocation/GC and latency were through the roof, they tried to add pooling but noone could figure out where objects could be released. One of the core classes was called Banana (changed name) but actually referred to bananas or to banana trees. And the list goes on (threading, database usage, still plenty to tell). Then someone, new to both the domain and the language, stepped up to rewrite everything, which took a bit more than a month. Ten years later the core of it didn't change much, they are still building on it and it's still smooth.
- 7thaccount 6y agoThe article was fine until it started randomly mentioning AI.
- MattGaiser 6y agoA lot would be gained by treating software like a pet rather than as a building. Regular brushing and care, occasional picking up of errant poop, vaccines, checkups with the vet, and the knowledge that Fido will only live about 10-15 years.
- peterburkimsher 6y agoPhoto from the article, colourised: https://imgbox.com/YzqSitam https://imgbox.com/YzqSitam
- imperialdrive 6y agoMany thanks.
- BXLE_1-1-BitIs1 6y agoWell there's this fashion of dumping 50+ IT guys. The systems they worked on will then stagger on. The new guys will throw on patches, but there's no money to really understand the system. The life support just gets more and more expensive. The smart way to migrate a legacy system is to keep your maintenance crew to retirement age. Once they're approaching retirement bring in some kids to migrate to the shiny new tech and use the old crew to make sure they don't drop stuff.
- phendrenad2 6y agoI wonder if in 100 or so years software development anthropologists (do those exist yet?) will study the effects of ageism in tech using historical hiring/layoff data.
- bm98 6y agoSoftware lifetime is an interesting topic. See for example [1], a 1992 paper that surveyed business applications and found an average lifetime of about 9 years, with a maximum of 20 years and a minimum of 2 years. 2 years versus 20 years! Things don't seem much different today. On the one hand, HTTP, HTML, and producing HTML server-side in a language like Java or Python and with a SQL database isn't all that different now than it was 20 years ago. There are Java web apps developed in 1999 that are kept up-to-date and work just fine in 2020. On the other hand, there are applications developed with popular technologies a just a few years ago that are hopelessly out-of-date and in need of a rewrite today. I see it as an end user too. I've been an ING Direct (now Capital One 360) customer for 18 years. The web app that I use in 2020 seems to do the same things as the one in 2002 (though it's slower today), but it seems like it's been completely rewritten at least three times. [1] https://scholar.google.com/scholar?cluster=238269968840043751 https://scholar.google.com/scholar?cluster=23826996884004375... EDIT: And HN! 13 years old and counting...
- adrianN 6y agoI'd bet that some of the systems that were 20 years old in 1992 are still running.
- SCHiM 6y agoA thing that I found missing is that legacy systems' influence is much larger than only the systems it directly touches. Through particular requirements that some of these systems have they might shape policy too! I primarily deal in security assessments of networks. Guess what happens to your password strength requirements if somewhere deep down in a dark and old server room there is a mainframe that can only handle 8 character lowercase passwords? Assume that this system has been the backbone of your, then hyper-modern, completely integrated IAM system for decades? ... That's right, security policy mandates 'simple' 8 character passwords... Not ideal.
- tonyedgecombe 6y agoThat sounds like my last bank. I used a 16 character password but could login with the first 8 characters and it wasn't case sensitive. Every time I interacted with them there were hints that things weren't quite right and sure enough they ended up in the news after a huge IT failure.
- cthonicthulu 6y agoShudder. That's horrifying enough to make me want to wall off that whole thing in a VPN which in and of itself has a stronger password, and expose access to it from a proxy.
- peterwwillis 6y agoSomething I'm not sure is clear from the article is that not all legacy is bad. The definition of the word 'legacy', essentially, is "something that is handed down [through generations]". You can hand down good things and bad. Banks often have incredibly old systems because their products don't change. A savings account in 1940 is pretty much the same as a savings account today. They basically have no need for DevOps, because they have no need to provide fast-moving, competitive products, and silos are a natural fit in a world of sensitive financial information. So some really old systems are pretty much fine as-is, as long as the hardware can keep running, and a few over the hill programmers are on call. Another example of useful legacy is old physical infrastructure. The Frankford Avenue Bridge in Philadelphia is a 73-foot long stone bridge that's 323 years old, and carries a substantial amount of car traffic. But it's also not being maintained like it should be, and the more-than-expected wear over time is slowly eroding it. That's the bad part of legacy systems: when you know they need maintenance and they don't get it, so they rot. Modern infrastructure can suffer the same fate. Concrete bridges, wooden houses, cars... they all need maintenance. So do modern computer systems, where they need to get regular security patches. So the problem is really just a maintenance problem, and we happen to let older systems go to hell more often than we replace newer ones.
- tonyedgecombe 6y agoA savings account in 1940 is pretty much the same as a savings account today. The way customers interact with their bank has changed massively over that period. In the UK traditional banks are losing customers to new upstarts that don't have to bear the cost of all that legacy. They are nimble and much more responsive to their customers.
- jaclaz 6y agoA reference for you: https://jdebp.eu/FGA/legacy-is-not-a-pejorative.html https://jdebp.eu/FGA/legacy-is-not-a-pejorative.html
- info781 6y agoA salesman will call anything that is not his product "legacy" so the word can get confusing. Also there is an issue when commercial vendors start to overcharge, and their software needs to be replaced with some thing cheaper. This can become a higher priority than replacing COBAL.
- faebi 6y agoHow much do legacy systems affect the success of a business? Are there clear cases where companies with or without legacy systems are doing any better or worse? I feel like both ways seem to work with the right amount of knowledge and leadership.