9 ms·

The little legacy code that could: a fable of software ownership

216 points by icoe 7y ago

bitwize 7y ago
This made me think of an enterprise-software version of Wreck-it Ralph. I'd watch that, but I'd be part of a small audience (though arguably, Tron qualifies).
- Asooka 7y ago
  A small audience, but one with a good amount of disposable income.
- aeorgnoieang 7y ago
  I'd watch a movie based on The Phoenix Project book.
  - garyrichardson 7y ago
    I wonder if someone has optioned The Phoenix Project?
magashna 7y ago
It's kinda dumb but I do like this children's book retelling of software development
- rocky1138 7y ago
  It's not dumb. Some of the most impactful stories are fables. You may also like Java Koans.
- rocky1138 7y ago
  I wanted to update my comment but for some reason I can't. I mistakenly said Java Koans when I actually meant The Codeless Code, located here http://thecodelesscode.com/contents http://thecodelesscode.com/contents. Sorry about that!
urthen 7y ago
Cute story, but I detect a certain bitterness and refusal to accept the "future" of software engineering is actually the "now." People didn't think up microservices yesterday, they aren't some hot new fad that'll be forgotten in a year, and they aren't being developed because people just have nothing better to do. They're replacing legacy systems which nobody wants to touch at a rapid clip. People don't like working on legacy systems because they're at an unfortunate intersection of being business critical yet flaky and hard to work on. In the context of this story, well... your protagonist is actually on life support. But not to worry! They'll be taken off it soon...
- aeorgnoieang 7y ago
  If anything, it's a pretty pragmatic view about the high frequency with which new systems intended to replace legacy ones fail to do so. Certainly applications can be supplanted or replaced, and often are, at the discretion of users and customers. But 'living' (running) systems can often only be successfully replaced in the manner of the Ship of Theseus.
- lsh 7y ago
  I think you missed the point. Whatever you call your code, monolith or microservice, it's still code and tomorrow or next year it will be legacy code. I'm not sure if the size of the codebase actually matters, developers just want to develop and will move on quickly to the next thing. It takes time and patience and persistence to maintain legacy code and the systems it's embedded in and it's far from sexy or exciting. The young dominate IT and are way more idealistic and filled with enthusiasm and new ideas than those of us who are getting on. They want to make their mark and solve the world's problems and that means more and more code! But new code! This new code will be better, you'll see.
- zbentley 7y ago
  > In the context of this story, well... your protagonist is actually on life support. But not to worry! They'll be taken off it soon... "Soon" is often...not soon. I have never, ever seen a legacy system of any substantial complexity taken fully offline. Nor have I seen even partial replacements for such a system deliver a reliable alternative anywhere close to on time. In fact, I've seen more half-baked successors be either scrapped or put on life support themselves than I have seen get even into the ballpark of what could generously be considered success.
larrymyers 7y ago
This is one of those interesting paradoxes that I haven't figured out yet. Owning the legacy software that runs the business tends to provide job security at your current job, but can hinder your professional growth at both your current job as well as any future jobs. Getting on to projects that are intended to replace legacy software tends to get you a lot of positive visibility politically and the ability to learn new technology and skills. If the new project fails there tends to be little fallout from the failure, since so many people are attached to the project at all levels. The goal posts will move so it can be reframed as a success. If legacy software fails it usually means long hours and a lot of people nervously asking you when it will be fixed. Unfortunately there's just not a lot of value to being able to put on your resume that you're an expert at an older language that nobody has heard of, built on an in-house framework that will never be used outside of your current company.
- aeorgnoieang 7y ago
  > Unfortunately there's just not a lot of value to being able to put on your resume that you're an expert at an older language that nobody has heard of, built on an in-house framework that will never be used outside of your current company. I think this is, fortunately, only approximately true. Most of my career to-date has involved exactly this kind of work and it's certainly not true that there's no value in it, even on a resumé or mentioned in a job interview. And certainly some non-zero amount of people derive considerable value from exactly that kind of experience as some non-zero number of employers or customers need someone with it. And, in a lot of ways, working on legacy systems can be both immensely fun and rewarding. Even just incremental improvement in or around such systems can itself be immensely valuable, e.g. setting-up automated builds or deployments, adding integration tests (to make future refactoring easier and safer), or rewriting or replacing a key portion of the system using newer tools or with a better design made possible due to the wisdom accumulated by a system performing real work over a significant period of time.
  - noir_lord 7y ago
    Most of my career has been working on legacy projects (or fixing broken ones), I like greenfield well enough but I’ve no preference for it over maintenance. There is as much pleasure in making something broken work properly as creating the thing (imo). It’s a steady career as well, most programming is maintenance outside of the fail fast world of startups.
aeorgnoieang 7y ago
> ... Newton’s Third Law of System Administration, which says “a system untouched will remain online.” We know this isn’t true after some point, but until that point, it’s followed by ops teams around the world. LOL
phonebanshee 7y ago
This would have been much improved if the cute legacy system has been speaking a dialect of Latin found in ninth-century Bavaria, with a slang vocabulary developed for its specific problem. And insisted that everyone else use that dialect when speaking to it.
dudul 7y ago
"legacy code runs the business". While I do agree with this general comment, I've also seen legacy code killing the business. I've seen legacy code so buggy that it couldn't be fixed, that new features required quarters to be added, that customer service team had to double size every year to help customers with their bad experience. When legacy code is not properly maintained, it can become this unescapable hell. Yes, it sort of works but at what cost?
- jrjarrett 7y ago
  I read this article and I have to disagree with one of the major premises that "no one wants to own legacy code" -- it's more that in a lot of organizations, you're not ALLOWED to own legacy code. What I mean is that you may want to take "ownership" in the sense of learning it and improving it but because it's not new development, large barriers to paying down technical debt and updating to newer development methodologies is disallowed. So it stays, untouched, bit-rotted, and inflexible.
avgDev 7y ago
I am currently rewriting a 10 year old VB application, which is in web forms. That application communicates with AS400 DB2 database. I get to use as400 here and there. My application is in .NET Core 2. There are some issues as anything IBM can be a pain in the ass to work with. Also, the as400 dev and me have a tremendous knowledge gap. I know everything new such as unit testing, proper source control, dependency injection, linq and ORM tools. However, when it comes to speed of queries he is much better at optimization. Since, he has worked with some ancient languages he has a better understanding of low-level programming. Honestly, it is pretty interesting at what speeds the current tools allow us to develop applications. For example, he would need to set up the DB, then security, then applications, stored procedures, map parameters in code and so on. I can accomplish most of these things with an ORM like Entity Framework in less than an hour and have a working project, obviously depending on the scale of it. However, both approaches have pros and cons. Having old legacy code is problematic. It is harder and harder to find developers and rates are going up as the BIG fish still rely on them and can pay more.
- larrik 7y ago
  I worked at an AS/400 shop out of college (this was in the 2000's, though). The other programmers there did NOT have a better understanding of low-level programming than me. In fact, it was hard to call them programmers at all.
  - yellowapple 7y ago
    Yeah, a lot of AS/400 programming is actually pretty high-level (comparatively speaking). COBOL in general was meant to be accessible to end users (or at most power users) such that they could readily define business logic without having to resort to something like assembly or PL/I or what have you.
    - larrik 7y ago
      Yeah, these guys didn't really know COBOL either. They used RPG (an AS/400 exclusive language).
backslash_16 7y ago
For anyone who wants to learn about working with and on a "legacy" code-base, check out Michael Feather's book "Working Effectively with Legacy Code" In the beginning it goes over code smells and how to find refacing seams in your language of choice (C, C++, or Java) and then each chapter is a group of techniques you can use. Working on a service that had both a very old monolith and some brand new microservices, I found it invaluable. I think the first lesson I applied from it was using pinning tests for safer refactoring.
- touristtam 7y ago
  > I think the first lesson I applied from it was using pinning tests for safer refactoring. Thanks for the book recommendation. Funny to realize I have done this in the past without prior knowledge; it just made sense.
supernomad 7y ago
I think its very interesting to think about this article in the context on the open source projects that relied upon by the vast majority of major applications and companies. Specifically OpenSSL comes to mind here. That library has been used/leveraged in so many ways to build empires. Yet it's currently struggling to find dedicated engineers that will work on its core logic. It's an unfortunate situation and while yes there is LibreSSL and BoringSSL its non-trivial to move to their implementations (I have tried, its wrought with peril). OpenSSL isn't on its own either, there are many open source libraries/applications that are struggling to find "owners". None of which can be "easily" replaced by any means. I think the comments about "heroic" culture being fostered by the lack of ownership is spot on, but it isn't a bad thing unless those heroic actors aren't actively trying to get attention put on said pieces of legendary software. Imo its better to have heroic actors than nothing, and especially in the context of an opensource application/library that is used worldwide, I hope we can find more to generate more interest in working on them.
- phonon 7y ago
  https://esr.gitlab.io/loadsharers/ https://esr.gitlab.io/loadsharers/ http://www.catb.org/esr/loadsharers/ http://www.catb.org/esr/loadsharers/
reallydude 7y ago
> Legacy means it worked. No and shame on you for pushing this idea, is my first reaction. "worked" doesn't mean anything. Positing this redefinition of Legacy (and implying what "worked" means) rubs me the wrong way, especially when tying it to the idea of code ownership. Maybe I just have a warped perspective, except that I've run into these scenarios. "worked" might mean "it was easy to fix and it broke a lot", "it calculated the numbers correctly but nobody knows how to change it without it breaking", or "it costs $5k a run but it gives us the right count of a database column". Legacy means it might be working, depending on your CURRENT needs. Your current needs change, even when the code does not. Code ownership isn't at a single level, which makes some of this analysis seem idealistic.
- davesmith1983 7y ago
  > Legacy means it might be working, depending on your CURRENT needs. Your current needs change, even when the code does not. No working should mean "It fulfills the requirements". If the requirements change then the code should change. The other things you have mentioned are important but they aren't necessarily requirements e.g. Uptime / Reliability / Performance might be requirements but without stating what they should be you can't say that a piece of code doesn't work if it "slow". This sort of stuff is real basic software engineering.
  - reallydude 7y ago
    > If the requirements change then the code should change. That's the nature of the term "Legacy", that you are using. It met requirements that are not the same today (sometimes it's just standards of coding). It didn't change, hence it's legacy code.
    - davesmith1983 7y ago
      There is all sorts of wrong here. You are conflating several concepts. Firstly I was commenting on your very strange definition of what working meant. What we (programmers) normally mean by working we mean "It fulfills the requirement that it was developed against". If the requirements change, the code must change to reflect the changes in requirements. If the software is changed it is a different version of that software. Legacy has nothing to do with that. Legacy until relatively recently meant "Not supported" i.e. Windows XP is "Legacy" whereas Windows 7 is not. Consider the scenario: I am asked to write a program to play sound. So lets make up some trite requirements. * It be able to play .wav files * Sound should always come out of the default audio device as designated by the OS. It is written and released. We will call it version 1.0.0 and it is supported to December 2021. People ask for more features. These new features are: * It is able to play .mp3 files * The ability to choose the audio device that you can play sound through. It is written and released and it is called version 1.1.0 and it is supported to December 2022. Now the requirements have changed however because we are still in the year 2019 they are both still supported. However the requirements to the software has changed and thus there is a new version of the software to reflect the change in requirements. That should be related in the change-log (remember those things). In 2022 version 1.0 will no longer supported. There will be no defect fixes to it, but it still works i.e. fulfills the original requirements as listed above. However under the more traditional definition of Legacy Software it is considered legacy. Things like code quality, maintainability etc. are separate issues.
mrfredward 7y ago
>The initial development cost of software rounds to zero when compared to operating costs in perpetuity. Mostly a good article, but what is with this quote? Aside from being factually wrong from a dollars and sense standpoint, doesn't it contradict their point that legacy code doesn't get enough love?
- stahnma 7y ago
  Author here. I don't think it's wrong in most (nearly all) cases. The cost of running software, keeping it updated, changing the HW it runs on, keeping it compliant, etc over time costs much more than the dev time to build it. Unless your service is completely replaced every couple years, I'd venture operations cost more than development. (even if dev handles the ops). The point of the statement is designing for operability helps with ownership and combats the "leave it in the corner and hope for the best" mentality.
  - krisrm 7y ago
    This seems like moving the goalposts though. "Much more" is not "rounds to zero". Software is just expensive, period - to build, and to maintain.
- kradroy 7y ago
  I don't think it contradicts the point. When you have a legacy system that "sits in the corner" earning money, you're [the company] paying engineers to retain their services in emergencies. Features get added too, if you're interested in keeping or gaining customers. So construction cost as a proportion of total cost does trend to zero. The company/stockholders doesn't care about hot new technologies. They care about revenue. Customers don't care their software is written in this or that framework or is powered by deep learning. They care about stable, usable, useful apps. Businesspeople know this. Many engineers do not. I think that disconnect causes much strife in tech businesses.
lecarore 7y ago
I really like how they tell the story at the beginning, I felt sorry for an hypothetical piece of legendary code
galaxyLogic 7y ago
"Why is there unclear ownership?" Because Extreme Programming (and other agile practices?) make Shared Code-Ownership a "value".
- mikekchar 7y ago
  Well, I think the article not so great on this one. There is unclear ownership because management thinks that software is finished at some point. They think in terms of large, vague blocks of functionality and have difficulty understanding things at a finer scale. When requests come in from users of the software, they are often ignored by management because none of the requests are large enough to fit into their large, vague ideas. However, all the requests put together add up to a very large impact. The result is that the legacy code gets neglected and new greenfield projects get the attention. Because nobody works with the legacy code, it becomes strange and foreign. It is built with older technology which is no longer sexy and is boring on your CV, so the movers and shakers in the IT group tend to avoid it. It's not code ownership that's a problem -- the whole company washes their hands of it. In terms of code ownership, you can run your group in a number of different ways. I've seen projects with code ownership work and I've seen projects without code ownership work. I vastly prefer the latter, personally. As long as you are churning the code regularly, a lack of code ownership means that internal conflicts about how things should be designed are forced out in the open. They don't fester for years and years, where groups of developers end up saying, "I can't work with that person. They are crazy." You have the conflict early when it doesn't have such a large impact and you sort it out early (Note: some people are just inflexible -- if you find that kind of person, knowing about it early is also good. You can deal with it). When you avoid code ownership, you are also forced to have code that is clear for your entire group. For example, if you have a single person and they work in isolation, their code may be impenetrable to the others. But if each person in the group has to work on the code, code that most people don't understand has very little chance of surviving. Overall, I find that staying away from code ownership results in considerably more maintainable code. However, there are clearly advantages to code ownership as well. One shouldn't snub their nose at the benefit of being able to see a piece of code and say, "I wrote that". Having ownership makes it easier to have pride in your work. It can be a very motivating factor. If you have people on your team who are feeling disconnected and don't feel like they are personally making an impact on the group, giving them a little piece of code to own can be very good for them. Similarly, sometimes you don't have a choice for your team. Sometimes you have a team of people who just don't work well with each other and there is nothing you can do about it. Partitioning the code and allowing these people to keep their distance may be the only thing you can do. Ideals are great and if you can achieve them, it's wonderful. But you have to be conscious of reality. People are people. I could go on and on, but I guess my point is that very often I see remarks like the one you made that seem to take a very superficial view of things. There seems to be no effort made to understand why other people have a differing viewpoint. Of course, you also get fan-boi style postings of, "The-new-hotness is the best thing because reasons" which are also unfortunate. Usually the truth is somewhere in between.
shyneeup 7y ago
I think one of the reasons why it's such a conundrum is because software development as a practice is just now getting to the level of complexity where large groups of people have collaborated over a large codebase over a long time. One of the problems that the microservice architecture is trying to solve is by making a a large software entity a collection of smaller easier to manage entities. It's like single cell organisms evolving to multicell organisms...each cell becomes smaller and simpler and more specialized and replacement of cells becomes easier and allows the overall lifeform to "scale" and become more complex. Mature software companies have gone through this transformation multiple times already in the form of, n-tier, SOA, microservices, and now serverless architectures. This is all part of a natural progression of making individual components simpler, the overall structure more granular, and as a result more resilient. This resiliency opens up new capabilities to scale a complex system. My long winded point being that legacy software will always exist yes, but each piece of legacy software is already getting smaller and more granular where rewriting it will eventually be a more continuous operation of refactoring small things--kind of like skin cells falling off your body or hair falling out.
gpvos 7y ago
SRE = Site Reliability Engineering, see https://en.wikipedia.org/wiki/Site_Reliability_Engineering https://en.wikipedia.org/wiki/Site_Reliability_Engineering MVP = (in this case, probably) Minimum Viable Product, see https://en.wikipedia.org/wiki/Minimum_viable_product https://en.wikipedia.org/wiki/Minimum_viable_product
Animats 7y ago
The underlying technology doesn't have to be obsolete to have legacy code problems. Linux, Windows, GCC, Microsoft Office - all have serious internal problems from legacy code, and are tough to maintain. It's not so bad for big projects with enough staff. It gets tough when the legacy code does something hard, and the maintenance team doesn't really understand why something was done in some way. Second Life, the virtual world, has that problem. It's written in C++, and some very good people wrote it about 10-15 years ago. They're all gone. The people who maintain it today are struggling to fix serious bugs that have been outstanding for 5-8 years. It's not just "legacy", it's "not web-like". It's a distributed system with tens of thousands of servers in one data center. It's a tightly coordinated soft real time system. There's a huge amount of in-memory state, which is changeable in real time yet is constantly being backed up. This is totally alien to people who only know transaction-type web-based systems. So they can't hire anybody and have them be productive quickly.
- bjt 7y ago
  Doesn't that describe a lot of MMOs too, though? The inventory/transaction systems seem like they'd have similar needs (but with a LOT more items in the Second Life database). And the in-world state is a lot more variable than in an MMO. But compared to MMOs, those seem more like differences in degree than differences in kind. Programmers with that background should be able to get productive a lot quicker than web devs I'd imagine.
  - Animats 7y ago
    Most major games are built on one of a few game engines (Unity, UE4, etc.) which have their own ecosystems. People become Unity or UE4 experts. They have forums, conferences, tech support. SL is its own pocket universe. Second Life is divided into "regions", 256m on a side, each managed by a separate process constantly communicating with its neighbors. This geographical distribution system is unique to Second Life. User avatars and objects can move from one region to another. You can look across region boundaries. Running Mono programs inside objects are stopped, frozen, copied across the network, and restarted on a different machine. Most big-world MMOs cheat somehow so that they don't have to really solve the distribution problem. They're often sharded, so that the number of players that can interact is limited. Or they're smaller. Second Life's world is 100x the size of GTA V. Or they're portal based; you can only get somewhere via a controlled portal. In Second Life, you can fly over the whole world. (Mostly. Fast vehicles hit bugs the devs have been unable to fix for a decade. Another "legacy" problem.) This is what the machinery for a "Ready Player One" or "Snow Crash" world looks like. It's not parallel enough, and the servers keep running out of CPU time on the main thread. Everything then gets sluggish in world. The system needs an overhaul to be more parallel internally on the core functions, and that's really hard, expensive, and needs a dev team the company lacks. Yet another legacy problem. The technology is all unique to this one system. The only thing that works even vaguely like this is the new Spatial OS from Improbable, which took 150 people to develop, is proprietary, and hasn't been shown to really scale yet. We'll know late this year, as Nostos, a new game from China, rolls out, how well it really scales. That's the first AAA title to use Spatial OS. Spatial OS has a deal with Google where it has to run on Google Cloud servers, which is scaring off most of the big game development shops. That costs too much, and betting your business on a lesser Google product usually ends badly. Hence the recruiting problem Linden Lab faces. You want to tie your career to this one-off strange system?

216 points by icoe 7y ago

This made me think of an enterprise-software version of Wreck-it Ralph. I'd watch that, but I'd be part of a small audience (though arguably, Tron qualifies).

Asooka 7y ago
A small audience, but one with a good amount of disposable income.
aeorgnoieang 7y ago
I'd watch a movie based on The Phoenix Project book.
- garyrichardson 7y ago
  I wonder if someone has optioned The Phoenix Project?

magashna 7y ago

It's kinda dumb but I do like this children's book retelling of software development

rocky1138 7y ago
It's not dumb. Some of the most impactful stories are fables. You may also like Java Koans.
rocky1138 7y ago
I wanted to update my comment but for some reason I can't. I mistakenly said Java Koans when I actually meant The Codeless Code, located here http://thecodelesscode.com/contents http://thecodelesscode.com/contents. Sorry about that!

urthen 7y ago

Cute story, but I detect a certain bitterness and refusal to accept the "future" of software engineering is actually the "now." People didn't think up microservices yesterday, they aren't some hot new fad that'll be forgotten in a year, and they aren't being developed because people just have nothing better to do. They're replacing legacy systems which nobody wants to touch at a rapid clip. People don't like working on legacy systems because they're at an unfortunate intersection of being business critical yet flaky and hard to work on. In the context of this story, well... your protagonist is actually on life support. But not to worry! They'll be taken off it soon...

aeorgnoieang 7y ago
If anything, it's a pretty pragmatic view about the high frequency with which new systems intended to replace legacy ones fail to do so. Certainly applications can be supplanted or replaced, and often are, at the discretion of users and customers. But 'living' (running) systems can often only be successfully replaced in the manner of the Ship of Theseus.
lsh 7y ago
I think you missed the point. Whatever you call your code, monolith or microservice, it's still code and tomorrow or next year it will be legacy code. I'm not sure if the size of the codebase actually matters, developers just want to develop and will move on quickly to the next thing. It takes time and patience and persistence to maintain legacy code and the systems it's embedded in and it's far from sexy or exciting. The young dominate IT and are way more idealistic and filled with enthusiasm and new ideas than those of us who are getting on. They want to make their mark and solve the world's problems and that means more and more code! But new code! This new code will be better, you'll see.
zbentley 7y ago
> In the context of this story, well... your protagonist is actually on life support. But not to worry! They'll be taken off it soon... "Soon" is often...not soon. I have never, ever seen a legacy system of any substantial complexity taken fully offline. Nor have I seen even partial replacements for such a system deliver a reliable alternative anywhere close to on time. In fact, I've seen more half-baked successors be either scrapped or put on life support themselves than I have seen get even into the ballpark of what could generously be considered success.

larrymyers 7y ago

This is one of those interesting paradoxes that I haven't figured out yet. Owning the legacy software that runs the business tends to provide job security at your current job, but can hinder your professional growth at both your current job as well as any future jobs. Getting on to projects that are intended to replace legacy software tends to get you a lot of positive visibility politically and the ability to learn new technology and skills. If the new project fails there tends to be little fallout from the failure, since so many people are attached to the project at all levels. The goal posts will move so it can be reframed as a success. If legacy software fails it usually means long hours and a lot of people nervously asking you when it will be fixed. Unfortunately there's just not a lot of value to being able to put on your resume that you're an expert at an older language that nobody has heard of, built on an in-house framework that will never be used outside of your current company.

aeorgnoieang 7y ago
> Unfortunately there's just not a lot of value to being able to put on your resume that you're an expert at an older language that nobody has heard of, built on an in-house framework that will never be used outside of your current company. I think this is, fortunately, only approximately true. Most of my career to-date has involved exactly this kind of work and it's certainly not true that there's no value in it, even on a resumé or mentioned in a job interview. And certainly some non-zero amount of people derive considerable value from exactly that kind of experience as some non-zero number of employers or customers need someone with it. And, in a lot of ways, working on legacy systems can be both immensely fun and rewarding. Even just incremental improvement in or around such systems can itself be immensely valuable, e.g. setting-up automated builds or deployments, adding integration tests (to make future refactoring easier and safer), or rewriting or replacing a key portion of the system using newer tools or with a better design made possible due to the wisdom accumulated by a system performing real work over a significant period of time.
- noir_lord 7y ago
  Most of my career has been working on legacy projects (or fixing broken ones), I like greenfield well enough but I’ve no preference for it over maintenance. There is as much pleasure in making something broken work properly as creating the thing (imo). It’s a steady career as well, most programming is maintenance outside of the fail fast world of startups.

aeorgnoieang 7y ago

> ... Newton’s Third Law of System Administration, which says “a system untouched will remain online.” We know this isn’t true after some point, but until that point, it’s followed by ops teams around the world. LOL

phonebanshee 7y ago

This would have been much improved if the cute legacy system has been speaking a dialect of Latin found in ninth-century Bavaria, with a slang vocabulary developed for its specific problem. And insisted that everyone else use that dialect when speaking to it.

dudul 7y ago

"legacy code runs the business". While I do agree with this general comment, I've also seen legacy code killing the business. I've seen legacy code so buggy that it couldn't be fixed, that new features required quarters to be added, that customer service team had to double size every year to help customers with their bad experience. When legacy code is not properly maintained, it can become this unescapable hell. Yes, it sort of works but at what cost?

jrjarrett 7y ago
I read this article and I have to disagree with one of the major premises that "no one wants to own legacy code" -- it's more that in a lot of organizations, you're not ALLOWED to own legacy code. What I mean is that you may want to take "ownership" in the sense of learning it and improving it but because it's not new development, large barriers to paying down technical debt and updating to newer development methodologies is disallowed. So it stays, untouched, bit-rotted, and inflexible.

avgDev 7y ago

I am currently rewriting a 10 year old VB application, which is in web forms. That application communicates with AS400 DB2 database. I get to use as400 here and there. My application is in .NET Core 2. There are some issues as anything IBM can be a pain in the ass to work with. Also, the as400 dev and me have a tremendous knowledge gap. I know everything new such as unit testing, proper source control, dependency injection, linq and ORM tools. However, when it comes to speed of queries he is much better at optimization. Since, he has worked with some ancient languages he has a better understanding of low-level programming. Honestly, it is pretty interesting at what speeds the current tools allow us to develop applications. For example, he would need to set up the DB, then security, then applications, stored procedures, map parameters in code and so on. I can accomplish most of these things with an ORM like Entity Framework in less than an hour and have a working project, obviously depending on the scale of it. However, both approaches have pros and cons. Having old legacy code is problematic. It is harder and harder to find developers and rates are going up as the BIG fish still rely on them and can pay more.

larrik 7y ago
I worked at an AS/400 shop out of college (this was in the 2000's, though). The other programmers there did NOT have a better understanding of low-level programming than me. In fact, it was hard to call them programmers at all.
- yellowapple 7y ago
  Yeah, a lot of AS/400 programming is actually pretty high-level (comparatively speaking). COBOL in general was meant to be accessible to end users (or at most power users) such that they could readily define business logic without having to resort to something like assembly or PL/I or what have you.
  - larrik 7y ago
    Yeah, these guys didn't really know COBOL either. They used RPG (an AS/400 exclusive language).

backslash_16 7y ago

For anyone who wants to learn about working with and on a "legacy" code-base, check out Michael Feather's book "Working Effectively with Legacy Code" In the beginning it goes over code smells and how to find refacing seams in your language of choice (C, C++, or Java) and then each chapter is a group of techniques you can use. Working on a service that had both a very old monolith and some brand new microservices, I found it invaluable. I think the first lesson I applied from it was using pinning tests for safer refactoring.

touristtam 7y ago
> I think the first lesson I applied from it was using pinning tests for safer refactoring. Thanks for the book recommendation. Funny to realize I have done this in the past without prior knowledge; it just made sense.

supernomad 7y ago

I think its very interesting to think about this article in the context on the open source projects that relied upon by the vast majority of major applications and companies. Specifically OpenSSL comes to mind here. That library has been used/leveraged in so many ways to build empires. Yet it's currently struggling to find dedicated engineers that will work on its core logic. It's an unfortunate situation and while yes there is LibreSSL and BoringSSL its non-trivial to move to their implementations (I have tried, its wrought with peril). OpenSSL isn't on its own either, there are many open source libraries/applications that are struggling to find "owners". None of which can be "easily" replaced by any means. I think the comments about "heroic" culture being fostered by the lack of ownership is spot on, but it isn't a bad thing unless those heroic actors aren't actively trying to get attention put on said pieces of legendary software. Imo its better to have heroic actors than nothing, and especially in the context of an opensource application/library that is used worldwide, I hope we can find more to generate more interest in working on them.

phonon 7y ago
https://esr.gitlab.io/loadsharers/ https://esr.gitlab.io/loadsharers/ http://www.catb.org/esr/loadsharers/ http://www.catb.org/esr/loadsharers/

reallydude 7y ago

> Legacy means it worked. No and shame on you for pushing this idea, is my first reaction. "worked" doesn't mean anything. Positing this redefinition of Legacy (and implying what "worked" means) rubs me the wrong way, especially when tying it to the idea of code ownership. Maybe I just have a warped perspective, except that I've run into these scenarios. "worked" might mean "it was easy to fix and it broke a lot", "it calculated the numbers correctly but nobody knows how to change it without it breaking", or "it costs $5k a run but it gives us the right count of a database column". Legacy means it might be working, depending on your CURRENT needs. Your current needs change, even when the code does not. Code ownership isn't at a single level, which makes some of this analysis seem idealistic.

davesmith1983 7y ago
> Legacy means it might be working, depending on your CURRENT needs. Your current needs change, even when the code does not. No working should mean "It fulfills the requirements". If the requirements change then the code should change. The other things you have mentioned are important but they aren't necessarily requirements e.g. Uptime / Reliability / Performance might be requirements but without stating what they should be you can't say that a piece of code doesn't work if it "slow". This sort of stuff is real basic software engineering.
- reallydude 7y ago
  > If the requirements change then the code should change. That's the nature of the term "Legacy", that you are using. It met requirements that are not the same today (sometimes it's just standards of coding). It didn't change, hence it's legacy code.
  - davesmith1983 7y ago
    There is all sorts of wrong here. You are conflating several concepts. Firstly I was commenting on your very strange definition of what working meant. What we (programmers) normally mean by working we mean "It fulfills the requirement that it was developed against". If the requirements change, the code must change to reflect the changes in requirements. If the software is changed it is a different version of that software. Legacy has nothing to do with that. Legacy until relatively recently meant "Not supported" i.e. Windows XP is "Legacy" whereas Windows 7 is not. Consider the scenario: I am asked to write a program to play sound. So lets make up some trite requirements. * It be able to play .wav files * Sound should always come out of the default audio device as designated by the OS. It is written and released. We will call it version 1.0.0 and it is supported to December 2021. People ask for more features. These new features are: * It is able to play .mp3 files * The ability to choose the audio device that you can play sound through. It is written and released and it is called version 1.1.0 and it is supported to December 2022. Now the requirements have changed however because we are still in the year 2019 they are both still supported. However the requirements to the software has changed and thus there is a new version of the software to reflect the change in requirements. That should be related in the change-log (remember those things). In 2022 version 1.0 will no longer supported. There will be no defect fixes to it, but it still works i.e. fulfills the original requirements as listed above. However under the more traditional definition of Legacy Software it is considered legacy. Things like code quality, maintainability etc. are separate issues.

mrfredward 7y ago

>The initial development cost of software rounds to zero when compared to operating costs in perpetuity. Mostly a good article, but what is with this quote? Aside from being factually wrong from a dollars and sense standpoint, doesn't it contradict their point that legacy code doesn't get enough love?

stahnma 7y ago
Author here. I don't think it's wrong in most (nearly all) cases. The cost of running software, keeping it updated, changing the HW it runs on, keeping it compliant, etc over time costs much more than the dev time to build it. Unless your service is completely replaced every couple years, I'd venture operations cost more than development. (even if dev handles the ops). The point of the statement is designing for operability helps with ownership and combats the "leave it in the corner and hope for the best" mentality.
- krisrm 7y ago
  This seems like moving the goalposts though. "Much more" is not "rounds to zero". Software is just expensive, period - to build, and to maintain.
kradroy 7y ago
I don't think it contradicts the point. When you have a legacy system that "sits in the corner" earning money, you're [the company] paying engineers to retain their services in emergencies. Features get added too, if you're interested in keeping or gaining customers. So construction cost as a proportion of total cost does trend to zero. The company/stockholders doesn't care about hot new technologies. They care about revenue. Customers don't care their software is written in this or that framework or is powered by deep learning. They care about stable, usable, useful apps. Businesspeople know this. Many engineers do not. I think that disconnect causes much strife in tech businesses.

lecarore 7y ago

I really like how they tell the story at the beginning, I felt sorry for an hypothetical piece of legendary code

galaxyLogic 7y ago

"Why is there unclear ownership?" Because Extreme Programming (and other agile practices?) make Shared Code-Ownership a "value".

mikekchar 7y ago
Well, I think the article not so great on this one. There is unclear ownership because management thinks that software is finished at some point. They think in terms of large, vague blocks of functionality and have difficulty understanding things at a finer scale. When requests come in from users of the software, they are often ignored by management because none of the requests are large enough to fit into their large, vague ideas. However, all the requests put together add up to a very large impact. The result is that the legacy code gets neglected and new greenfield projects get the attention. Because nobody works with the legacy code, it becomes strange and foreign. It is built with older technology which is no longer sexy and is boring on your CV, so the movers and shakers in the IT group tend to avoid it. It's not code ownership that's a problem -- the whole company washes their hands of it. In terms of code ownership, you can run your group in a number of different ways. I've seen projects with code ownership work and I've seen projects without code ownership work. I vastly prefer the latter, personally. As long as you are churning the code regularly, a lack of code ownership means that internal conflicts about how things should be designed are forced out in the open. They don't fester for years and years, where groups of developers end up saying, "I can't work with that person. They are crazy." You have the conflict early when it doesn't have such a large impact and you sort it out early (Note: some people are just inflexible -- if you find that kind of person, knowing about it early is also good. You can deal with it). When you avoid code ownership, you are also forced to have code that is clear for your entire group. For example, if you have a single person and they work in isolation, their code may be impenetrable to the others. But if each person in the group has to work on the code, code that most people don't understand has very little chance of surviving. Overall, I find that staying away from code ownership results in considerably more maintainable code. However, there are clearly advantages to code ownership as well. One shouldn't snub their nose at the benefit of being able to see a piece of code and say, "I wrote that". Having ownership makes it easier to have pride in your work. It can be a very motivating factor. If you have people on your team who are feeling disconnected and don't feel like they are personally making an impact on the group, giving them a little piece of code to own can be very good for them. Similarly, sometimes you don't have a choice for your team. Sometimes you have a team of people who just don't work well with each other and there is nothing you can do about it. Partitioning the code and allowing these people to keep their distance may be the only thing you can do. Ideals are great and if you can achieve them, it's wonderful. But you have to be conscious of reality. People are people. I could go on and on, but I guess my point is that very often I see remarks like the one you made that seem to take a very superficial view of things. There seems to be no effort made to understand why other people have a differing viewpoint. Of course, you also get fan-boi style postings of, "The-new-hotness is the best thing because reasons" which are also unfortunate. Usually the truth is somewhere in between.

shyneeup 7y ago

I think one of the reasons why it's such a conundrum is because software development as a practice is just now getting to the level of complexity where large groups of people have collaborated over a large codebase over a long time. One of the problems that the microservice architecture is trying to solve is by making a a large software entity a collection of smaller easier to manage entities. It's like single cell organisms evolving to multicell organisms...each cell becomes smaller and simpler and more specialized and replacement of cells becomes easier and allows the overall lifeform to "scale" and become more complex. Mature software companies have gone through this transformation multiple times already in the form of, n-tier, SOA, microservices, and now serverless architectures. This is all part of a natural progression of making individual components simpler, the overall structure more granular, and as a result more resilient. This resiliency opens up new capabilities to scale a complex system. My long winded point being that legacy software will always exist yes, but each piece of legacy software is already getting smaller and more granular where rewriting it will eventually be a more continuous operation of refactoring small things--kind of like skin cells falling off your body or hair falling out.

gpvos 7y ago

SRE = Site Reliability Engineering, see https://en.wikipedia.org/wiki/Site_Reliability_Engineering https://en.wikipedia.org/wiki/Site_Reliability_Engineering MVP = (in this case, probably) Minimum Viable Product, see https://en.wikipedia.org/wiki/Minimum_viable_product https://en.wikipedia.org/wiki/Minimum_viable_product

Animats 7y ago

The underlying technology doesn't have to be obsolete to have legacy code problems. Linux, Windows, GCC, Microsoft Office - all have serious internal problems from legacy code, and are tough to maintain. It's not so bad for big projects with enough staff. It gets tough when the legacy code does something hard, and the maintenance team doesn't really understand why something was done in some way. Second Life, the virtual world, has that problem. It's written in C++, and some very good people wrote it about 10-15 years ago. They're all gone. The people who maintain it today are struggling to fix serious bugs that have been outstanding for 5-8 years. It's not just "legacy", it's "not web-like". It's a distributed system with tens of thousands of servers in one data center. It's a tightly coordinated soft real time system. There's a huge amount of in-memory state, which is changeable in real time yet is constantly being backed up. This is totally alien to people who only know transaction-type web-based systems. So they can't hire anybody and have them be productive quickly.

bjt 7y ago
Doesn't that describe a lot of MMOs too, though? The inventory/transaction systems seem like they'd have similar needs (but with a LOT more items in the Second Life database). And the in-world state is a lot more variable than in an MMO. But compared to MMOs, those seem more like differences in degree than differences in kind. Programmers with that background should be able to get productive a lot quicker than web devs I'd imagine.
- Animats 7y ago
  Most major games are built on one of a few game engines (Unity, UE4, etc.) which have their own ecosystems. People become Unity or UE4 experts. They have forums, conferences, tech support. SL is its own pocket universe. Second Life is divided into "regions", 256m on a side, each managed by a separate process constantly communicating with its neighbors. This geographical distribution system is unique to Second Life. User avatars and objects can move from one region to another. You can look across region boundaries. Running Mono programs inside objects are stopped, frozen, copied across the network, and restarted on a different machine. Most big-world MMOs cheat somehow so that they don't have to really solve the distribution problem. They're often sharded, so that the number of players that can interact is limited. Or they're smaller. Second Life's world is 100x the size of GTA V. Or they're portal based; you can only get somewhere via a controlled portal. In Second Life, you can fly over the whole world. (Mostly. Fast vehicles hit bugs the devs have been unable to fix for a decade. Another "legacy" problem.) This is what the machinery for a "Ready Player One" or "Snow Crash" world looks like. It's not parallel enough, and the servers keep running out of CPU time on the main thread. Everything then gets sluggish in world. The system needs an overhaul to be more parallel internally on the core functions, and that's really hard, expensive, and needs a dev team the company lacks. Yet another legacy problem. The technology is all unique to this one system. The only thing that works even vaguely like this is the new Spatial OS from Improbable, which took 150 people to develop, is proprietary, and hasn't been shown to really scale yet. We'll know late this year, as Nostos, a new game from China, rolls out, how well it really scales. That's the first AAA title to use Spatial OS. Spatial OS has a deal with Google where it has to run on Google Cloud servers, which is scaring off most of the big game development shops. That costs too much, and betting your business on a lesser Google product usually ends badly. Hence the recruiting problem Linden Lab faces. You want to tie your career to this one-off strange system?