8 ms·

I work on some software of which the oldest parts of the source code date back to about 2009. Over the years, some very smart (some of them dangerously smart an

I work on some software of which the oldest parts of the source code date back to about 2009. Over the years, some very smart (some of them dangerously smart and woefully inexperienced, and clearly - not at all their fault - not properly mentored or supervised) people worked on it and left. What we have now is frequently a mystery. Simple changes are difficult, difficult changes verge on the impossible. Every new feature requires reverse-engineering of the existing code. Sometimes literally 95% of the time is spent reverse-engineering the existing code (no exaggeration - we measured it); changes can take literally 20 times as long as they should while we work out what the existing code does (and also, often not quite the same, what it's meant to do, which is sometimes simply impossible to ever know). Pieces are gradually being documented as we work out what they do, but layers of cruft from years gone by from people with deadlines to meet and no chance of understanding the existing code sit like landmines and sometimes like unbreakable bonds that can never be undone. In our estimates, every time we have to rely on existing functionality that should be rock solid reliable and completely understood yet that we have not yet had to fully reverse-engineer, we mark it "high risk, add a month". The time I found that someone had rewritten several pieces of the Qt libraries (without documenting what, or why) was devastating; it took away one of the cornerstones I'd been relying on, the one marked "at least I know I can trust the Qt libraries". It doesn't matter how smart we are, how skilled a coder we are, how genius our algorithms are; if we write something that can't be understood by the next person to read it, and isn't properly documented somewhere in some way that our fifth replacement can find easily five years later - if we write something of which even the purpose, let alone the implementation, will take someone weeks to reverse engineer - we're writing legacy code on day one and, while we may be skilled programmers, we're truly Godawful software engineers.

tracer4201 7y ago
Engineers are often ready to scrap things and start over, but sometimes it’s worth having a couple senior/principal/staff level folks do a quick evaluation and figure out if you’re needlessly duct taping an unmanageable piece of service/application/whatever abstraction and actually scrapping it might be the better option. From my own experience, you don’t actually know how the existing system works, the expected business rules were never documented, etc, and so even starting over isn’t really a less ambiguous path.
- EliRivers 7y ago
  so even starting over isn’t really a less ambiguous path. Sure isn't. We don't know what the software does right now, but there are customers all over the world using it and paying tens of thousands per year for support contracts; since we don't know quite what it does, we don't know quite what they're doing with it. We can pretty much guarantee that if we started over, we would miss a great many requirements (and a great many bugs that happen to work for various specific customers) and a lot of customers would be very unhappy. We don't know what it does, they don't know what it does, but it's working for them at the moment. Oh Gods, even the unit tests. There are some. Sometimes one fails. We stare at it. It was expecting this value to be 3, now the value is 4. Why was it expected to be 3? Is it wrong that it's now 4, or have we changed the behaviour such that the correct answer is now 4 and we need to change the test? Undocumented unit tests are worthless. I'm not asking for a lot of documentation - just commentary in the code next to them would be fine - but if I can't see what's actually being tested and how that "correct" value was ascertained, when it fails, I don't know if the bug is in the code or the test. Useless unit test.
  - ownagefool 7y ago
    So you probably want to log the requests, figure out what people are doing.
    - EliRivers 7y ago
      There are no requests to log. It's not a web app. It's installed on at least one and up to many networked windows machines on a given customer site, where that network is isolated and contains numerous IP and serial cable connected pieces of specialist hardware (no two customers have the same sets of hardware); we make them dance together. There is no interaction outside that network.
      - ownagefool 7y ago
        Fair enough, that does make the problem harder.
dang 7y ago
Peter Naur's classic 1985 essay "Programming as Theory Building" argues that this is because a program is not its source code. A program is a shared mental construct (he uses the word theory) that lives in the minds of the people who work on it. If you lose the people, you lose the program. The code is merely a written representation of the program, and it's lossy, so you can't reconstruct a program from its code. https://news.ycombinator.com/item?id=10833278 https://news.ycombinator.com/item?id=10833278 https://news.ycombinator.com/item?id=7491661 https://news.ycombinator.com/item?id=7491661
- Waterluvian 7y ago
  Thanks for sharing this. I've been looking for this for a long time to replace my own poor attempt at saying the same: The code you write isn't the hard part. It's not the interesting part. And it's not really why you're paid. What I'm trying to imbue into you is the discipline to design and document and plan for the future. Saying you can "whip up a script this afternoon" is not impressive or commendable. It does harm without the other pieces.
  - closeparen 7y ago
    Here's the thing: when I see people spend all their time on design discussions and documents, their work isn't any better for it. Usually it's worse. The documents are not clear or helpful. The designs encoded therein are not better than the first pass knee-jerk thought. The points raised in reviews are just bikeshedding. The process encourages explosion of complexity, making mountains of molehills, and sometimes can be a refuge for people who don't actually have the skills to hide out in. I do very much respect when someone can plop down some working code this afternoon vs. organize an 8-person kickoff meeting next week. If it's a genuinely large project, the working code is a skeleton POC, and we can run a design process using what we learned from it. But most of the time it's not, and that engineer has saved us collectively hundreds of hours of waste talking to death something that just isn't that hard. I think one of the critical senior engineer skills is having correct intuitions about how much time and energy should be sunk on a particular task. Sometimes that means asking people to slow down, collaborate, and think. Equally often, it means asking them to put down the calendar invites, make a choice, and bang something out. We have this bizarre cost model where code is expensive and meetings are free; nothing could be further from the truth.
tus88 7y ago
Legacy code from 2009. I'm getting old.
- EliRivers 7y ago
  Admittedly playing around with the definition of "legacy code" (and I get what you meant - I'm just playing with the words!), this particular set of code is legacy code from 2009, but I think I could argue that some of the world's legacy code was written yesterday, and there is code from decades ago that is not legacy code. This particular set, though, was legacy code pretty much the week after it was written :(
  - olikas 7y ago
    I have played with this idea recently lately. I consider almost all code out there as legacy. I was talking about it at a conference also explaining how I like working with legacy code. (https://youtu.be/n0XCMHrSbwc https://youtu.be/n0XCMHrSbwc) I think I’m the odd one who actually like working with legacy code.
    - repsilat 7y ago
      > I think I’m the odd one who actually like working with legacy code. I had a fun bit of legacy code wrangling a month or so back. I looked at production profiles of some very compute-heavy jobs (we have great tools for that) and found we were spending ~half of my pay on cloud compute in this one library function -- deep equality checks for a kind of object. The equality-checking code itself is very slow, significantly slower than hand-written alternatives, but the objects are complicated (though luckily tree-like in structure) so hand-writing equality comparison functions would be a nightmare. Many of these structures had "id" fields, though. Gasp, could we just compare ids instead of doing deep equality checks? So I asked around, and nobody knew. Nobody even knew what the code was for. It'd be trivial to replace the deep equality checks with id comparisons, but we can't do it. Seeing how some legacy code works is one thing. Data comes in here, data goes out there. Knowing which invariants the code is trying to maintain, though, what properties it assumes about the data here or there (Sorted? Unique? Up to date? All belongs to the same user?) makes making changes difficult, though. It means you need to write defensive code that the original authors would find ridiculous, because you don't have their institutional knowledge, and can't derive it locally in the code, and can't inspect it at runtime (or be sure that prod won't send a counter-example the day after you deploy to production.)
mooreds 7y ago
How do you sell the reverse engineering? How does "the business" feel about this situation? How does the team feel about it?
- rocky1138 7y ago
  You don't have to sell anything. It's part of getting the job done.
  - mooreds 7y ago
    In an ideal world, correct. If this is the case: > changes can take literally 20 times as long as they should while we work out what the existing code does I imagine you have to sell the extra time to (aka convince folks this is the right path) _somebody_. Maybe I've been working at the wrong places :) .
    - EliRivers 7y ago
      No reverse-engineering - no new feature or bugfix. That's all there is to it.
      - mooreds 7y ago
        Did the business have to get bit a few times (with, say production outages?) before this became the protocol?
        EliRivers 7y ago
        "Production outages" aren't something we have to worry about; we don't run a service and once something is working on a customer site, they basically don't tinker around with the configuration or setup. They just use it. But there have certainly been projects that massively overran because the estimates assumed that pieces that already existed were understood and reliable (sometimes they were reliable; they reliably did something, we just thought they were meant to do something else). This happens less now that we assume any existing piece that we don't know about already will turn out to be a minefield.
WalterBright 7y ago
When I have some time available, but not enough time to really get into something, I just pick up a piece of code and try to improve it. Over time, I hope this leads to gradual removal of the cruft that accumulates over time.
closeparen 7y ago
There are worse things than no documentation when doing one of these excavations. Design docs that were aspirational and only partially implemented. Code comments that are incorrect, or reflect how things used to work. Reams of beautiful code and through documentation that it turns out are not the production implementation anymore because of a feature flag change or migration elsewhere. Migration trackers with open action items and deadlines 7 years in the past, creepily abandoned mid-flight like Pripyat. Detective work can go a long way, but in situations like this there's no substitute for ownership continuity and institutional memory.
zackmorris 7y ago
See Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations. - M. Conway This happened to me on a project. I spent months documenting the Byzantine labyrinth of hundreds of interwoven services that had been patched for a decade by a hundred developers. Much of it had been done under pair programming, so required pair programming or a 120 IQ just to decipher it. In the end I was squeezed out not because I was wrong, but because my method of solving longstanding issues from first principles didn't fit within the company politics of the client's organization. The gist of it is that clients mostly see short term personal productivity and not the long term gains from removing technical debt or avoiding it altogether by way of the developer's experience. Senior engineers can eventually reach a point where there isn't much for them to contribute on a project because every last issue could have been avoided through proper planning and engineering in the first place. In my case, my greatest strengths (like refactoring ability) were overlooked and my greatest weaknesses (like implementing features at high speed by repeating myself or ignoring project conventions because I don't know any better - something I can no longer do) were brought to the forefront.
- repsilat 7y ago
  A year ago I thought Conway's law was nonsense, but recent experience has led me to walk some of that back. I would replace "are constrained to" with "grow to", or perhaps just "often". On my first day of my current job, Mufasa (my boss) took me up on Pride Rock at sunrise, and we cast our eyes over the system diagram of the back-end supporting our product. And he told me, "We work on this box here." Our team's box gets data from that team's box, and we output data to be consumed by this other team's box... "Any what about that shadowy land?" I asked, pointing to the arrows coming from outside the diagram. "That is the Front-end. You must never go there." So yeah, Conway's law through and through. This is a big company, though, and a long-lived product. At my last job, at a startup that was ~3 years old, things were intentionally very different. This month you and a few others worked on feature X. Not on this system or that system, you were implementing something for the customer. Maybe you're a backend engineer, but you still wrote the database migration, the queries, the endpoints, the front-end logic, markup, CSS, whatever. Some of it you sucked at, but a lot of that was handled by good team composition. If another engineer on that feature knew CSS better, they mostly styled the things and you reviewed their work. Maybe you styled the next thing. I was frustrated in that job, because I didn't like writing CSS, and I wasn't good at it, and I was very good at some other things. Why couldn't I just do those things I was good at? It would have kept me a lot more productive, and saved (and made) the company a lot of money! Why did I have to do things I was bad at? Did they think engineers were interchangeable? Well, now I know. In my current job, working on five different features in my box in the system diagram, I'm super productive when pointed in the right direction with enough to work on and some autonomy. That just doesn't happen as often. We need to communicate between teams a lot more. We need to align our quarterly deliverables. We duplicate functionality on both sides of an interface, (or in each of N components), with subtly different semantics, because nobody technical is responsible for the feature. Show-stopping bugs before launch are a fact of life, because integration tests between our components suck. Generously (though for other reasons too), we're 1/3 as productive as my last gig was. Conway's law is strangling us.
techslave 7y ago
most important part is hidden in there: > with deadlines to meet the reason the company is there, employing you now, is that deadlines were met.
- EliRivers 7y ago
  That is also the reason that the company is now sometimes spending 100000 dollars on work that should cost 5000. A few years ago, the whole edifice was dangerously close to becoming literally unmaintainable; collapsing under its own complexity, the interest payments on the technical debt requiring more than 100% of available output. Those deadlines were frequently missed, by the way. Sometimes by months. I think not quite ever by years.
z3t4 7y ago
Could start with regression testing, each time you break something, write a test so that you do not break the same thing again. After a time you will have a decent test suit. Also write tests for all new stuff. And also write tests for everything you change. Would also be an idea to start using version control, so that each change is documented in the commit message. Both tests and SCM you can start with on any stage eg. even on legacy code. Another strategy would be to move some functionality into a micro service. For example instead implement new feature X, make it a standalone service that not only is decoupled from the monolith, it should not even share the same db, it should be completely standalone.
- EliRivers 7y ago
  While we do use some of those methods, sadly, we are not in possession of much of the hardware that our software runs. Making it impossible for us to fully test ourselves; we can usually bank on a period of testing on a customer site during installation, but after that, we're out of luck. Often the hardware claims to support some protocol, but we've found that really, that means "a variation on the theme of a given protocol" so ensuring that we still meet the protocol isn't quite a guarantee of success. Customers are often sympathetic to the idea that their hardware doesn't quite work as advertised - but not very sympathetic. There is often literally no way for us to know that we've broken it until a customer upgrades and discovers it no longer runs their hardware as it used to. Which does happen.
  - elcritch 7y ago
    Can you record the actual machine protocol? Then use that recorded data in regression/integration future tests.
BurningFrog 7y ago
> Pieces are gradually being documented as we work out what they do Nothing wrong with that, but it's far more important to write TESTS for the code as you understand it. Once you have tests for a subsystem, you can rewrite, refactor and/or replace it with confidence!
Consultant32452 7y ago
At my shop some guy created a new ui framework. Three years later he's still there only one who can use it. It looks pretty, so it's impossible to convince management that it's done more harm than good.

tracer4201 7y ago

Engineers are often ready to scrap things and start over, but sometimes it’s worth having a couple senior/principal/staff level folks do a quick evaluation and figure out if you’re needlessly duct taping an unmanageable piece of service/application/whatever abstraction and actually scrapping it might be the better option. From my own experience, you don’t actually know how the existing system works, the expected business rules were never documented, etc, and so even starting over isn’t really a less ambiguous path.

EliRivers 7y ago
so even starting over isn’t really a less ambiguous path. Sure isn't. We don't know what the software does right now, but there are customers all over the world using it and paying tens of thousands per year for support contracts; since we don't know quite what it does, we don't know quite what they're doing with it. We can pretty much guarantee that if we started over, we would miss a great many requirements (and a great many bugs that happen to work for various specific customers) and a lot of customers would be very unhappy. We don't know what it does, they don't know what it does, but it's working for them at the moment. Oh Gods, even the unit tests. There are some. Sometimes one fails. We stare at it. It was expecting this value to be 3, now the value is 4. Why was it expected to be 3? Is it wrong that it's now 4, or have we changed the behaviour such that the correct answer is now 4 and we need to change the test? Undocumented unit tests are worthless. I'm not asking for a lot of documentation - just commentary in the code next to them would be fine - but if I can't see what's actually being tested and how that "correct" value was ascertained, when it fails, I don't know if the bug is in the code or the test. Useless unit test.
- ownagefool 7y ago
  So you probably want to log the requests, figure out what people are doing.
  - EliRivers 7y ago
    There are no requests to log. It's not a web app. It's installed on at least one and up to many networked windows machines on a given customer site, where that network is isolated and contains numerous IP and serial cable connected pieces of specialist hardware (no two customers have the same sets of hardware); we make them dance together. There is no interaction outside that network.
    - ownagefool 7y ago
      Fair enough, that does make the problem harder.

dang 7y ago

Peter Naur's classic 1985 essay "Programming as Theory Building" argues that this is because a program is not its source code. A program is a shared mental construct (he uses the word theory) that lives in the minds of the people who work on it. If you lose the people, you lose the program. The code is merely a written representation of the program, and it's lossy, so you can't reconstruct a program from its code. https://news.ycombinator.com/item?id=10833278 https://news.ycombinator.com/item?id=10833278 https://news.ycombinator.com/item?id=7491661 https://news.ycombinator.com/item?id=7491661

Waterluvian 7y ago
Thanks for sharing this. I've been looking for this for a long time to replace my own poor attempt at saying the same: The code you write isn't the hard part. It's not the interesting part. And it's not really why you're paid. What I'm trying to imbue into you is the discipline to design and document and plan for the future. Saying you can "whip up a script this afternoon" is not impressive or commendable. It does harm without the other pieces.
- closeparen 7y ago
  Here's the thing: when I see people spend all their time on design discussions and documents, their work isn't any better for it. Usually it's worse. The documents are not clear or helpful. The designs encoded therein are not better than the first pass knee-jerk thought. The points raised in reviews are just bikeshedding. The process encourages explosion of complexity, making mountains of molehills, and sometimes can be a refuge for people who don't actually have the skills to hide out in. I do very much respect when someone can plop down some working code this afternoon vs. organize an 8-person kickoff meeting next week. If it's a genuinely large project, the working code is a skeleton POC, and we can run a design process using what we learned from it. But most of the time it's not, and that engineer has saved us collectively hundreds of hours of waste talking to death something that just isn't that hard. I think one of the critical senior engineer skills is having correct intuitions about how much time and energy should be sunk on a particular task. Sometimes that means asking people to slow down, collaborate, and think. Equally often, it means asking them to put down the calendar invites, make a choice, and bang something out. We have this bizarre cost model where code is expensive and meetings are free; nothing could be further from the truth.

tus88 7y ago

Legacy code from 2009. I'm getting old.

EliRivers 7y ago
Admittedly playing around with the definition of "legacy code" (and I get what you meant - I'm just playing with the words!), this particular set of code is legacy code from 2009, but I think I could argue that some of the world's legacy code was written yesterday, and there is code from decades ago that is not legacy code. This particular set, though, was legacy code pretty much the week after it was written :(
- olikas 7y ago
  I have played with this idea recently lately. I consider almost all code out there as legacy. I was talking about it at a conference also explaining how I like working with legacy code. (https://youtu.be/n0XCMHrSbwc https://youtu.be/n0XCMHrSbwc) I think I’m the odd one who actually like working with legacy code.
  - repsilat 7y ago
    > I think I’m the odd one who actually like working with legacy code. I had a fun bit of legacy code wrangling a month or so back. I looked at production profiles of some very compute-heavy jobs (we have great tools for that) and found we were spending ~half of my pay on cloud compute in this one library function -- deep equality checks for a kind of object. The equality-checking code itself is very slow, significantly slower than hand-written alternatives, but the objects are complicated (though luckily tree-like in structure) so hand-writing equality comparison functions would be a nightmare. Many of these structures had "id" fields, though. Gasp, could we just compare ids instead of doing deep equality checks? So I asked around, and nobody knew. Nobody even knew what the code was for. It'd be trivial to replace the deep equality checks with id comparisons, but we can't do it. Seeing how some legacy code works is one thing. Data comes in here, data goes out there. Knowing which invariants the code is trying to maintain, though, what properties it assumes about the data here or there (Sorted? Unique? Up to date? All belongs to the same user?) makes making changes difficult, though. It means you need to write defensive code that the original authors would find ridiculous, because you don't have their institutional knowledge, and can't derive it locally in the code, and can't inspect it at runtime (or be sure that prod won't send a counter-example the day after you deploy to production.)

mooreds 7y ago

How do you sell the reverse engineering? How does "the business" feel about this situation? How does the team feel about it?

rocky1138 7y ago
You don't have to sell anything. It's part of getting the job done.
- mooreds 7y ago
  In an ideal world, correct. If this is the case: > changes can take literally 20 times as long as they should while we work out what the existing code does I imagine you have to sell the extra time to (aka convince folks this is the right path) _somebody_. Maybe I've been working at the wrong places :) .
  - EliRivers 7y ago
    No reverse-engineering - no new feature or bugfix. That's all there is to it.
    - mooreds 7y ago
      Did the business have to get bit a few times (with, say production outages?) before this became the protocol?
      - EliRivers 7y ago
        "Production outages" aren't something we have to worry about; we don't run a service and once something is working on a customer site, they basically don't tinker around with the configuration or setup. They just use it. But there have certainly been projects that massively overran because the estimates assumed that pieces that already existed were understood and reliable (sometimes they were reliable; they reliably did something, we just thought they were meant to do something else). This happens less now that we assume any existing piece that we don't know about already will turn out to be a minefield.

WalterBright 7y ago

When I have some time available, but not enough time to really get into something, I just pick up a piece of code and try to improve it. Over time, I hope this leads to gradual removal of the cruft that accumulates over time.

closeparen 7y ago

There are worse things than no documentation when doing one of these excavations. Design docs that were aspirational and only partially implemented. Code comments that are incorrect, or reflect how things used to work. Reams of beautiful code and through documentation that it turns out are not the production implementation anymore because of a feature flag change or migration elsewhere. Migration trackers with open action items and deadlines 7 years in the past, creepily abandoned mid-flight like Pripyat. Detective work can go a long way, but in situations like this there's no substitute for ownership continuity and institutional memory.

zackmorris 7y ago

See Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations. - M. Conway This happened to me on a project. I spent months documenting the Byzantine labyrinth of hundreds of interwoven services that had been patched for a decade by a hundred developers. Much of it had been done under pair programming, so required pair programming or a 120 IQ just to decipher it. In the end I was squeezed out not because I was wrong, but because my method of solving longstanding issues from first principles didn't fit within the company politics of the client's organization. The gist of it is that clients mostly see short term personal productivity and not the long term gains from removing technical debt or avoiding it altogether by way of the developer's experience. Senior engineers can eventually reach a point where there isn't much for them to contribute on a project because every last issue could have been avoided through proper planning and engineering in the first place. In my case, my greatest strengths (like refactoring ability) were overlooked and my greatest weaknesses (like implementing features at high speed by repeating myself or ignoring project conventions because I don't know any better - something I can no longer do) were brought to the forefront.

repsilat 7y ago
A year ago I thought Conway's law was nonsense, but recent experience has led me to walk some of that back. I would replace "are constrained to" with "grow to", or perhaps just "often". On my first day of my current job, Mufasa (my boss) took me up on Pride Rock at sunrise, and we cast our eyes over the system diagram of the back-end supporting our product. And he told me, "We work on this box here." Our team's box gets data from that team's box, and we output data to be consumed by this other team's box... "Any what about that shadowy land?" I asked, pointing to the arrows coming from outside the diagram. "That is the Front-end. You must never go there." So yeah, Conway's law through and through. This is a big company, though, and a long-lived product. At my last job, at a startup that was ~3 years old, things were intentionally very different. This month you and a few others worked on feature X. Not on this system or that system, you were implementing something for the customer. Maybe you're a backend engineer, but you still wrote the database migration, the queries, the endpoints, the front-end logic, markup, CSS, whatever. Some of it you sucked at, but a lot of that was handled by good team composition. If another engineer on that feature knew CSS better, they mostly styled the things and you reviewed their work. Maybe you styled the next thing. I was frustrated in that job, because I didn't like writing CSS, and I wasn't good at it, and I was very good at some other things. Why couldn't I just do those things I was good at? It would have kept me a lot more productive, and saved (and made) the company a lot of money! Why did I have to do things I was bad at? Did they think engineers were interchangeable? Well, now I know. In my current job, working on five different features in my box in the system diagram, I'm super productive when pointed in the right direction with enough to work on and some autonomy. That just doesn't happen as often. We need to communicate between teams a lot more. We need to align our quarterly deliverables. We duplicate functionality on both sides of an interface, (or in each of N components), with subtly different semantics, because nobody technical is responsible for the feature. Show-stopping bugs before launch are a fact of life, because integration tests between our components suck. Generously (though for other reasons too), we're 1/3 as productive as my last gig was. Conway's law is strangling us.

techslave 7y ago

most important part is hidden in there: > with deadlines to meet the reason the company is there, employing you now, is that deadlines were met.

EliRivers 7y ago
That is also the reason that the company is now sometimes spending 100000 dollars on work that should cost 5000. A few years ago, the whole edifice was dangerously close to becoming literally unmaintainable; collapsing under its own complexity, the interest payments on the technical debt requiring more than 100% of available output. Those deadlines were frequently missed, by the way. Sometimes by months. I think not quite ever by years.

z3t4 7y ago

Could start with regression testing, each time you break something, write a test so that you do not break the same thing again. After a time you will have a decent test suit. Also write tests for all new stuff. And also write tests for everything you change. Would also be an idea to start using version control, so that each change is documented in the commit message. Both tests and SCM you can start with on any stage eg. even on legacy code. Another strategy would be to move some functionality into a micro service. For example instead implement new feature X, make it a standalone service that not only is decoupled from the monolith, it should not even share the same db, it should be completely standalone.

EliRivers 7y ago
While we do use some of those methods, sadly, we are not in possession of much of the hardware that our software runs. Making it impossible for us to fully test ourselves; we can usually bank on a period of testing on a customer site during installation, but after that, we're out of luck. Often the hardware claims to support some protocol, but we've found that really, that means "a variation on the theme of a given protocol" so ensuring that we still meet the protocol isn't quite a guarantee of success. Customers are often sympathetic to the idea that their hardware doesn't quite work as advertised - but not very sympathetic. There is often literally no way for us to know that we've broken it until a customer upgrades and discovers it no longer runs their hardware as it used to. Which does happen.
- elcritch 7y ago
  Can you record the actual machine protocol? Then use that recorded data in regression/integration future tests.

BurningFrog 7y ago

> Pieces are gradually being documented as we work out what they do Nothing wrong with that, but it's far more important to write TESTS for the code as you understand it. Once you have tests for a subsystem, you can rewrite, refactor and/or replace it with confidence!

Consultant32452 7y ago

At my shop some guy created a new ui framework. Three years later he's still there only one who can use it. It looks pretty, so it's impossible to convince management that it's done more harm than good.