9 ms·

I specialise in joining teams with problems (inability to ship, reliability issues, performance issues, etc.) At least thrice I joined teams where all original

I specialise in joining teams with problems (inability to ship, reliability issues, performance issues, etc.) At least thrice I joined teams where all original developers and the manager was long gone. The situation for some reason looked very similar in all cases. The team is complaining about supposedly bad design that is preventing them from getting work done. But the system is also large and complex and they are unable to fix/rewrite it. Lo and behold, in two out of three cases I found that the original developers actually did pretty good job designing the system but in all cases any one of new developers understands only small part of the design and is unable to contribute changes across the entire system, there is a core of the system that nobody understands even though everybody uses and the team has made changes that are against original ideas of the system quickly creating more problems and exploding complexity. In all cases developers are only able to comfortably do some kinds of changes to the system that are routinely done (like adding new fields to forms, messages, etc.) that do not change how the system functions. Changing the basic structure of the system is a taboo. In essence, developers treat "legacy" code as if it was some kind of framework or language runtime -- to be used but never to be studied to have its internals understood. (I will never understand this approach to development) Making "non-standard" changes (yes, one team used this exact term!) pretty much always ends up with something resembling long archeological expedition with ancient traps, diseases and wild animals.

pistolpete333 4y ago
As an anecdote to how developers treat "legacy" code: As a fresh developer out of college I had the mindset that frameworks were something you use and they always work and all you need is the documentation. My first job had a lot of "core" code written by another team in another office we would use with exactly 0 documentation. I was trying to get our UI to do something specific but couldn't figure out how to do it by just looking at the functions available, so I ask my manager and he immediately knows what to do, makes 2 small changes (in my defense it was pretty hacky how it was set up imo) and I get what I want. I remember being moderately pissed because I was already feeling the heat of asking my manager too many simple questions and I just apparently asked another one, so I say something along the lines of "How the hell would I ever have known how to do that, there are no docs available to say that is possible and that is not obvious at all." He just looks at me and says "You read the core code" and I remember having my mind blown at missing something so obvious for so long. After that I voraciously read almost all "framework" code I could get my hands on, I asked more intelligent questions and my relationship improved my manager and eventually came to be considered a generally good developer. Once I had this breakthrough I was seeing it in younger developers all the time, and now any time someone asks for general advice on becoming a better developer one of the things I say is "read a lot of code", and more specifically "read the code in your frameworks"
soared 4y ago
Similarly I’ve seen sales and business teams start using/selling the product in a way it wasn’t intended, and then difficulties arise when the product can’t deliver what it’s been sold to do. My currently company only did a specific niche in digital marketing but for sales goals needed to sell outside of that niche. Unsurprisingly, the product didn’t work for those use cases.
jacobsenscott 4y ago
> as much as possible, do not allow custom patterns/structures This is key. Most engineers want to build frameworks instead of applications. But leave the frameworks to the framework builders unless you were hired to build a framework. I can jump into a 10 year old rails code base and be productive right away if they stuck to the framework defaults. But invariably someone read some blog posts about how "rails doesn't scale" and built some rats nest of service objects/decorators/portsAndAdapters wtfs and it takes some serious code spelunking to figure out what's going on (hint - the end result is always creating or updating a database record, or making a single api call). I would argue microservices also encourage bitrot. You can leave some service running on some ancient libraries for years it doesn't need changes. But if you have a monolith where you constantly keep the dependencies up to date - everything stays up to date.
- twawaaay 4y ago
  I am sorry I have quickly deleted that part with advices while you must have been responding to it. But this in my experience is truly important -- from the very start plan how a new member of the team is going to get really proficient at the system. Ideally, they should be able to immediately recognise/reuse patterns/idioms they already know. People bitch and moan about boilerplate of Java applications. But when I join the project and I see controllers, models, repositories, entities, services, etc. I am pretty much at home as I know what all those things are doing and how they are connected to each other. Whenever developers decide to run with their intellectual desires it ends badly. I try to explain to people that typical business backends applications do not have custom problems. Most of those REST services are just supposed to receive query, run a piece of business logic, translate it into database call and then map response into specified format.
  - dan-robertson 4y ago
    Maybe we work on wildly different things, but I feel like this is looking at things from too low a level. That is, I think uncommon or unfamiliar patterns are much less of an issue than no one knowing the whole critical path of a system or undocumented invariants, especially between systems. Eg maybe you must restart certain things in a certain order after making Config changes. Maybe there are times when restarts are/aren’t safe. Maybe you need to have a certain OS config parameter set to get good performance and new boxes won’t have it set automatically. Maybe there is an accidental circular or nearly-circular dependency between two systems and you’ll have a terrible time if you accidentally create downtime for both at the same time. Maybe you have a binary that’s been lying around and running as some corn job but no one has compiled/deployed it since 2015 and you don’t have the source code or really know what it does. Mostly I’m thinking of more generic services rather than pure ‘business logic’ systems. These are the sorts of problems that Twitter might have in the future. Some of these are problems that Facebook might have gotten to discover in their last big outage.
belfalas 4y ago
> In essence, developers treat "legacy" code as if it was some kind of framework or language runtime -- to be used but never to be studied to have its internals understood. (I will never understand this approach to development) I was baffled by this as well for 15+ years until just last year I had a key insight: most developers don’t like to read. In fact, most people don’t like to read. Most people are intrinsically lazy and since growing your skills involves hard work, they don’t do it unless pushed to do so, either by life circumstance or necessity. Time and time again people have asked me: “how do I get really good at $tech or software engineering?” I recommend a few books or send them my standard list of must-reads. In 999/1000 cases the person approaching me gives me that defeated look that I know means: “Oh. I was hoping you’d say something not involving working hard.” Most people are polite enough not to say it to my face. When folks have problems with HTTP I ask them if they’ve read RFC 2616, and if not, I offer to bring them a nicely printed and bound copy. It typically goes unread though some use it as a reference. Only once did someone follow my recommendation and read it cover to cover. Folks seems to lack a kind of basic economic perspective. Spend a week, month, something along those lines growing your skills (or even just 30 minutes reading the docs on that thing you just spent 2 hours declaring idiotic). Yes, this means probably giving up your favorite Netflix binge, video game, or drinking on weeknights in favor of study. But once these skills are internalized they pay back for a lifetime with good returns. Yet for some strange reason people prefer to keep thinking short term and so they never learn what they’re doing. It used to make me sad but I’m making peace with it.
- ornornor 4y ago
  I see where you’re coming from and this was definitely my attitude in my twenties: do the hard work to understand the underlying concepts and not just become a code monkey. But I must admit as I get older, I don’t have as much time or patience to devote to this and I’m getting to terms with getting good enough to make a living rather than really understand $new_technology. I think this is partly from getting old and not wanting to sink so much of my own time and mental energy on computer stuff, but also from realizing how little it matters to most PHBs/middle managers who basically hold your fate in their hands. I’ve become much more cynical after a decade in the industry and I’ve realized how much of my own energy and health I have wasted in trying to get a deeper understanding of programming concepts. It was personally satisfying for a while, but it doesn’t pay off for most people. Most orgs want it done fast and cheap, they don’t want squeaky wheels, PHBs think in quarters and only want the thing to not blow up until after they get promoted in a year or two. Developers who deliver that kind of low cost code fast (albeit low quality) have much more stable careers, better promotions, and get bonuses. The rest of us ruin our mental health, paint targets on our backs, and don’t get as far. But man do we know our shit.
Heyso 4y ago
> Making "non-standard" changes [...] ressembling an long archeological expedition... This. This is how I felt trying to understand legacy core code. I kind of like that though. The idea of being able to improve/rewrite a decrepit piece of code excite me. But you won't be productive, and getting a (somewhat) complete and precise understanding of what it does and doesn't, take a lot of extra time.
nice2meetu 4y ago
I joined a big tech co and dove in to a huge unwieldy code base that had gone through the process you described above. I did a lot of reading of the code, and soon became the only one that understood it and could make big architectural changes. But.. that is a dead-end job. There is no career path being the maintainer of a large legacy code-base. It was pretty good if you wanted to rest-and-vest, I could do whatever I wanted. But there were no great features, no demo's, no selling of the work. Even small changes were painful. Moved to a different team where everything is new and building from scratch and suddenly opportunities open up.
irjustin 4y ago
Late to the party but a potentially leading question if you ever see this: - Was the original team slowly replaced due to cost cutting measures and thusly the people who joined aren't compensated well enough to dive super deep into the system to understand it? At some point people need to be compensated to care is my theory. Engineers who inherit large systems are either scared or simply don't care, possibly a mixture. But compensate them enough in both salary, time, freedom to understand the code which is clearly critical in maintaining the long term viability of the system. Thoughts?
- achenet 4y ago
  mostly agree. If I feel underpaid, I'll do the strict minimum to not get fired and spend my energy looking for another job.
  - twawaaay 4y ago
    So here is one thing I differ with you on. If I feel underpaid, I will do my normal work as best as I can and start looking for another job. I always try to leave on best terms. You really don't risk anything, you actually improve your possibility of fixing the problem without leaving. And in my case it led to at least couple more opportunities. Not to mention the world seems small enough and on many occasions I joined the company only to learn I will be working in vicinity of somebody I worked with in the past.

pistolpete333 4y ago

As an anecdote to how developers treat "legacy" code: As a fresh developer out of college I had the mindset that frameworks were something you use and they always work and all you need is the documentation. My first job had a lot of "core" code written by another team in another office we would use with exactly 0 documentation. I was trying to get our UI to do something specific but couldn't figure out how to do it by just looking at the functions available, so I ask my manager and he immediately knows what to do, makes 2 small changes (in my defense it was pretty hacky how it was set up imo) and I get what I want. I remember being moderately pissed because I was already feeling the heat of asking my manager too many simple questions and I just apparently asked another one, so I say something along the lines of "How the hell would I ever have known how to do that, there are no docs available to say that is possible and that is not obvious at all." He just looks at me and says "You read the core code" and I remember having my mind blown at missing something so obvious for so long. After that I voraciously read almost all "framework" code I could get my hands on, I asked more intelligent questions and my relationship improved my manager and eventually came to be considered a generally good developer. Once I had this breakthrough I was seeing it in younger developers all the time, and now any time someone asks for general advice on becoming a better developer one of the things I say is "read a lot of code", and more specifically "read the code in your frameworks"

soared 4y ago

Similarly I’ve seen sales and business teams start using/selling the product in a way it wasn’t intended, and then difficulties arise when the product can’t deliver what it’s been sold to do. My currently company only did a specific niche in digital marketing but for sales goals needed to sell outside of that niche. Unsurprisingly, the product didn’t work for those use cases.

jacobsenscott 4y ago

> as much as possible, do not allow custom patterns/structures This is key. Most engineers want to build frameworks instead of applications. But leave the frameworks to the framework builders unless you were hired to build a framework. I can jump into a 10 year old rails code base and be productive right away if they stuck to the framework defaults. But invariably someone read some blog posts about how "rails doesn't scale" and built some rats nest of service objects/decorators/portsAndAdapters wtfs and it takes some serious code spelunking to figure out what's going on (hint - the end result is always creating or updating a database record, or making a single api call). I would argue microservices also encourage bitrot. You can leave some service running on some ancient libraries for years it doesn't need changes. But if you have a monolith where you constantly keep the dependencies up to date - everything stays up to date.

twawaaay 4y ago
I am sorry I have quickly deleted that part with advices while you must have been responding to it. But this in my experience is truly important -- from the very start plan how a new member of the team is going to get really proficient at the system. Ideally, they should be able to immediately recognise/reuse patterns/idioms they already know. People bitch and moan about boilerplate of Java applications. But when I join the project and I see controllers, models, repositories, entities, services, etc. I am pretty much at home as I know what all those things are doing and how they are connected to each other. Whenever developers decide to run with their intellectual desires it ends badly. I try to explain to people that typical business backends applications do not have custom problems. Most of those REST services are just supposed to receive query, run a piece of business logic, translate it into database call and then map response into specified format.
- dan-robertson 4y ago
  Maybe we work on wildly different things, but I feel like this is looking at things from too low a level. That is, I think uncommon or unfamiliar patterns are much less of an issue than no one knowing the whole critical path of a system or undocumented invariants, especially between systems. Eg maybe you must restart certain things in a certain order after making Config changes. Maybe there are times when restarts are/aren’t safe. Maybe you need to have a certain OS config parameter set to get good performance and new boxes won’t have it set automatically. Maybe there is an accidental circular or nearly-circular dependency between two systems and you’ll have a terrible time if you accidentally create downtime for both at the same time. Maybe you have a binary that’s been lying around and running as some corn job but no one has compiled/deployed it since 2015 and you don’t have the source code or really know what it does. Mostly I’m thinking of more generic services rather than pure ‘business logic’ systems. These are the sorts of problems that Twitter might have in the future. Some of these are problems that Facebook might have gotten to discover in their last big outage.

belfalas 4y ago

> In essence, developers treat "legacy" code as if it was some kind of framework or language runtime -- to be used but never to be studied to have its internals understood. (I will never understand this approach to development) I was baffled by this as well for 15+ years until just last year I had a key insight: most developers don’t like to read. In fact, most people don’t like to read. Most people are intrinsically lazy and since growing your skills involves hard work, they don’t do it unless pushed to do so, either by life circumstance or necessity. Time and time again people have asked me: “how do I get really good at $tech or software engineering?” I recommend a few books or send them my standard list of must-reads. In 999/1000 cases the person approaching me gives me that defeated look that I know means: “Oh. I was hoping you’d say something not involving working hard.” Most people are polite enough not to say it to my face. When folks have problems with HTTP I ask them if they’ve read RFC 2616, and if not, I offer to bring them a nicely printed and bound copy. It typically goes unread though some use it as a reference. Only once did someone follow my recommendation and read it cover to cover. Folks seems to lack a kind of basic economic perspective. Spend a week, month, something along those lines growing your skills (or even just 30 minutes reading the docs on that thing you just spent 2 hours declaring idiotic). Yes, this means probably giving up your favorite Netflix binge, video game, or drinking on weeknights in favor of study. But once these skills are internalized they pay back for a lifetime with good returns. Yet for some strange reason people prefer to keep thinking short term and so they never learn what they’re doing. It used to make me sad but I’m making peace with it.

ornornor 4y ago
I see where you’re coming from and this was definitely my attitude in my twenties: do the hard work to understand the underlying concepts and not just become a code monkey. But I must admit as I get older, I don’t have as much time or patience to devote to this and I’m getting to terms with getting good enough to make a living rather than really understand $new_technology. I think this is partly from getting old and not wanting to sink so much of my own time and mental energy on computer stuff, but also from realizing how little it matters to most PHBs/middle managers who basically hold your fate in their hands. I’ve become much more cynical after a decade in the industry and I’ve realized how much of my own energy and health I have wasted in trying to get a deeper understanding of programming concepts. It was personally satisfying for a while, but it doesn’t pay off for most people. Most orgs want it done fast and cheap, they don’t want squeaky wheels, PHBs think in quarters and only want the thing to not blow up until after they get promoted in a year or two. Developers who deliver that kind of low cost code fast (albeit low quality) have much more stable careers, better promotions, and get bonuses. The rest of us ruin our mental health, paint targets on our backs, and don’t get as far. But man do we know our shit.

Heyso 4y ago

> Making "non-standard" changes [...] ressembling an long archeological expedition... This. This is how I felt trying to understand legacy core code. I kind of like that though. The idea of being able to improve/rewrite a decrepit piece of code excite me. But you won't be productive, and getting a (somewhat) complete and precise understanding of what it does and doesn't, take a lot of extra time.

nice2meetu 4y ago

I joined a big tech co and dove in to a huge unwieldy code base that had gone through the process you described above. I did a lot of reading of the code, and soon became the only one that understood it and could make big architectural changes. But.. that is a dead-end job. There is no career path being the maintainer of a large legacy code-base. It was pretty good if you wanted to rest-and-vest, I could do whatever I wanted. But there were no great features, no demo's, no selling of the work. Even small changes were painful. Moved to a different team where everything is new and building from scratch and suddenly opportunities open up.

irjustin 4y ago

Late to the party but a potentially leading question if you ever see this: - Was the original team slowly replaced due to cost cutting measures and thusly the people who joined aren't compensated well enough to dive super deep into the system to understand it? At some point people need to be compensated to care is my theory. Engineers who inherit large systems are either scared or simply don't care, possibly a mixture. But compensate them enough in both salary, time, freedom to understand the code which is clearly critical in maintaining the long term viability of the system. Thoughts?

achenet 4y ago
mostly agree. If I feel underpaid, I'll do the strict minimum to not get fired and spend my energy looking for another job.
- twawaaay 4y ago
  So here is one thing I differ with you on. If I feel underpaid, I will do my normal work as best as I can and start looking for another job. I always try to leave on best terms. You really don't risk anything, you actually improve your possibility of fixing the problem without leaving. And in my case it led to at least couple more opportunities. Not to mention the world seems small enough and on many occasions I joined the company only to learn I will be working in vicinity of somebody I worked with in the past.