5 ms·

> Are there any other products with such a small team and a huge userbase? My game Nebulous was 1.5 devs (one full time one part time) and multiple millions o

> Are there any other products with such a small team and a huge userbase? My game Nebulous was 1.5 devs (one full time one part time) and multiple millions of MAU. 9.5 years later it's still going well. > when the business requires more coding and technical debt comes how do they manage it Delete bad code. Replace with good code. Sounds simple enough but in my experience at mega and mid corps, step 1 is almost never done. Whether that's because of ego or chasing local optima I'm not sure - probably a mix of both.

cbm-vic-20 2y ago
It's due to fear. Fear of breaking something that may depend on that bad code. Test automation rarely covers every possible case, and nobody wants to be on the hook when some code changes cause other stuff to break.
- shadowgovt 2y ago
  And for all of the benefits of process, I have never met a level of documentation, verification, or testing that matches the advantages of having the entire code base originating from one mind. Generally this is not tractable because it cannot scale. But there are certain applications where it scales fine.
  - mattgreenrocks 2y ago
    I don't think we need nearly as many devs as we think. We do need someone who is taking a hardline approach on limiting the amount of scope tackled at once, and then fewer devs that are downstream of that. It is hubris to think that every problem admits the same solution, namely, throw as many devs as we can at it and hope for the best. But business isn't really known for being reflective.
  - swiftcoder 2y ago
    > Generally this is not tractable because it cannot scale The real question here is scale in terms of what? Because a lot of folks are out here trying to scale people/careers, not software. It's extremely noticeable at BigCorps. Why do we need to scale this project from 3 -> 30 -> 300 developers? Because that's the number of reports to promo from Manager I -> Manager II -> Director
- mritchie712 2y ago
  it's always fear for me. I'd rather leave it commented out for a decade or so to be safe.
  - Feathercrown 2y ago
    I'm a monster, I see code that's been commented out for more than 2 months or so and nuke it unless I know it's needed. We have Git, it'll be fiiine
    - radlad 2y ago
      Right. The only time I sometimes leave a commented out line of code is if it's temporarily broken and will be uncommented soon; or if it's by-far the most obvious way to do something, but does not work for some reason, and then there's a comment above about why not above.
    - astrospective 2y ago
      I bring this up in PRs, as I know with my own experience I don’t always mean to commit it commented out, sometimes it was just for testing, or it should have been deleted after I was done refactoring.
  - devjab 2y ago
    I never understood commenting out code when you have version control. I get why people do it, I’ve done it myself and then two days later been confused which of the 3 commented out function was actually the most recent. It’s infinitely more clear from the version control since the history is there for you to zoom through.
    - block_dagger 2y ago
      I think a lot of devs prefer the hover-to-blame feature in their IDE vs searching through history on GitHub.
    - wink 2y ago
      You would find it if you knew it was there. I only leave it commented out because it has a reason (they all say that, right?). I suppose the best way would be to provide a meaningful comment "This is the place where 15 lines of coded finally found their resting place, deleted after the bug they solved was eliminated elsewhere". But in reality, I've never seen a single of the "we could find it it in git" ever actually find it in git.
- atomicnumber3 2y ago
  That, and most orgs simply do not reward or even pretend to care about these kinds of improvements. If you delete bad (but working) code, and replace it with good (and, let's assume best-case scenario - also working) code, what has actually changed for the business? Nothing. Except that in 3 years the junior dev that gets a ticket about doing something in this area will come in and not notice the code isn't a dumpster fire. Or, in 3 years, you won't notice that you didn't have to optimize this code a year ago. What they do notice is that you were insisting on working on some mumbo jumbo and ok good they're done now they can actually work on something useful. Haha aren't these devs quirky? Sometimes they take a few days and work on something weird, and all the other senior devs nod and salute solemnly and I'm too scared to ask for more details, but they don't usually take too long so let's just indulge them for a few days to keep them happy so they don't leave too. It takes a very, very deeply engineering-first org to really cultivate this intentionally. And similarly it seems like succeeding as a startup requires at least a decent amount of shipping some shit code fast so you get a series B, so usually you don't start in this posture and never shift into it before it's far too late. And also unfortunately, devs often _do_ spend time optimizing/refactoring personal pet peeves as opposed to things that might have a good chance of mattering. I once saw another senior dev spend a week optimizing string allocations on our hot path. Our owner loves people who can do this kind of stuff, so it got a lot of praise. The microbenchmarks looked great, pretty graphs. Users noticed nothing, the actual metrics we track literally did not change, and now the already-complicated hot path is decorated with some contorted string-allocation-avoiding warts here and there and the next person to go in and change the code is _definitely_ going to keep doing that pattern, for sure. Meanwhile our oauth flow is still a tortured, unloved, twisted writhing mass of pain and suffering that prints bug reports like CVS receipts. So... extraordinarily difficult to intentionally cultivate a culture that does this judiciously.
  - BubbleRings 2y ago
    You should write a book on this stuff.
  - block_dagger 2y ago
    One of the most rewarding aspects of my previous career at a company spanning from a startup to an IPO and beyond was deleting bad code and replacing it in a massive Rails app that was touched by hundreds of devs in a high churn environment. I also took on fixing massive schema inefficiencies that had a lot of risk of breaking nearly every other team's flow. It took a lot of careful work and communication across multi-year goals that I managed, mostly alone. I was allowed to do this by a few early folks who believed I was doing a good service for the company in the long run but kept hinting it was a bad career choice for me personally. I believe I was eventually let go for making these massive improvements instead of adding that green button that the new Product guy wanted. No regrets.
- klabb3 2y ago
  Can confirm. One of my proudest moments was deleting thousands of LOC of copy-paste-modify garbage. However I introduced one bug that broke another team which used an undocumented feature. It was fixed soon, but yeah, still not great. And very few people would have taken that on, I was not a career chaser. Some would say it’s the other teams fault for not adding a cross-test against my teams code. And while that would have solved it, some things are hard to test. Even in companies who have good testing standards some things are still hard-to-impossible to test. In my humble opinion tests are great if and only if they are hermetic and fast. Unfortunately, the important things that can go wrong are usually the least testable. In either case, in a non-perfect world (ie ~all large companies and most small ones) people optimize for not breaking things, and there’s a solid argument for that being a local optima, both for short term stability and career wise.
  - giantrobot 2y ago
    > In my humble opinion tests are great if and only if they are hermetic and fast. Unfortunately, the important things that can go wrong are usually the least testable. Integration tests are hard. A lot of time it's because deployment is very seat of the pants. Even with tightly managed deployment the test environment needs to be representative of the production environment. Just setting that up is time consuming and expensive. Then actually doing tests where the test environment has useful amounts of instrumentation without major performance or behavioral penalties.
dowager_dan99 2y ago
I go further and simplify: "delete code" whenever and where possible. The term "Tech Debt" is really overloaded; I think the idea that "all code is liability" is better for framing the issue and strategies.
- BubbleRings 2y ago
  It is so good to see someone say that. I don’t code anymore, but as a systems engineer on different (often troubled) projects, I started developing a bit of a specialty in deleting crufty old collections of files. Sometimes multiple terabytes in a day, directories sitting around looking like they might be important, in some random corner of storage. You have to be good at your job, good at the specialty, and more interested in doing the right thing for the company (and more irritated at the stupidity of the files being there 10 years after they were needed) than you are at looking productive to management. Management does not want to hear “well there was a directory structure of two million files that was a backup of a Linux machine from 8 years ago, I spent two days extracting the dozen files that we might need some day, getting the okay to proceed, and deleting the files.”
moritonal 2y ago
Fleet commander or I'm guessing .IO?
- grog454 2y ago
  .io. There have been more Nebulous's since I last checked :)
noprocrasted 2y ago
> Delete bad code. Replace with good code Your points are valid but there's also the issue that the more developers you have the more communication overhead there is, which makes large changes to the codebase hard/impossible. With a handful of devs you can jump on a call, brainstorm for an hour or two and come to a mutual agreement, then one can submit a several-thousand-line PR refactoring the whole thing and nobody would bat an eye. This kind of coordination is impossible in larger teams, if anything just because everyone is busy and can't afford to spend a couple hours brainstorming + subsequently get acquainted with the new code, but also because the more people the more opinions and mismatched incentives (bad or overly complex code might imply busywork which some people thrive on, so refactoring it to no longer require said busywork is a downside in their eyes).
- gregors 2y ago
  >>>> This kind of coordination is impossible in larger teams, And that's the real reason service architecture took off
Consultant32452 2y ago
I have no incentive to delete bad code and replace it with good code when doing megacorp work. For things I own, it's situational.

cbm-vic-20 2y ago

It's due to fear. Fear of breaking something that may depend on that bad code. Test automation rarely covers every possible case, and nobody wants to be on the hook when some code changes cause other stuff to break.

shadowgovt 2y ago
And for all of the benefits of process, I have never met a level of documentation, verification, or testing that matches the advantages of having the entire code base originating from one mind. Generally this is not tractable because it cannot scale. But there are certain applications where it scales fine.
- mattgreenrocks 2y ago
  I don't think we need nearly as many devs as we think. We do need someone who is taking a hardline approach on limiting the amount of scope tackled at once, and then fewer devs that are downstream of that. It is hubris to think that every problem admits the same solution, namely, throw as many devs as we can at it and hope for the best. But business isn't really known for being reflective.
- swiftcoder 2y ago
  > Generally this is not tractable because it cannot scale The real question here is scale in terms of what? Because a lot of folks are out here trying to scale people/careers, not software. It's extremely noticeable at BigCorps. Why do we need to scale this project from 3 -> 30 -> 300 developers? Because that's the number of reports to promo from Manager I -> Manager II -> Director
mritchie712 2y ago
it's always fear for me. I'd rather leave it commented out for a decade or so to be safe.
- Feathercrown 2y ago
  I'm a monster, I see code that's been commented out for more than 2 months or so and nuke it unless I know it's needed. We have Git, it'll be fiiine
  - radlad 2y ago
    Right. The only time I sometimes leave a commented out line of code is if it's temporarily broken and will be uncommented soon; or if it's by-far the most obvious way to do something, but does not work for some reason, and then there's a comment above about why not above.
  - astrospective 2y ago
    I bring this up in PRs, as I know with my own experience I don’t always mean to commit it commented out, sometimes it was just for testing, or it should have been deleted after I was done refactoring.
- devjab 2y ago
  I never understood commenting out code when you have version control. I get why people do it, I’ve done it myself and then two days later been confused which of the 3 commented out function was actually the most recent. It’s infinitely more clear from the version control since the history is there for you to zoom through.
  - block_dagger 2y ago
    I think a lot of devs prefer the hover-to-blame feature in their IDE vs searching through history on GitHub.
  - wink 2y ago
    You would find it if you knew it was there. I only leave it commented out because it has a reason (they all say that, right?). I suppose the best way would be to provide a meaningful comment "This is the place where 15 lines of coded finally found their resting place, deleted after the bug they solved was eliminated elsewhere". But in reality, I've never seen a single of the "we could find it it in git" ever actually find it in git.
atomicnumber3 2y ago
That, and most orgs simply do not reward or even pretend to care about these kinds of improvements. If you delete bad (but working) code, and replace it with good (and, let's assume best-case scenario - also working) code, what has actually changed for the business? Nothing. Except that in 3 years the junior dev that gets a ticket about doing something in this area will come in and not notice the code isn't a dumpster fire. Or, in 3 years, you won't notice that you didn't have to optimize this code a year ago. What they do notice is that you were insisting on working on some mumbo jumbo and ok good they're done now they can actually work on something useful. Haha aren't these devs quirky? Sometimes they take a few days and work on something weird, and all the other senior devs nod and salute solemnly and I'm too scared to ask for more details, but they don't usually take too long so let's just indulge them for a few days to keep them happy so they don't leave too. It takes a very, very deeply engineering-first org to really cultivate this intentionally. And similarly it seems like succeeding as a startup requires at least a decent amount of shipping some shit code fast so you get a series B, so usually you don't start in this posture and never shift into it before it's far too late. And also unfortunately, devs often _do_ spend time optimizing/refactoring personal pet peeves as opposed to things that might have a good chance of mattering. I once saw another senior dev spend a week optimizing string allocations on our hot path. Our owner loves people who can do this kind of stuff, so it got a lot of praise. The microbenchmarks looked great, pretty graphs. Users noticed nothing, the actual metrics we track literally did not change, and now the already-complicated hot path is decorated with some contorted string-allocation-avoiding warts here and there and the next person to go in and change the code is _definitely_ going to keep doing that pattern, for sure. Meanwhile our oauth flow is still a tortured, unloved, twisted writhing mass of pain and suffering that prints bug reports like CVS receipts. So... extraordinarily difficult to intentionally cultivate a culture that does this judiciously.
- BubbleRings 2y ago
  You should write a book on this stuff.
- block_dagger 2y ago
  One of the most rewarding aspects of my previous career at a company spanning from a startup to an IPO and beyond was deleting bad code and replacing it in a massive Rails app that was touched by hundreds of devs in a high churn environment. I also took on fixing massive schema inefficiencies that had a lot of risk of breaking nearly every other team's flow. It took a lot of careful work and communication across multi-year goals that I managed, mostly alone. I was allowed to do this by a few early folks who believed I was doing a good service for the company in the long run but kept hinting it was a bad career choice for me personally. I believe I was eventually let go for making these massive improvements instead of adding that green button that the new Product guy wanted. No regrets.
klabb3 2y ago
Can confirm. One of my proudest moments was deleting thousands of LOC of copy-paste-modify garbage. However I introduced one bug that broke another team which used an undocumented feature. It was fixed soon, but yeah, still not great. And very few people would have taken that on, I was not a career chaser. Some would say it’s the other teams fault for not adding a cross-test against my teams code. And while that would have solved it, some things are hard to test. Even in companies who have good testing standards some things are still hard-to-impossible to test. In my humble opinion tests are great if and only if they are hermetic and fast. Unfortunately, the important things that can go wrong are usually the least testable. In either case, in a non-perfect world (ie ~all large companies and most small ones) people optimize for not breaking things, and there’s a solid argument for that being a local optima, both for short term stability and career wise.
- giantrobot 2y ago
  > In my humble opinion tests are great if and only if they are hermetic and fast. Unfortunately, the important things that can go wrong are usually the least testable. Integration tests are hard. A lot of time it's because deployment is very seat of the pants. Even with tightly managed deployment the test environment needs to be representative of the production environment. Just setting that up is time consuming and expensive. Then actually doing tests where the test environment has useful amounts of instrumentation without major performance or behavioral penalties.

dowager_dan99 2y ago

I go further and simplify: "delete code" whenever and where possible. The term "Tech Debt" is really overloaded; I think the idea that "all code is liability" is better for framing the issue and strategies.

BubbleRings 2y ago
It is so good to see someone say that. I don’t code anymore, but as a systems engineer on different (often troubled) projects, I started developing a bit of a specialty in deleting crufty old collections of files. Sometimes multiple terabytes in a day, directories sitting around looking like they might be important, in some random corner of storage. You have to be good at your job, good at the specialty, and more interested in doing the right thing for the company (and more irritated at the stupidity of the files being there 10 years after they were needed) than you are at looking productive to management. Management does not want to hear “well there was a directory structure of two million files that was a backup of a Linux machine from 8 years ago, I spent two days extracting the dozen files that we might need some day, getting the okay to proceed, and deleting the files.”

moritonal 2y ago

Fleet commander or I'm guessing .IO?

grog454 2y ago
.io. There have been more Nebulous's since I last checked :)

noprocrasted 2y ago

> Delete bad code. Replace with good code Your points are valid but there's also the issue that the more developers you have the more communication overhead there is, which makes large changes to the codebase hard/impossible. With a handful of devs you can jump on a call, brainstorm for an hour or two and come to a mutual agreement, then one can submit a several-thousand-line PR refactoring the whole thing and nobody would bat an eye. This kind of coordination is impossible in larger teams, if anything just because everyone is busy and can't afford to spend a couple hours brainstorming + subsequently get acquainted with the new code, but also because the more people the more opinions and mismatched incentives (bad or overly complex code might imply busywork which some people thrive on, so refactoring it to no longer require said busywork is a downside in their eyes).

gregors 2y ago
>>>> This kind of coordination is impossible in larger teams, And that's the real reason service architecture took off

Consultant32452 2y ago

I have no incentive to delete bad code and replace it with good code when doing megacorp work. For things I own, it's situational.