7 ms·

I work at a company that grew from a few hundred to a few thousand people in a few years' time, and there was an identical mindset and identical problems over t

I work at a company that grew from a few hundred to a few thousand people in a few years' time, and there was an identical mindset and identical problems over that growth period. I started during that growth period, and 10 years later, we're still digging ourselves out of those mistakes. One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity. I think there's an argument that this could be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now. I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before. EDIT: Clarified a few points.

jamil7 6y ago
Apart from the one catastrophic failure you mention it sounds like, from a business perspective this worked? You were able to extend your tech debt long enough to start generating money and are now in a position to pay it back. From an engineering perspective I agree it's a grind to pay all this back and lose so much velocity. Are their business strategies that favour slower more careful engineering practises? mission-critical systems come to mind but maybe also nth-movers could be put in there? Where you know what you're building and competing with but want to make the best version of X.
- mlyle 6y ago
  I think that they're saying, in retrospect, they could have started making some payments on the debt a bit earlier when the business had scale and sustainability. And that in turn this would have avoided a lot of unpleasantness. It's hard to change culture; you tend to keep doing what worked before even if it's not optimal anymore. Yesterday's adaptive behavior is today's unfortunate practice.
  - jamil7 6y ago
    Yeah OK that makes sense.
- montagg 6y ago
  Well, yes, I am saying from a business perspective this worked. but I also think we got lucky. Around five years ago, a little before the high profile failure, we were starting to lose marketshare, and though we've been able to claw most of it back, I think it's far more due to missteps of our competitors than our own performance. Had we invested in more sustainable development earlier, I think we would have been able to move faster to take advantage of new opportunities, diversify our revenue streams earlier (adding to them now is hamstrung in large part because of tech debt), and head off competitors faster. In fact, once we started investing in that kind of development, we did do that with a successful new feature that we developed much more quickly than we could have in the past, and it's because of that success that leaders are finally starting to prioritize building standardized and reusable infrastructure. EDIT: I think another way to say this is, once we knew our strengths and how we'd be successful in the future, we should have operationalized that and hardened it, so we'd have a sustainable advantage over competitors. Instead, we kept the "move fast, be flexible, and throw man-hours at problems" mentality too long, and it put us at risk.
  - OliverGilan 6y ago
    Could there be a hybrid approach? Have teams that focus on being a first mover and shipping and then once there's a product on the market, you have a secondary team focusing on fixing the tech debt early while the other team continues to iterate
macjohnmcc 6y ago
I worked at a small company that was always chasing the next potential sale. Potential customer mentions feature A and all of a sudden all development shifts towards developing that feature. The vast majority of the time the potential customer never became a customer. In the meantime competition focused on their own plan of features and our product wound up with features no one ever wanted and was missing critical functionality everybody demanded.
- drspacemonkey 6y ago
  I've had the exact same experience. Throw in a large dose of "no repeat customers" due to us prioritizing Feature A for Potential Customer B over fixing existing gaps that affected actual customers, and it was a never-ending parade of haphazardly created, bug-riddled new features that never got improved. Made every single day a death march and a company that survives purely on the sunk cost fallacy on the part of the investors. They've already dropped $MILLIONS into building this thing, and the CEO swears that this next big deal will be what finally pushes the company into the stratosphere. As miserable as it was to work there, I'm honestly impressed they're able to keep that zombie moving.
  - ryandrake 6y ago
    Yup, and the requesting "customer" always convinces your CEO with the same argument: "Just add this feature, sell it to us, then it's part of your product forever and you can sell it to many other customers! We're paying for your NRE!" and CEO falls for it!
    - macjohnmcc 6y ago
      The company I worked for didn't even need that incentive just the potential to make a sale! We added features just to complete bullet points on a marketing sheet even if we never made a sale of the feature.
jorblumesea 6y ago
The core issue, from what I've seen, is one of incentives. Many leaders at many companies are compensated on growth and short term factors. It's hard to find a leader that is willing to take a 5 year look, or even find a leader that is planning on being around 5+ years. The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out. I just can't see many leaders thinking 5 years out.
- coredog64 6y ago
  I would add that when it’s not a revolving door it’s frequently someone who used to be technical and still fancies themselves as an elite coder. That in turn leads to low level decisions (how and not what) being made at the top. As a result, you’re chasing “cool” rather than “best long term decision” in service of the ego of someone who is already being appropriately compensated monetarily.
satyrnein 6y ago
My previous employer prioritized shipping, won the market while having both more features and more technical debt than competitors, bought those competitors out as they started failing, and then migrated their clients over and threw their beautiful (but less featured) systems right into the garbage. To this day, I sometimes think back to how a particular competitor had the right schema for solving current problems, but they didn't even survive to the current day, which takes precedence.
RandoHolmes 6y ago
> One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced. In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain. An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.

jamil7 6y ago

Apart from the one catastrophic failure you mention it sounds like, from a business perspective this worked? You were able to extend your tech debt long enough to start generating money and are now in a position to pay it back. From an engineering perspective I agree it's a grind to pay all this back and lose so much velocity. Are their business strategies that favour slower more careful engineering practises? mission-critical systems come to mind but maybe also nth-movers could be put in there? Where you know what you're building and competing with but want to make the best version of X.

mlyle 6y ago
I think that they're saying, in retrospect, they could have started making some payments on the debt a bit earlier when the business had scale and sustainability. And that in turn this would have avoided a lot of unpleasantness. It's hard to change culture; you tend to keep doing what worked before even if it's not optimal anymore. Yesterday's adaptive behavior is today's unfortunate practice.
- jamil7 6y ago
  Yeah OK that makes sense.
montagg 6y ago
Well, yes, I am saying from a business perspective this worked. but I also think we got lucky. Around five years ago, a little before the high profile failure, we were starting to lose marketshare, and though we've been able to claw most of it back, I think it's far more due to missteps of our competitors than our own performance. Had we invested in more sustainable development earlier, I think we would have been able to move faster to take advantage of new opportunities, diversify our revenue streams earlier (adding to them now is hamstrung in large part because of tech debt), and head off competitors faster. In fact, once we started investing in that kind of development, we did do that with a successful new feature that we developed much more quickly than we could have in the past, and it's because of that success that leaders are finally starting to prioritize building standardized and reusable infrastructure. EDIT: I think another way to say this is, once we knew our strengths and how we'd be successful in the future, we should have operationalized that and hardened it, so we'd have a sustainable advantage over competitors. Instead, we kept the "move fast, be flexible, and throw man-hours at problems" mentality too long, and it put us at risk.
- OliverGilan 6y ago
  Could there be a hybrid approach? Have teams that focus on being a first mover and shipping and then once there's a product on the market, you have a secondary team focusing on fixing the tech debt early while the other team continues to iterate

macjohnmcc 6y ago

I worked at a small company that was always chasing the next potential sale. Potential customer mentions feature A and all of a sudden all development shifts towards developing that feature. The vast majority of the time the potential customer never became a customer. In the meantime competition focused on their own plan of features and our product wound up with features no one ever wanted and was missing critical functionality everybody demanded.

drspacemonkey 6y ago
I've had the exact same experience. Throw in a large dose of "no repeat customers" due to us prioritizing Feature A for Potential Customer B over fixing existing gaps that affected actual customers, and it was a never-ending parade of haphazardly created, bug-riddled new features that never got improved. Made every single day a death march and a company that survives purely on the sunk cost fallacy on the part of the investors. They've already dropped $MILLIONS into building this thing, and the CEO swears that this next big deal will be what finally pushes the company into the stratosphere. As miserable as it was to work there, I'm honestly impressed they're able to keep that zombie moving.
- ryandrake 6y ago
  Yup, and the requesting "customer" always convinces your CEO with the same argument: "Just add this feature, sell it to us, then it's part of your product forever and you can sell it to many other customers! We're paying for your NRE!" and CEO falls for it!
  - macjohnmcc 6y ago
    The company I worked for didn't even need that incentive just the potential to make a sale! We added features just to complete bullet points on a marketing sheet even if we never made a sale of the feature.

jorblumesea 6y ago

The core issue, from what I've seen, is one of incentives. Many leaders at many companies are compensated on growth and short term factors. It's hard to find a leader that is willing to take a 5 year look, or even find a leader that is planning on being around 5+ years. The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out. I just can't see many leaders thinking 5 years out.

coredog64 6y ago
I would add that when it’s not a revolving door it’s frequently someone who used to be technical and still fancies themselves as an elite coder. That in turn leads to low level decisions (how and not what) being made at the top. As a result, you’re chasing “cool” rather than “best long term decision” in service of the ego of someone who is already being appropriately compensated monetarily.

satyrnein 6y ago

My previous employer prioritized shipping, won the market while having both more features and more technical debt than competitors, bought those competitors out as they started failing, and then migrated their clients over and threw their beautiful (but less featured) systems right into the garbage. To this day, I sometimes think back to how a particular competitor had the right schema for solving current problems, but they didn't even survive to the current day, which takes precedence.

RandoHolmes 6y ago

> One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced. In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain. An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.