5 ms·

I'm going to be contrary and say "that's not possible". At least, it's impossible to maintain feature velocity, even if you maintain development velocity. The

I'm going to be contrary and say "that's not possible". At least, it's impossible to maintain feature velocity, even if you maintain development velocity. The reason is simple, in the beginning you're starting with no legacy code. Features are simple to add, because there is no history to work around. However, even a month in, features are going to start blending together, and new features will have to work around existing features. Your databases will get new tables, and the existing tables will get more columns. The response code is going to fork as the API versions increment. The API space is going to grow. Technology which handled 1-1000 requests per minute will fall over when they start getting hit with 10,000 requests per minute, and require optimizations, indexes, caches, and other complexities in the stack which aren't new features. Startups are fun to work for early on, when complications like these don't exist. It requires more than "the best engineers" to keep up development velocity, let alone feature velocity.

hinkley 4y ago
If you treat performance as a feature, then you can do a lot better by this yardstick than most. The problem is that so many people don’t know what that feels like that they fight you on it. I’ve heard stories about managers trying to dismantle processes I “forced” onto people only to find out they didn’t want to give them up. They just didn’t understand why I was so vehement that we have them, until they had them. Most people assume that suffering is the natural order of things instead of the default state. So it doesn’t really track for them that choosing your form of suffering today can limit the amount you’re doing later.
mescortes 4y ago
I completely agree that it is not possible to maintain the same velocity. I have always optimized to decrease velocity as little as possible as you scale the team. That you really want to avoid is doubling your team and your velocity dropping by 50% or more. Ideally you double you team and velocity decreases by 10-15%.
JJMcJ 4y ago
To use Frederick Brooks's term, the optimizations, the special cases, the interactions with other parts of the system, start to become essential complexities as the system grows. The skill of your developers is all that stands between the code base and the total chaos of insurmountable technical debt.
- hinkley 4y ago
  Only to the extent that presence of mind and ethics are skills instead of something you are. People don’t clever their way out of these holes. That requires wisdom, not cleverness.
narag 4y ago
They started with "best practices" that maybe slowed them down at the start, but surely made the scaling smoother. Of course scale will have some effect, but the recommendations seem very sensible like keeping teams small and independent.
jahewson 4y ago
Most of the time, yeah that's how it plays out; it's easy to start from a blank slate. But where I differ is that I'd claim that most early startups don't even have much velocity. They don't actually move with speed, but with haste. All possible corners are cut. As the team scales, the debt comes due and forward progress stalls. What strikes me about this article is the forethought that was put into the engineering from the start - this is a company that really does have velocity and wants to maintain as much of it as possible.
- int0x2e 4y ago
  One thing I think can really help is to have two different modes within the same team. You move fast and cut all corners and rapidly iterate when you have an idea by building a prototype, with the only goal being maximizing the speed at which you're learning if/what works and what does. Then you basically throw away all that prototype code and build the feature properly with careful attention to code quality and complexity, testing (++) - since now you actually know what you're building and that it's likely a good idea. Rinse and repeat, so that you only have prototype code where you're actively learning, to be immediately replaced by properly engineered solutions or wiped away (with the same haste you used to create it initially - no "rarely used, badly engineered" features should stay in your codebase due to priority / resources / attention constraints). The biggest challenge I've had with this approach has been building and fostering the trust and culture needed to have everyone truly go "all in" for both aspects/modes of working. This breaks down quickly if you have managers or PMs drive engineering to keep the prototype code in for longer, or if engineering insists on not cutting corners during the exploration/prototyping phase...
  - jasondigitized 4y ago
    From https://www.stickyminds.com/aricle/hudsons-bay-start https://www.stickyminds.com/aricle/hudsons-bay-start England's King Charles II chartered the Hudson's Bay Company in 1670. It is considered by many to be the world's oldest continuing commercial enterprise. The company outfitted fur traders for expeditions out of Hudson Bay in Canada. In this part of Canada, referred to as "north of summer," having the right supplies in the correct amounts was literally a matter of survival. And, of course, the company's survival also depended on productive—and completed—expeditions. So the company provisioned the expedition's canoes with the necessary supplies for the trip. Then they sent the expedition team a short distance in their canoes to camp overnight. This was a test. The first camp was merely a "pull out," commonly called for many years a "Hudson's Bay Start" (HBS). Said Sir Samuel Benfield Steele in 1874, "[This was] very necessary so that before finally launching into the unknown one could see that nothing has been forgotten, or that if one had taken too much, being so near to the base, the mistake could be easily corrected." This risk reduction technique was not free. It cost the team precious time in a short trading season. They had shipping deadlines to meet. I can imagine them saying "Yes, but we really have to be on our way or we'll miss our shipping schedule."
chrismarlow9 4y ago
If companies took the concept of an 'error budget' (every SRE principle seems to be important to companies except this one) seriously and saw it as a signal instead of an annoyance there would be some ebb and flow in this realm instead of a continued compounding increase where stability sits on the backs of a few people with the tribal knowledge. Just my 2 cents.
- groby_b 4y ago
  Say more, please. How does an error budget account for a lack of experts in the stability area? I can't seem to be able to make that connection, and I'd love to learn.
  - bigbluedots 4y ago
    I'm guessing the suggestion is: once the error budget is spent, no new feature work is accepted until causes of the errors are fixed
    - kubanczyk 4y ago
      "Fixing the error" is at times subjective. A less nuanced approach is to freeze non-reliability-improving changes (i.e. merges) until the production meets SLO again. That is the canonical example policy given in Google's SRE Book.
    - groby_b 4y ago
      Sure, that's solid policy, but I'm not sure it really addresses the "tribal knowledge" issue per se. I mean, I'd like to think that fixing reliability issues increases knowledge, but decades on multithreaded codebases littered with "sleep(0); // just in case" point in a different direction. All I've found to work so far is a deliberate attempt to shift to more active knowledge sharing. I was hoping to learn a few new tricks from OP, but "reliability freezes" are not it, based on my experience.
shadowtree 4y ago
When we started in 2009, we did 3 major releases a year (enterprise software, deep vertical SaaS). We now do monthly releases into production, with 80+% marketshare and 500mil rev just in my product line. Scaling is hard, but doable. Takes a lot of willpower and fighting against gravity.
twblalock 4y ago
I agree that you can't maintain the same velocity you started with as the company grows, but there are definitely ways to keep as much of it as you can. Some companies slow down a lot more than others. In my experience, it ultimately comes down to SDLC. Teams need to be able to work (mostly) independently of each other. This means being strict about using API boundaries rather than sharing database tables between multiple apps, and avoiding breaking API changes that force you to deploy software written by multiple teams in lockstep. That is the real reason microservices are valuable: they decouple teams.
ignoramous 4y ago
> Startups are fun to work for early on, when complications like these don't exist. It requires more than "the best engineers" to keep up development velocity, let alone feature velocity. This blog post (which is pretty high quality for a content-marketing piece) talks about exactly these issues: https://archive.is/FQKJH https://archive.is/FQKJH (athenian.com), Scaling Your Team From 5 to 250 Engineers.

hinkley 4y ago

If you treat performance as a feature, then you can do a lot better by this yardstick than most. The problem is that so many people don’t know what that feels like that they fight you on it. I’ve heard stories about managers trying to dismantle processes I “forced” onto people only to find out they didn’t want to give them up. They just didn’t understand why I was so vehement that we have them, until they had them. Most people assume that suffering is the natural order of things instead of the default state. So it doesn’t really track for them that choosing your form of suffering today can limit the amount you’re doing later.

mescortes 4y ago

I completely agree that it is not possible to maintain the same velocity. I have always optimized to decrease velocity as little as possible as you scale the team. That you really want to avoid is doubling your team and your velocity dropping by 50% or more. Ideally you double you team and velocity decreases by 10-15%.

JJMcJ 4y ago

To use Frederick Brooks's term, the optimizations, the special cases, the interactions with other parts of the system, start to become essential complexities as the system grows. The skill of your developers is all that stands between the code base and the total chaos of insurmountable technical debt.

hinkley 4y ago
Only to the extent that presence of mind and ethics are skills instead of something you are. People don’t clever their way out of these holes. That requires wisdom, not cleverness.

narag 4y ago

They started with "best practices" that maybe slowed them down at the start, but surely made the scaling smoother. Of course scale will have some effect, but the recommendations seem very sensible like keeping teams small and independent.

jahewson 4y ago

Most of the time, yeah that's how it plays out; it's easy to start from a blank slate. But where I differ is that I'd claim that most early startups don't even have much velocity. They don't actually move with speed, but with haste. All possible corners are cut. As the team scales, the debt comes due and forward progress stalls. What strikes me about this article is the forethought that was put into the engineering from the start - this is a company that really does have velocity and wants to maintain as much of it as possible.

int0x2e 4y ago
One thing I think can really help is to have two different modes within the same team. You move fast and cut all corners and rapidly iterate when you have an idea by building a prototype, with the only goal being maximizing the speed at which you're learning if/what works and what does. Then you basically throw away all that prototype code and build the feature properly with careful attention to code quality and complexity, testing (++) - since now you actually know what you're building and that it's likely a good idea. Rinse and repeat, so that you only have prototype code where you're actively learning, to be immediately replaced by properly engineered solutions or wiped away (with the same haste you used to create it initially - no "rarely used, badly engineered" features should stay in your codebase due to priority / resources / attention constraints). The biggest challenge I've had with this approach has been building and fostering the trust and culture needed to have everyone truly go "all in" for both aspects/modes of working. This breaks down quickly if you have managers or PMs drive engineering to keep the prototype code in for longer, or if engineering insists on not cutting corners during the exploration/prototyping phase...
- jasondigitized 4y ago
  From https://www.stickyminds.com/aricle/hudsons-bay-start https://www.stickyminds.com/aricle/hudsons-bay-start England's King Charles II chartered the Hudson's Bay Company in 1670. It is considered by many to be the world's oldest continuing commercial enterprise. The company outfitted fur traders for expeditions out of Hudson Bay in Canada. In this part of Canada, referred to as "north of summer," having the right supplies in the correct amounts was literally a matter of survival. And, of course, the company's survival also depended on productive—and completed—expeditions. So the company provisioned the expedition's canoes with the necessary supplies for the trip. Then they sent the expedition team a short distance in their canoes to camp overnight. This was a test. The first camp was merely a "pull out," commonly called for many years a "Hudson's Bay Start" (HBS). Said Sir Samuel Benfield Steele in 1874, "[This was] very necessary so that before finally launching into the unknown one could see that nothing has been forgotten, or that if one had taken too much, being so near to the base, the mistake could be easily corrected." This risk reduction technique was not free. It cost the team precious time in a short trading season. They had shipping deadlines to meet. I can imagine them saying "Yes, but we really have to be on our way or we'll miss our shipping schedule."

chrismarlow9 4y ago

If companies took the concept of an 'error budget' (every SRE principle seems to be important to companies except this one) seriously and saw it as a signal instead of an annoyance there would be some ebb and flow in this realm instead of a continued compounding increase where stability sits on the backs of a few people with the tribal knowledge. Just my 2 cents.

groby_b 4y ago
Say more, please. How does an error budget account for a lack of experts in the stability area? I can't seem to be able to make that connection, and I'd love to learn.
- bigbluedots 4y ago
  I'm guessing the suggestion is: once the error budget is spent, no new feature work is accepted until causes of the errors are fixed
  - kubanczyk 4y ago
    "Fixing the error" is at times subjective. A less nuanced approach is to freeze non-reliability-improving changes (i.e. merges) until the production meets SLO again. That is the canonical example policy given in Google's SRE Book.
  - groby_b 4y ago
    Sure, that's solid policy, but I'm not sure it really addresses the "tribal knowledge" issue per se. I mean, I'd like to think that fixing reliability issues increases knowledge, but decades on multithreaded codebases littered with "sleep(0); // just in case" point in a different direction. All I've found to work so far is a deliberate attempt to shift to more active knowledge sharing. I was hoping to learn a few new tricks from OP, but "reliability freezes" are not it, based on my experience.

shadowtree 4y ago

When we started in 2009, we did 3 major releases a year (enterprise software, deep vertical SaaS). We now do monthly releases into production, with 80+% marketshare and 500mil rev just in my product line. Scaling is hard, but doable. Takes a lot of willpower and fighting against gravity.

twblalock 4y ago

I agree that you can't maintain the same velocity you started with as the company grows, but there are definitely ways to keep as much of it as you can. Some companies slow down a lot more than others. In my experience, it ultimately comes down to SDLC. Teams need to be able to work (mostly) independently of each other. This means being strict about using API boundaries rather than sharing database tables between multiple apps, and avoiding breaking API changes that force you to deploy software written by multiple teams in lockstep. That is the real reason microservices are valuable: they decouple teams.

ignoramous 4y ago

> Startups are fun to work for early on, when complications like these don't exist. It requires more than "the best engineers" to keep up development velocity, let alone feature velocity. This blog post (which is pretty high quality for a content-marketing piece) talks about exactly these issues: https://archive.is/FQKJH https://archive.is/FQKJH (athenian.com), Scaling Your Team From 5 to 250 Engineers.