5 ms·

Velocity vs. Impact

52 points by kiyanwang 4y ago

maximilianroos 4y ago
This makes a strawman of "Velocity" The purpose of Velocity is not to ship mediocre things quickly. It's to figure out if you're building something people want, by showing it to them and seeing if they use it. Maybe you can introspect what will have impact? Great if you can! But generally folks are overconfident in their beliefs & underconfident in their actions, and so ship far too slowly.
- ricardobeat 4y ago
  > The purpose of Velocity is not to ship mediocre things quickly It may not be the purpose, but it's the end result a lot of the time. As put by another commenter, if you aim for velocity, you get velocity. If as a consequence you never really have time to explore and gather feedback, the results of those quick experiments will never be meaningful or reliable enough to feed into the next iteration.
  - maximilianroos 4y ago
    Any advice that isn't just "Be optimal" is susceptible to this criticism! Useful advice is useful because it pushes us towards better tradeoffs. My claim is that generally folks ship too slowly. But obviously if we only care about velocity, that's bad, in the same way it's probably bad to only eat vegetables, or to work out every single day. But then the objection is not "Velocity is bad", it's that "Shipping bad things is bad". In the same way an unbalanced diet is bad, or working out when injured is bad.
Jtsummers 4y ago
What a long article to ultimately get to the realization (but fail to state it): If you pick a quantitative measure and optimize for it as a target, that's what you get. If you optimize for "velocity" (# of changes, here) then you get velocity, a lot of changes. If you optimize for "high impact" changes then you get high impact changes. This is not shocking. What also shouldn't be shocking is that if the two are only poorly correlated then you'll end up not getting meaningful improvements in the other measure, and may in fact reduce the other if there is a conflict between them. The bigger problem is that most organizations don't know what they want to optimize or how to measure the things they want to optimize. So they find proxies like "velocity" or "impact" and end up optimizing for them instead of what they actually want. In other words, Goodhart's Law.
- marcosdumay 4y ago
  It's very likely that velocity is very strongly correlated with impact, that's not exactly the problem. The thing that people mostly don't expect is that if you completely optimize for some metric, you will harm any metric that is not perfectly aligned with it. It doesn't matter if the correlation is very good, as long as it's not perfect it will be harmed.
  - Jtsummers 4y ago
    Right. I was more referring to the article's definition of velocity which was number of changes ("launching more, faster"). It's easy for that to be misaligned with anything useful. You can make a lot of small things with little "impact" or value if your goal is just the number of changes. If you have a saner definition of velocity which is based on the size or scope and not just the raw number of changes, then you can pair that with prioritization based on expected value and your measures will be better aligned.
    - marcosdumay 4y ago
      But Agile's velocity is basically that. It's a measure of how long it will take you to change course if you discover your current design is bad and you should be working on something slightly different. Again, this is very likely very correlated with impact. But you seem to want it to be a measure of productivity; it's not.
      - Jtsummers 4y ago
        I gave two definitions of velocity and you just used "that". I have no idea if we're in agreement or disagreement now. If you mean that Scrum uses sizes, then yeah. It tends to be better aligned with what you want because it uses sizes and priorities. It doesn't just count the raw number of changes as the target, even for its velocity measure (which isn't the target, but a guide, at least in theory). The target in Scrum is getting what the customer (again, in theory) wants as quickly as possible which means focusing on priorities. Velocity is there to see how well you're operating rather than being a goal. If you mean Agile (which version?) doesn't use sizes but just a raw count of changes for velocity, then we're in disagreement. I mean, there probably are idiots out there who do that. But that's not what's encouraged by anyone that cares about getting shit done. It's a useless measure especially if it becomes your target for optimization. Anyone can get high velocity if all changes count the same. Submit 500 change requests to add 500 logging statements. Wow they're fast! Also they're gaming the system.
ojbyrne 4y ago
There may be a reasonable qualitative argument to favor Velocity over Impact, but the pseudo-quantitative argument made in the article is not really useful. You could easily adjust the numbers chosen to support the exact opposite argument.
Nevermark 4y ago
For changes that can be A/B tested, the negative outcomes can be ignored since they will be jettisoned. That changes the calculus of impact from positive-negative, to just positive. A big improvement. And presumably, the positive improvements are multiplicative. (i.e 1.01^100 ~= 2.7) > (1 + 100 * (1.01 -1) = 2.0) Whether with prior research, or after the fact verification, allowing your customer to be the source of what is, or is not, a good market fit always wins. But first, do no harm!
latchkey 4y ago
> I’m here to argue that development velocity (whatever that means) and launch throughput are entirely the wrong optimizations. Development velocity is pretty well defined in agile. The fact that the author starts of with admitting no understanding of what it means and then goes on to say that it is a wrong optimization, just seems off. When I worked at Pivotal Labs, velocity was king. It was controlled entirely by the development team as we would the point stories and do the work. Velocity (points added up from a weeks worth of stories finished and then averaged over time) was tracked very closely and easy to measure. Since everything was team and pair based, there were no pointing fingers at individuals... you had to complete things together. It was a very simple and easily computed metric that really showed output and allowed for great long term project estimates. We would always hit our expected run rate because of it. PM's could tell, in advance, when features were going to land. It really made for an excellent working environment where we could get into work in the morning and leave at the end of the day, without having to ever stay late to finish expectations.
q7xvh97o2pDhNrh 4y ago
The author's fixation on "improve your success percentage" is actually a highly toxic way to shape a team culture, even if it does (supposedly) yield some benefits. A good leader should let people know it's safe to fail. That's how we create an environment for boldness. If someone launched 10 failures and 10 successes this year, a very reasonable goal would be to launch 100 failures next year. Go fail more. Go fail all the time. You know how you'll pack that many failures into a year? Well, you'll have to launch more often. If you want to launch every feature more often, you'll need to launch features earlier, when they're smaller. If you want to launch smaller features, you need to know how to scope down. To know what scope you can cut for that many features, you'll need to talk to users more often — probably all the time. Et voila. So upping your rate of failure by 10x will mean you've upped your rate of learning by (at least) 10x. That will likely take you way further than some navel-gazing and overly-negative focus on "failing less often." Success comes from getting more turns at bat. You rack up way more strikes — and way more home runs.
drewcoo 4y ago
When did "experiment" come to mean "here's some data I made up to fit my narrative?" I think there are better ways to argue for Big Design Up Front. https://en.wikipedia.org/wiki/Big_Design_Up_Front https://en.wikipedia.org/wiki/Big_Design_Up_Front
jcparkyn 4y ago
> Research of A/B experiments carried out independently by multiple companies suggests that, at best, 1 in 3 ideas will have any measurable impact, and the industry average is likely 1 in 7 (14%). Everything else either has no impact or creates negative impact Some fairly big leaps here: - There's a big gap between "measurable impact" and "no impact", which is mostly dependent on measurement accuracy. For example, driving my car to work has no measurable impact on the air quality of earth as a whole, but that doesn't mean it has no impact. - The link for "1 in 3" doesn't even seem to mention that, as far as I can find. The closest is "At Google and Bing, only about 10% to 20% of experiments generate positive results". This is pretty skewed by the fact that those companies intentionally make minor or unintuitive changes to A/B test them, but can't really be generalised to all feature work at other companies. The link for "1 in 7" is also specifically for A/B testing, so generalising that to all "ideas" is quite a stretch.
vlovich123 4y ago
One limitation of this is when you do “no impact” work in year 1 and 2 that’s high value impact work in year 3. Sure, if everything is low hanging fruit go for it. But sometimes it can take longer to build something that has 10x impact. eg teams A, B, and C hit a wall because all their time is spent fighting fires because they didn’t invest in getting ahead of tech debt / product growth. Of course, it’s hard to make the case that that’s the case and you can be very wrong in which case you waste a lot of time. But making sure that you are managing smoldering fires and preventing them from becoming infernos is high impact “invisible” work - did you prevent the inferno or was it unlikely to ever catch in the first place”. The usual approach is to let things start tipping and hope you have enough lead time between “this is starting to become a problem” and “product growth has to cease and we have to start shedding customers”

52 points by kiyanwang 4y ago

This makes a strawman of "Velocity" The purpose of Velocity is not to ship mediocre things quickly. It's to figure out if you're building something people want, by showing it to them and seeing if they use it. Maybe you can introspect what will have impact? Great if you can! But generally folks are overconfident in their beliefs & underconfident in their actions, and so ship far too slowly.

ricardobeat 4y ago
> The purpose of Velocity is not to ship mediocre things quickly It may not be the purpose, but it's the end result a lot of the time. As put by another commenter, if you aim for velocity, you get velocity. If as a consequence you never really have time to explore and gather feedback, the results of those quick experiments will never be meaningful or reliable enough to feed into the next iteration.
- maximilianroos 4y ago
  Any advice that isn't just "Be optimal" is susceptible to this criticism! Useful advice is useful because it pushes us towards better tradeoffs. My claim is that generally folks ship too slowly. But obviously if we only care about velocity, that's bad, in the same way it's probably bad to only eat vegetables, or to work out every single day. But then the objection is not "Velocity is bad", it's that "Shipping bad things is bad". In the same way an unbalanced diet is bad, or working out when injured is bad.

Jtsummers 4y ago

What a long article to ultimately get to the realization (but fail to state it): If you pick a quantitative measure and optimize for it as a target, that's what you get. If you optimize for "velocity" (# of changes, here) then you get velocity, a lot of changes. If you optimize for "high impact" changes then you get high impact changes. This is not shocking. What also shouldn't be shocking is that if the two are only poorly correlated then you'll end up not getting meaningful improvements in the other measure, and may in fact reduce the other if there is a conflict between them. The bigger problem is that most organizations don't know what they want to optimize or how to measure the things they want to optimize. So they find proxies like "velocity" or "impact" and end up optimizing for them instead of what they actually want. In other words, Goodhart's Law.

marcosdumay 4y ago
It's very likely that velocity is very strongly correlated with impact, that's not exactly the problem. The thing that people mostly don't expect is that if you completely optimize for some metric, you will harm any metric that is not perfectly aligned with it. It doesn't matter if the correlation is very good, as long as it's not perfect it will be harmed.
- Jtsummers 4y ago
  Right. I was more referring to the article's definition of velocity which was number of changes ("launching more, faster"). It's easy for that to be misaligned with anything useful. You can make a lot of small things with little "impact" or value if your goal is just the number of changes. If you have a saner definition of velocity which is based on the size or scope and not just the raw number of changes, then you can pair that with prioritization based on expected value and your measures will be better aligned.
  - marcosdumay 4y ago
    But Agile's velocity is basically that. It's a measure of how long it will take you to change course if you discover your current design is bad and you should be working on something slightly different. Again, this is very likely very correlated with impact. But you seem to want it to be a measure of productivity; it's not.
    - Jtsummers 4y ago
      I gave two definitions of velocity and you just used "that". I have no idea if we're in agreement or disagreement now. If you mean that Scrum uses sizes, then yeah. It tends to be better aligned with what you want because it uses sizes and priorities. It doesn't just count the raw number of changes as the target, even for its velocity measure (which isn't the target, but a guide, at least in theory). The target in Scrum is getting what the customer (again, in theory) wants as quickly as possible which means focusing on priorities. Velocity is there to see how well you're operating rather than being a goal. If you mean Agile (which version?) doesn't use sizes but just a raw count of changes for velocity, then we're in disagreement. I mean, there probably are idiots out there who do that. But that's not what's encouraged by anyone that cares about getting shit done. It's a useless measure especially if it becomes your target for optimization. Anyone can get high velocity if all changes count the same. Submit 500 change requests to add 500 logging statements. Wow they're fast! Also they're gaming the system.

ojbyrne 4y ago

There may be a reasonable qualitative argument to favor Velocity over Impact, but the pseudo-quantitative argument made in the article is not really useful. You could easily adjust the numbers chosen to support the exact opposite argument.

Nevermark 4y ago

For changes that can be A/B tested, the negative outcomes can be ignored since they will be jettisoned. That changes the calculus of impact from positive-negative, to just positive. A big improvement. And presumably, the positive improvements are multiplicative. (i.e 1.01^100 ~= 2.7) > (1 + 100 * (1.01 -1) = 2.0) Whether with prior research, or after the fact verification, allowing your customer to be the source of what is, or is not, a good market fit always wins. But first, do no harm!

latchkey 4y ago

> I’m here to argue that development velocity (whatever that means) and launch throughput are entirely the wrong optimizations. Development velocity is pretty well defined in agile. The fact that the author starts of with admitting no understanding of what it means and then goes on to say that it is a wrong optimization, just seems off. When I worked at Pivotal Labs, velocity was king. It was controlled entirely by the development team as we would the point stories and do the work. Velocity (points added up from a weeks worth of stories finished and then averaged over time) was tracked very closely and easy to measure. Since everything was team and pair based, there were no pointing fingers at individuals... you had to complete things together. It was a very simple and easily computed metric that really showed output and allowed for great long term project estimates. We would always hit our expected run rate because of it. PM's could tell, in advance, when features were going to land. It really made for an excellent working environment where we could get into work in the morning and leave at the end of the day, without having to ever stay late to finish expectations.

q7xvh97o2pDhNrh 4y ago

The author's fixation on "improve your success percentage" is actually a highly toxic way to shape a team culture, even if it does (supposedly) yield some benefits. A good leader should let people know it's safe to fail. That's how we create an environment for boldness. If someone launched 10 failures and 10 successes this year, a very reasonable goal would be to launch 100 failures next year. Go fail more. Go fail all the time. You know how you'll pack that many failures into a year? Well, you'll have to launch more often. If you want to launch every feature more often, you'll need to launch features earlier, when they're smaller. If you want to launch smaller features, you need to know how to scope down. To know what scope you can cut for that many features, you'll need to talk to users more often — probably all the time. Et voila. So upping your rate of failure by 10x will mean you've upped your rate of learning by (at least) 10x. That will likely take you way further than some navel-gazing and overly-negative focus on "failing less often." Success comes from getting more turns at bat. You rack up way more strikes — and way more home runs.

drewcoo 4y ago

When did "experiment" come to mean "here's some data I made up to fit my narrative?" I think there are better ways to argue for Big Design Up Front. https://en.wikipedia.org/wiki/Big_Design_Up_Front https://en.wikipedia.org/wiki/Big_Design_Up_Front

jcparkyn 4y ago

> Research of A/B experiments carried out independently by multiple companies suggests that, at best, 1 in 3 ideas will have any measurable impact, and the industry average is likely 1 in 7 (14%). Everything else either has no impact or creates negative impact Some fairly big leaps here: - There's a big gap between "measurable impact" and "no impact", which is mostly dependent on measurement accuracy. For example, driving my car to work has no measurable impact on the air quality of earth as a whole, but that doesn't mean it has no impact. - The link for "1 in 3" doesn't even seem to mention that, as far as I can find. The closest is "At Google and Bing, only about 10% to 20% of experiments generate positive results". This is pretty skewed by the fact that those companies intentionally make minor or unintuitive changes to A/B test them, but can't really be generalised to all feature work at other companies. The link for "1 in 7" is also specifically for A/B testing, so generalising that to all "ideas" is quite a stretch.

vlovich123 4y ago

One limitation of this is when you do “no impact” work in year 1 and 2 that’s high value impact work in year 3. Sure, if everything is low hanging fruit go for it. But sometimes it can take longer to build something that has 10x impact. eg teams A, B, and C hit a wall because all their time is spent fighting fires because they didn’t invest in getting ahead of tech debt / product growth. Of course, it’s hard to make the case that that’s the case and you can be very wrong in which case you waste a lot of time. But making sure that you are managing smoldering fires and preventing them from becoming infernos is high impact “invisible” work - did you prevent the inferno or was it unlikely to ever catch in the first place”. The usual approach is to let things start tipping and hope you have enough lead time between “this is starting to become a problem” and “product growth has to cease and we have to start shedding customers”