3 ms·

I wasn't sure what to thing about Cycle Analytics, and the metrics it provides ("time from thoughts to issue", "time from issue to code", "time spent reviewing"

I wasn't sure what to thing about Cycle Analytics, and the metrics it provides ("time from thoughts to issue", "time from issue to code", "time spent reviewing"). IMHO these numbers may be too synthetic to give meaningful information. Plus, once we have this kind of dashboard, there is a risk of starting to optimize for the numbers (instead of optimizing the reality underneath). That said, I see with this demo how these metrics could be useful to track, well, when reviews are taking too much time (needs more people? better repartition of reviewers?) – or when maintenance tasks are slowing new features (needs better tests? more maintainers?). I guess I'll have to try this out :)

sytse 10y ago
For more detail about the numbers see https://gitlab.com/help/user/project/cycle_analytics#how-the-data-is-measured https://gitlab.com/help/user/project/cycle_analytics#how-the... Do you mean that number are to coarse to indicate a specific problem? Of course there is the problem of gaming the numbers. But we do think that compared to many other ways of measuring productivity (for example the number of issues solved) this is relatively robust against manipulation. Getting something out sooner is better most of the time. But thanks for the thoughts and I hope you try it out soon.
- jacques_chester 10y ago
  > relatively robust against manipulation My own view is that robustness to manipulation can't come from metric design, it can only come from culture. If metrics are tied to reward or punishment, then no matter how clever they are, they will be gamed.
  - sytse 10y ago
    I agree that everything will be gamed and that culture is the best protection. I do think that some metrics are easier to manipulate (cyclomatic complexity comes to mind) and some metrics don't correspond with effectiveness (lines of code written).
jacques_chester 10y ago
At Pivotal some of the PMs have kicked around "Time to Value", which is the gap between an entry in Pivotal Tracker and a buck being turned on that feature. Then there's Time to Customer Value, which is the time it takes before a customer using the feature turns a buck on it. > there is a risk of starting to optimize for the numbers There absolutely is. You can only use your own judgment of the balance of risk between flying blind and becoming obsessed with instrumentation.
- sytse 10y ago
  Interesting. What is a buck? Is it a feature flag?
  - sciurus 10y ago
    A buck is a dollar. To "turn a buck" means to make money. https://en.wikipedia.org/wiki/Slang_terms_for_money#United_States https://en.wikipedia.org/wiki/Slang_terms_for_money#United_S...
    - sytse 10y ago
      Ah, makes sense, thanks. I get the Time to Customer Value now (customer of the user ordering due to the new feature). Not sure about the Time to Value. Who is billing who? Pivotal the user?
      - jacques_chester 10y ago
        We sell a bunch of software subscriptions these days. If someone buys or increases their subscription on a new feature, that'd tick the marker. It's probably not precisely measurable, but it's a great thought exercise. It prompts us to think about the entire process from noticing an idea to a customer deciding to buy. And Time to Customer's Customer Value -- I forgot that TTC^2V was the alternative name -- is even better. It makes us think about the journey from us noticing a need, developing it, releasing, customer installing, their team using it, releasing to their customers, who decide it's worth paying for. When you turn that into a loop, you get something very like the Gitlab master plan. We call it the "Circle of Code", Onsi Fakhouri talked about it early this year: https://youtu.be/7APZD0me1nU?t=23m6s https://youtu.be/7APZD0me1nU?t=23m6s

sytse 10y ago

For more detail about the numbers see https://gitlab.com/help/user/project/cycle_analytics#how-the-data-is-measured https://gitlab.com/help/user/project/cycle_analytics#how-the... Do you mean that number are to coarse to indicate a specific problem? Of course there is the problem of gaming the numbers. But we do think that compared to many other ways of measuring productivity (for example the number of issues solved) this is relatively robust against manipulation. Getting something out sooner is better most of the time. But thanks for the thoughts and I hope you try it out soon.

jacques_chester 10y ago
> relatively robust against manipulation My own view is that robustness to manipulation can't come from metric design, it can only come from culture. If metrics are tied to reward or punishment, then no matter how clever they are, they will be gamed.
- sytse 10y ago
  I agree that everything will be gamed and that culture is the best protection. I do think that some metrics are easier to manipulate (cyclomatic complexity comes to mind) and some metrics don't correspond with effectiveness (lines of code written).

jacques_chester 10y ago

At Pivotal some of the PMs have kicked around "Time to Value", which is the gap between an entry in Pivotal Tracker and a buck being turned on that feature. Then there's Time to Customer Value, which is the time it takes before a customer using the feature turns a buck on it. > there is a risk of starting to optimize for the numbers There absolutely is. You can only use your own judgment of the balance of risk between flying blind and becoming obsessed with instrumentation.

sytse 10y ago
Interesting. What is a buck? Is it a feature flag?
- sciurus 10y ago
  A buck is a dollar. To "turn a buck" means to make money. https://en.wikipedia.org/wiki/Slang_terms_for_money#United_States https://en.wikipedia.org/wiki/Slang_terms_for_money#United_S...
  - sytse 10y ago
    Ah, makes sense, thanks. I get the Time to Customer Value now (customer of the user ordering due to the new feature). Not sure about the Time to Value. Who is billing who? Pivotal the user?
    - jacques_chester 10y ago
      We sell a bunch of software subscriptions these days. If someone buys or increases their subscription on a new feature, that'd tick the marker. It's probably not precisely measurable, but it's a great thought exercise. It prompts us to think about the entire process from noticing an idea to a customer deciding to buy. And Time to Customer's Customer Value -- I forgot that TTC^2V was the alternative name -- is even better. It makes us think about the journey from us noticing a need, developing it, releasing, customer installing, their team using it, releasing to their customers, who decide it's worth paying for. When you turn that into a loop, you get something very like the Gitlab master plan. We call it the "Circle of Code", Onsi Fakhouri talked about it early this year: https://youtu.be/7APZD0me1nU?t=23m6s https://youtu.be/7APZD0me1nU?t=23m6s