7 ms·

"Every measure of a good developer is qualitative. That makes them all subjective." ^ This post scratches the surface on a timely and relevant question many te

• Eiriksmal 10y ago
  Engineer happily using GitPrime here. After sitting in on a Skype session we had with your team once upon a time, I took your earlier (pre-Impact) observations on the correlation between commit volume and overall performance to heart and consciously focused on chunking my work more. It will come as no surprise to you that this strategy's helped me stay consistently in the upper right of the 2x2 chart. Thanks, Ben!
  • thebent 10y ago
    Ah nice, @Eiriksmal — 'once upon a time…' and 'pre-impact', sounds like this was a while ago now :) Encouraging to hear that this has been helpful for you.
  • Xcelerate 10y ago
    > I took your earlier observations on the correlation between commit volume and overall performance What does this mean? The best developers tend to have the most commits?
    • jiaweihli 10y ago
      I can believe that there's _some_ correlation. From what I've seen, a high frequency of commits means either: - Has very little understanding and churns out code one day and removes it the next. - Thinks a lot about their code, and divides them into digestible, independent chunks.
      • ____nope 10y ago
        So it correlates either to a lazy developer or a really good developer? That isn't much of a correlation.
        • jiaweihli 10y ago
          It's some correlation, and it's more useful than the full spectrum.
• gaastonsr 10y ago
  Love this, in my experience also, good programmers make a lot of small commits. While less experienced people do ocasional commits regardless if they are big or small. Something your chart doesn't show is people who make a lot of commits/work but the quality is subpar. They behave as prolific programmers but still need to improve.
  • thebent 10y ago
    Thanks for checking it out @gaastonsr — yeah, you make a great point. we've figured out some ways to account for that in the software itself.
• shandor 10y ago
  That's actually fascinating. Is the formula for the mentioned "developer impact" a hush-hush trade secret, or can you go into more detail on how you calculate it? Would be really interested in hearing more!
  • thebent 10y ago
    @shandor — yeah, there's some secret sauce happening in that impact metric, but this post breaks it down so you get a sense of how it's calculated and how teams are using it: https://blog.gitprime.com/impact-a-better-way-to-measure-codebase-change https://blog.gitprime.com/impact-a-better-way-to-measure-cod...
• debaserab2 10y ago
  I agree with the overall message of the post but not the analysis behind it. It depends completely on the codebase and where you as an engineer are assigned to work in that codebase. There's so many potential ways for bias to be introduced into that kind of analysis (e.g., what if I am a junior dev assigned to copy edits to the static sections of the website?). There's a qualitative perspective to commits that's going completely ignored in this post.
  • marktangotango 10y ago
    This ties into what I was talking about above with respect to "heavy algorithmic stuff". Most developers can work with intricate legacy code by adding an if here and there to handle special cases. Very few can recognize that it's encoding a state machine with state transition function Y, and rewrite it. Even then, very few have enough 'attention to detail' to do the rewrite, and NOT lose any legacy functionality. This is an example of what I consider to be 'hard stuff'.
    • Bahamut 10y ago
      This is why I like when people contribute to major open source projects - chances are good they will be exposed to some of these hard problems, which can help them solve future ones.
  • thebent 10y ago
    @debaserab2 100% agree, well said.
• mkalygin 10y ago
  Nice article indeed, thanks for sharing. I'm currently struggle to go from perfectionist with low frequency to high frequency. I've also noticed that experienced programmers do their commits more frequently and they do the right job - they spend less time on unimportant details. It also depends on the codebase. A programmer who works on similar codebases for 10 years is more likely to have a high impact than the one who work less and consider these codebases rather new. While their approach and skill set may be similar.
• Quarrelsome 10y ago
  surely this assessment introduces false positives? Someone that implements a poorly specified feature exceptionally well suffers rework, not because their code is poor but due to outside factors. Someone who cherry picks easy items is reflected as prolific where a perfectionist might just be given more difficult problems. Idk, I worry that people will interpret this backwards and rather than having characteristics of a good developer ape those characteristics to appear as a good developer.
  • nine_k 10y ago
    Whatever you measure, when people know it, will affect how people act. People will try to move the needle in the direction they see profitable, even unconsciously.
  • Bartweiss 10y ago
    I think Goodhart's Law is in effect here. As a management metric, I would find this concerning - perverse incentives and false positives all over the place. But as a personal observation, I'm going to take it to heart. This (like most other metrics) seems most useful from a starting point of "while doing my daily work and seeking to write good code, I will apply this". That doesn't make a good tool for deciding who's talented, but I'm still going to remember it.
• cbcoutinho 10y ago
  You guys maintain an excellent blog - I would really like to keep up-to-date, but I don't want emails sent to me. Can you guys supply an RSS feed link? If you have one, I wasn't able to find it.
  • thebent 10y ago
    @cbcoutinho appreciate that. i think this should work: https://blog.gitprime.com/rss.xml https://blog.gitprime.com/rss.xml
• SeanDav 10y ago
  The danger of developing a qualitative metric for work/job performance, is you now have a recipe to game the system for anyone that is so inclined. Of course, if this metric is used as an indicator for promotion and raises, nearly everyone will be inclined to game the system. It might not even be as overt as that. It is a well known phenomenon that what you measure, you improve. So as soon as you have a quantitative metric for work performance, almost by definition, it can become problematic to use, unless you are very careful how it is used. This is not to say that finding a qualitative method is useless, far from it. It can lead to valuable insights and improve overall quality of the team/business.
  • SeanDav 10y ago
    Oops messed up my qualitative and quantitative. All references should have been "quantitative". This is what happens before the first coffee!
• zaptheimpaler 10y ago
  Im currently working on performance optimization. Often my job is days (even weeks) of running tests and reading metrics, followed by a 5 line change one day that significantly speeds things up. This is one reason why metrics cant hope to capture everything. Maybe its not all subjective but a lot of it is - especially for more senior people, its about how much contribution they have to the overall business goals rather than any data you might find in the codebase. Ofcourse there are people who do this much better/faster than me, but the nature of the job still means the actual code output is pretty low. I can imagine many other functions where code output is super high (adding lots of text of any kind - blogs, templates, new library/config) or super low. I don't think code output correlates well to productivity.
  • Bartweiss 10y ago
    I have a longstanding suspicion that these issues are why there are so many different 'measurement' structures for programming, which don't ever converge to a few winners. I totally believe the GitPrime result is accurate, but my first thought was "I wonder which teams this doesn't apply to?" 'Programmers' is a coherent grouping in terms of "people who write code as a job", but it doesn't generalize well to "people whose daily work is similar". At the extremes, it's a bit like calling a typist, a copy editor, and a poet all "writers" because they produce text. That's a useful grouping when you're designing Microsoft Word, but it's a lot less useful when you're judging everyone on "lines written today". (I worry that there's a value implication in those three careers, and it's not intended; I just wanted vastly different daily workflows.) And so the GitPrime result seems useful to know about, but there are clear cases like yours where its inapplicable. I suspect it would also run into issues with things like embedded or critical-failure code - NASA's commit patterns might have nothing in common with the industry standard. Even in my largely standard work, I realize that of my last two projects, one had ~50 commits and the other had ~3. The 3-commit one was a tech-debt project with lots of reading and documentation work, and I'm skeptical the two are at all comparable. Even within similar work, I imagine things like branching strategies and test coverage alter commit frequency. For the near future, we'll probably continue to see the value of a given assessment stay company-and-role specific.
  • hermitdev 10y ago
• ktRolster 10y ago
  If you use a metric like that to affect my pay, I will happily maximize the metric to ensure I get paid as much as possible. As a bonus for you, I will also try to write good code.
  • touristtam 10y ago
    Gamification at its finest. I copmpletely agree with you. :)
  • wccrawford 10y ago
    So true! I have yet to see a metric that wasn't gamed somehow, even by people who are generally honest and upright otherwise. If you base someone's livelihood on a number, expect them to attempt to maximize that number, probably at the expense of other numbers that you may or may not be considering.
    • ad-hominem 10y ago
      Gaming a metric your employer uses to evaluate you isn't inherently immoral or dishonest.
  • ronald_raygun 10y ago
    Reminds me of a good essay - "You are what you measure"
  • thebent 10y ago
    “to affect my pay” ← totally agree with you here. This is a big part of why we think engineering metrics are a tough nut to crack: there’s a fair bit of research that suggests tying KPIs to compensation is a huge anti-pattern for certain types of work (chiefly those involving lots of novel problem solving). Highly recommend Daniel Pink’s “A Whole New Mind” on this topic; it’s pretty fascinating stuff.
    • thebent 10y ago
      Highly recommend Daniel Pink’s “A Whole New Mind” on this topic Correction, the Pink title I was referring to was "Drive: The Surprising Truth About What Motivates Us" Pink does a great job unpacking research that demonstrates how this type of 'affect my pay' dynamic backfires for engineering.
• deleted 10y ago
  [deleted]
• bananarepdev 10y ago
  So you are not measuring the value delivered by the developer or his team to the business, but rather the amount of work done.
• Bartweiss 10y ago
  This is really interesting, thank you. It's one of the only clear-metric breakdowns that actually makes sense in light of my programming experiences. I'm hesitant about the idea of using it as a management tool (easy to game, and high commit count or even 'high impact' isn't proof of quality). And, as some other people noted, it's quite role-specific - a performance engineer's commit count won't be comparable to a line engineer. But as a personal insight, or a mentoring tip? This seems excellent. I know I want to write good code, I'm not going to game my personal metrics, so a clear guideline like "break up code more" is a great thing to have.
  • thebent 10y ago
    @Bartweiss thanks for the feedback. The thing about “gaming the system”, is that it presumes an adversarial relationship between engineers and non-engineers. That’s pretty unfortunate, and it’s our view that a good portion of this is due to non-engineers not really understanding what happens software development. But yeah, my co-founder has a pretty interesting take on gaming the system. TL;DR: So, can you game the system? Absolutely. The point is to game the system: https://intercom.help/gitprime/general/cant-you-just-game-the-metrics https://intercom.help/gitprime/general/cant-you-just-game-th...
    • Bartweiss 10y ago
      Thanks, that was a great quick read. Adversarial work does seem like it's largely a sign of communication breakdown. I think 'gaming' isn't always as adversarial as it seems, though. I was considering it in terms of Goodhart's Law, where even with good intentions, representative metrics become less representative as you organize around them. The "active days" entry in your link seems like an example; if everyone pushes most days for the sake of pushing, then active days isn't feedback on programmer skill. Of course, your link touches on that. "Everyone pushes every day" is a great outcome, even if the metric is no longer clear. The common trend is to pick good metrics, then struggle to keep them relevant. I really like the alternative of just setting metrics that will be good after people optimize.
    • hulahoof 10y ago
• JamesBarney 10y ago
  What was the dataset you guys used? Was it open-sourced or closed source?
• pmarreck 10y ago
  Quantitative information can only ever be used to inform qualitative valuation. It should not be absolutely linked (in other words, you should not make hire/fire decisions solely based on metrics of any sort). Your work is definitely interesting, though. Note that every metric can be gamed, by the way.