3 ms·

Interpreting the metrics is always going to be a hard thing and I really can't see how this can become a SAAS business. The metrics generation part can certain

by sdesol 11y ago

Interpreting the metrics is always going to be a hard thing and I really can't see how this can become a SAAS business. The metrics generation part can certainly be, but how you should react to the numbers will depend on the software being developed, the people developing the software, and so forth. Basically there are a lot of variables to consider. However, if step 1 (collecting the metrics) is done poorly, you'll have no real way of making sense of what is going on and people tend to use this to push some agenda. The most important thing in my mind, is being able to capture information in a non-objective way. Developers, manager, project managers, etc. should be able to scrutinize the numbers. And to be able to do this, you need to be able to capture every line change and be able to abstract this change in a way that makes sense to upper management. For example, since GitLab is a hot topic right now, I've decided to index it to see what is going. If you goto https://github.com/gitlabhq/gitlabhq/graphs/contributors https://github.com/gitlabhq/gitlabhq/graphs/contributors you'll find GitHub's metrics for GitLab development. I like GitHub, but none of their metrics, provides any indicators for code quality or gives you any indicators of where to look for potential code quality problems. The closes thing is this: https://github.com/gitlabhq/gitlabhq/graphs/code-frequency https://github.com/gitlabhq/gitlabhq/graphs/code-frequency but their date range is way too great to provide any meaningful metrics. Those large spikes in the past are creating way too much noise. However if you goto http://imgur.com/lTW3lZh http://imgur.com/lTW3lZh you'll find a screenshot that shows the code churn for the master branch for the last 30 days. And that large spike not too long ago should make you wonder what's going on, and this is where being able to dig into the number is important. Since everything is captured in a SQL database, a simple query shows the following: 13376 (total), 13376 (add), 0 (chg), 0 (del), fixtures/emojis/index.json which explains everything. What this means to you will vary and it's why I don't think you can really make a SAAS business out of interpreting the numbers. The best you can do is capture everything under the sun and make it as easy as possible for them to make sense of the numbers.

sytse 11y ago
Cool, we're thinking of adding code analytics features to GitLab 8.5, what do you think of https://gitlab.com/gitlab-org/gitlab-ee/issues/112 https://gitlab.com/gitlab-org/gitlab-ee/issues/112 ?
- sdesol 11y ago
  I was actually going to fire off an email to you or comment on the issue at GitLab to further discuss GitLab's metrics plans/ambitions. Having spent years refining my indexing engine, I know being able to capture data accurately is not a trivial thing to do at the enterprise level (hundreds of repositories with 10's of thousands of branches) and the bullet points in the issue are certainly lofty if you want to make the numbers meaningful and drillable. I also can't overstate the importance of branch level indexing, as this is what people always neglect to do, since it doesn't scale well. However with branch level indexing, you can answer questions like where do we stand with release x and y very easily. Using GitLab development as an example, if a project manager wants to know how aligned release 8.4 and master are, the only options they have right now is to do a compare, which results in this: https://gitlab.com/gitlab-org/gitlab-ce/compare/master...8-4-stable https://gitlab.com/gitlab-org/gitlab-ce/compare/master...8-4... which can be quite overwhelming. Or you can tell them how many commits they are behind and ahead of one another which is interesting, but doesn't put things into perspective. However, if you've indexed things at the branch level, you'll be able to provide them with a higher level picture like this: http://imgur.com/lzHlk2P http://imgur.com/lzHlk2P which shows the deviation didn't start until around the 12th. And with further manipulation, you can show them something like this http://imgur.com/yewQJBc http://imgur.com/yewQJBc which shows the unique commits per branch. And from there, you can start drilling into the data like so http://imgur.com/58FExxL http://imgur.com/58FExxL to see what commits they were and what issues they fixed. Without branch level indexing, providing this type of metrics would not be possible. And as I've stated before, if you are focused on capturing the smallest detail, you'll be able to make the metrics meaningful for everybody. Let me know what is the best way to get in touch, as making my technology available for GitLab is definitely on the roadmap.
  - sytse 11y ago
    Looks cool, good point to make provide the analytics per branch. I would love to get in touch, please email sytse@ company domain.

by sdesol 11y ago

sytse 11y ago

Cool, we're thinking of adding code analytics features to GitLab 8.5, what do you think of https://gitlab.com/gitlab-org/gitlab-ee/issues/112 https://gitlab.com/gitlab-org/gitlab-ee/issues/112 ?

sdesol 11y ago
I was actually going to fire off an email to you or comment on the issue at GitLab to further discuss GitLab's metrics plans/ambitions. Having spent years refining my indexing engine, I know being able to capture data accurately is not a trivial thing to do at the enterprise level (hundreds of repositories with 10's of thousands of branches) and the bullet points in the issue are certainly lofty if you want to make the numbers meaningful and drillable. I also can't overstate the importance of branch level indexing, as this is what people always neglect to do, since it doesn't scale well. However with branch level indexing, you can answer questions like where do we stand with release x and y very easily. Using GitLab development as an example, if a project manager wants to know how aligned release 8.4 and master are, the only options they have right now is to do a compare, which results in this: https://gitlab.com/gitlab-org/gitlab-ce/compare/master...8-4-stable https://gitlab.com/gitlab-org/gitlab-ce/compare/master...8-4... which can be quite overwhelming. Or you can tell them how many commits they are behind and ahead of one another which is interesting, but doesn't put things into perspective. However, if you've indexed things at the branch level, you'll be able to provide them with a higher level picture like this: http://imgur.com/lzHlk2P http://imgur.com/lzHlk2P which shows the deviation didn't start until around the 12th. And with further manipulation, you can show them something like this http://imgur.com/yewQJBc http://imgur.com/yewQJBc which shows the unique commits per branch. And from there, you can start drilling into the data like so http://imgur.com/58FExxL http://imgur.com/58FExxL to see what commits they were and what issues they fixed. Without branch level indexing, providing this type of metrics would not be possible. And as I've stated before, if you are focused on capturing the smallest detail, you'll be able to make the metrics meaningful for everybody. Let me know what is the best way to get in touch, as making my technology available for GitLab is definitely on the roadmap.
- sytse 11y ago
  Looks cool, good point to make provide the analytics per branch. I would love to get in touch, please email sytse@ company domain.