3 ms·

This is cool and I appreciate software development insights is getting some love. Full disclosure, I wouldn't say I'm working on the same thing, but we definit

by sdesol 6y ago

This is cool and I appreciate software development insights is getting some love. Full disclosure, I wouldn't say I'm working on the same thing, but we definitely have some overlap, when it comes to software development insights, which is gaining steam. Your dashboard gave me some interesting food for thought, so I'll do the same and share with you, some of the things that I'm working on, that you are free to incorporate into openbase. https://imgur.com/Y06Tk3f https://imgur.com/Y06Tk3f https://imgur.com/Yjk2dzR https://imgur.com/Yjk2dzR https://imgur.com/RJw4ygS https://imgur.com/RJw4ygS https://imgur.com/xXi9EMd https://imgur.com/xXi9EMd https://imgur.com/nROCA0O https://imgur.com/nROCA0O https://imgur.com/xhb2haD https://imgur.com/xhb2haD https://imgur.com/sZ2YpIM https://imgur.com/sZ2YpIM https://imgur.com/HVSNhS0 https://imgur.com/HVSNhS0 https://imgur.com/TnInleD https://imgur.com/TnInleD My focus is not on the discovery part, which you guys are working on, but rather, I'm more focused on providing "business intelligence" for the software development lifecycle. I do agree with you that code metrics can only tell you so much, as it's not really meant for discovery. Code insights might provide things like stability, investment, complexity, etc., but like you said, you can't tell from code insights, if the code is easy to use or not. Discovery is a serious problem and GitHub has certainty not solved this problem yet, so what you guys are working on, is a problem worth solving in my opinion.

liorgrossman 6y ago
Thank you for sharing this! As you mentioned, we're focused on the discovery part, and package insights is a part of that, but so are reviews, ratings, search, and categorization. Your tool looks like a great solution for software teams to get insights into their code and dev processes, this also connects with the "developer velocity" trend I've heard several times in recent years. Looks like quite a lot of work too, kudos! Is it in production with some customers?
- sdesol 6y ago
  > Looks like quite a lot of work too, kudos! Thanks. I don't think people realize how complex it is to generate actionable code insights. Indexing code is easy, indexing history is an order of a magnitude more difficult, in my opinion. > Is it in production with some customers? No, the product isn't finished yet (I pivoted a bit from my initial idea). My goal is to have it ready by the end of this month, so that I can start indexing and making the indexed information publicly available. I was hoping to have a docker image/bare metal install packages available by the end of this month, but I don't think this will be feasible. Right now I'm continuously indexing 4,000 repos, but the challenge is really with GitHub and rate limits. When I have the public site up and running, I'll reach out to GitHub to see what can be done, as I would like to make the indexed information freely available for anybody to tap into, including Openbase, if I'm capturing things that you guys aren't.
  - liorgrossman 6y ago
    Yeah, it is pretty darn complicated! A couple of things that helped us with the rate limits are: * Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * Fetching the data incrementally - namely, the first fetch is kinda of large, but then we're only trying to get the minimal set of changes/detla to a repo on a regular basis! That sounds awesome, I can't wait to see the product when it goes live!
    - sdesol 6y ago
      > Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * I really need the entire history, so limiting time ranges isn't possible. The biggest issue is really with cloning/fetching, since GitHub doesn't like people synching thousands of repos on a daily basis. Outside of those things, my search and analytics engine doesn't really need GitHub, since GitHub is unable to provide the data that I need anyways. I talked about it before, but what I'm really doing is moving Git's history into SQL (postgres in my case) and Lucene, so that I can slice and dice its history, to surface developer insights. Hence me calling it "business intelligence" for your codes history. >That sounds awesome, I can't wait to see the product when it goes live! Thanks and I think your product has potential as well and you should look at the Enterprise angle. Discovery in large companies is actually a serious problem. Companies really need a single place that they can go to, to see all that is available. GitHub, with their "Discuss" feature gets you part of the way, but companies still need a centralized place that can help them easily discover/discuss what is available internally.

by sdesol 6y ago

liorgrossman 6y ago

Thank you for sharing this! As you mentioned, we're focused on the discovery part, and package insights is a part of that, but so are reviews, ratings, search, and categorization. Your tool looks like a great solution for software teams to get insights into their code and dev processes, this also connects with the "developer velocity" trend I've heard several times in recent years. Looks like quite a lot of work too, kudos! Is it in production with some customers?

sdesol 6y ago
> Looks like quite a lot of work too, kudos! Thanks. I don't think people realize how complex it is to generate actionable code insights. Indexing code is easy, indexing history is an order of a magnitude more difficult, in my opinion. > Is it in production with some customers? No, the product isn't finished yet (I pivoted a bit from my initial idea). My goal is to have it ready by the end of this month, so that I can start indexing and making the indexed information publicly available. I was hoping to have a docker image/bare metal install packages available by the end of this month, but I don't think this will be feasible. Right now I'm continuously indexing 4,000 repos, but the challenge is really with GitHub and rate limits. When I have the public site up and running, I'll reach out to GitHub to see what can be done, as I would like to make the indexed information freely available for anybody to tap into, including Openbase, if I'm capturing things that you guys aren't.
- liorgrossman 6y ago
  Yeah, it is pretty darn complicated! A couple of things that helped us with the rate limits are: * Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * Fetching the data incrementally - namely, the first fetch is kinda of large, but then we're only trying to get the minimal set of changes/detla to a repo on a regular basis! That sounds awesome, I can't wait to see the product when it goes live!
  - sdesol 6y ago
    > Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * I really need the entire history, so limiting time ranges isn't possible. The biggest issue is really with cloning/fetching, since GitHub doesn't like people synching thousands of repos on a daily basis. Outside of those things, my search and analytics engine doesn't really need GitHub, since GitHub is unable to provide the data that I need anyways. I talked about it before, but what I'm really doing is moving Git's history into SQL (postgres in my case) and Lucene, so that I can slice and dice its history, to surface developer insights. Hence me calling it "business intelligence" for your codes history. >That sounds awesome, I can't wait to see the product when it goes live! Thanks and I think your product has potential as well and you should look at the Enterprise angle. Discovery in large companies is actually a serious problem. Companies really need a single place that they can go to, to see all that is available. GitHub, with their "Discuss" feature gets you part of the way, but companies still need a centralized place that can help them easily discover/discuss what is available internally.