3 ms·

> Looks like quite a lot of work too, kudos! Thanks. I don't think people realize how complex it is to generate actionable code insights. Indexing code is ea

by sdesol 6y ago

> Looks like quite a lot of work too, kudos! Thanks. I don't think people realize how complex it is to generate actionable code insights. Indexing code is easy, indexing history is an order of a magnitude more difficult, in my opinion. > Is it in production with some customers? No, the product isn't finished yet (I pivoted a bit from my initial idea). My goal is to have it ready by the end of this month, so that I can start indexing and making the indexed information publicly available. I was hoping to have a docker image/bare metal install packages available by the end of this month, but I don't think this will be feasible. Right now I'm continuously indexing 4,000 repos, but the challenge is really with GitHub and rate limits. When I have the public site up and running, I'll reach out to GitHub to see what can be done, as I would like to make the indexed information freely available for anybody to tap into, including Openbase, if I'm capturing things that you guys aren't.

liorgrossman 6y ago
Yeah, it is pretty darn complicated! A couple of things that helped us with the rate limits are: * Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * Fetching the data incrementally - namely, the first fetch is kinda of large, but then we're only trying to get the minimal set of changes/detla to a repo on a regular basis! That sounds awesome, I can't wait to see the product when it goes live!
- sdesol 6y ago
  > Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * I really need the entire history, so limiting time ranges isn't possible. The biggest issue is really with cloning/fetching, since GitHub doesn't like people synching thousands of repos on a daily basis. Outside of those things, my search and analytics engine doesn't really need GitHub, since GitHub is unable to provide the data that I need anyways. I talked about it before, but what I'm really doing is moving Git's history into SQL (postgres in my case) and Lucene, so that I can slice and dice its history, to surface developer insights. Hence me calling it "business intelligence" for your codes history. >That sounds awesome, I can't wait to see the product when it goes live! Thanks and I think your product has potential as well and you should look at the Enterprise angle. Discovery in large companies is actually a serious problem. Companies really need a single place that they can go to, to see all that is available. GitHub, with their "Discuss" feature gets you part of the way, but companies still need a centralized place that can help them easily discover/discuss what is available internally.

by sdesol 6y ago

liorgrossman 6y ago

Yeah, it is pretty darn complicated! A couple of things that helped us with the rate limits are: * Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * Fetching the data incrementally - namely, the first fetch is kinda of large, but then we're only trying to get the minimal set of changes/detla to a repo on a regular basis! That sounds awesome, I can't wait to see the product when it goes live!

sdesol 6y ago
> Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * I really need the entire history, so limiting time ranges isn't possible. The biggest issue is really with cloning/fetching, since GitHub doesn't like people synching thousands of repos on a daily basis. Outside of those things, my search and analytics engine doesn't really need GitHub, since GitHub is unable to provide the data that I need anyways. I talked about it before, but what I'm really doing is moving Git's history into SQL (postgres in my case) and Lucene, so that I can slice and dice its history, to surface developer insights. Hence me calling it "business intelligence" for your codes history. >That sounds awesome, I can't wait to see the product when it goes live! Thanks and I think your product has potential as well and you should look at the Enterprise angle. Discovery in large companies is actually a serious problem. Companies really need a single place that they can go to, to see all that is available. GitHub, with their "Discuss" feature gets you part of the way, but companies still need a centralized place that can help them easily discover/discuss what is available internally.