3 ms·

Yeah, it is pretty darn complicated! A couple of things that helped us with the rate limits are: * Limiting most of our graphs/reporting to the past 3 years (s

Yeah, it is pretty darn complicated! A couple of things that helped us with the rate limits are: * Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * Fetching the data incrementally - namely, the first fetch is kinda of large, but then we're only trying to get the minimal set of changes/detla to a repo on a regular basis! That sounds awesome, I can't wait to see the product when it goes live!

sdesol 6y ago
> Limiting most of our graphs/reporting to the past 3 years (some repos are pretty old) * I really need the entire history, so limiting time ranges isn't possible. The biggest issue is really with cloning/fetching, since GitHub doesn't like people synching thousands of repos on a daily basis. Outside of those things, my search and analytics engine doesn't really need GitHub, since GitHub is unable to provide the data that I need anyways. I talked about it before, but what I'm really doing is moving Git's history into SQL (postgres in my case) and Lucene, so that I can slice and dice its history, to surface developer insights. Hence me calling it "business intelligence" for your codes history. >That sounds awesome, I can't wait to see the product when it goes live! Thanks and I think your product has potential as well and you should look at the Enterprise angle. Discovery in large companies is actually a serious problem. Companies really need a single place that they can go to, to see all that is available. GitHub, with their "Discuss" feature gets you part of the way, but companies still need a centralized place that can help them easily discover/discuss what is available internally.