5 ms·

Looks pretty nice! The readme says it's not useful for repos with >2000 commits, but I wonder if we could hack together some sort of cache for it to facilitate

by wrigby 9y ago

Looks pretty nice! The readme says it's not useful for repos with >2000 commits, but I wonder if we could hack together some sort of cache for it to facilitate delta updates?

sdesol 9y ago
> we could hack together some sort of cache for it to facilitate delta updates? Crunching diffs at scale is actually an extremely I/O intensive operation and I can say this from experience, since my product is doing just that. You can see an example of what I mean at https://public.gitsense.com/insight/github?r=Microsoft/vscode#b%3Dgithub%3AMicrosoft%2Fvscode%3Amaster%26t%3Dcommits https://public.gitsense.com/insight/github?r=Microsoft/vscod... If you scroll down, you can see how the indexed information is used. The red, green, and blue bars, denotes how many lines were deleted, added and changed. The red, green and blue bars in the second row, shows how many lines were deleted, added and changed, which were non blank/comment lines. If you want to crunch diffs at scale, you really have to throw CPU cores, RAM and multiple SSDs at the problem. The more CPU cores, translates to more diffs that can be processed in parallel. The more RAM, translates to better Kernel page caching. And multiple SSDs in RAID 0 configuration, translates to better read and write speeds.
- mrmondo 9y ago
  Do you think it’s possible to calculate the relationship between the depth of the git data and the number of I/O operations? This is purely out of interest, our storage performs at around 2-4M random 4K read IOP/s per node and I was wondering (just for fun) how it might look thrown a good I/O crunch task like this.
  - sdesol 9y ago
    > Do you think it’s possible to calculate the relationship between the depth of the git data and the number of I/O operations It's actually pretty straight forward since you know exactly what operations has to be done. With very good I/O, you may find the CPU will become your bottleneck.
    - mrmondo 9y ago
      Yeah then it becomes even more interesting to me ;) especially when it comes to NUMA etc...

by wrigby 9y ago

Looks pretty nice! The readme says it's not useful for repos with >2000 commits, but I wonder if we could hack together some sort of cache for it to facilitate delta updates?

sdesol 9y ago

> we could hack together some sort of cache for it to facilitate delta updates? Crunching diffs at scale is actually an extremely I/O intensive operation and I can say this from experience, since my product is doing just that. You can see an example of what I mean at https://public.gitsense.com/insight/github?r=Microsoft/vscode#b%3Dgithub%3AMicrosoft%2Fvscode%3Amaster%26t%3Dcommits https://public.gitsense.com/insight/github?r=Microsoft/vscod... If you scroll down, you can see how the indexed information is used. The red, green, and blue bars, denotes how many lines were deleted, added and changed. The red, green and blue bars in the second row, shows how many lines were deleted, added and changed, which were non blank/comment lines. If you want to crunch diffs at scale, you really have to throw CPU cores, RAM and multiple SSDs at the problem. The more CPU cores, translates to more diffs that can be processed in parallel. The more RAM, translates to better Kernel page caching. And multiple SSDs in RAID 0 configuration, translates to better read and write speeds.

mrmondo 9y ago
Do you think it’s possible to calculate the relationship between the depth of the git data and the number of I/O operations? This is purely out of interest, our storage performs at around 2-4M random 4K read IOP/s per node and I was wondering (just for fun) how it might look thrown a good I/O crunch task like this.
- sdesol 9y ago
  > Do you think it’s possible to calculate the relationship between the depth of the git data and the number of I/O operations It's actually pretty straight forward since you know exactly what operations has to be done. With very good I/O, you may find the CPU will become your bottleneck.
  - mrmondo 9y ago
    Yeah then it becomes even more interesting to me ;) especially when it comes to NUMA etc...