4 ms·

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years. Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.

inigyou 1mo ago
It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely? > Us PHP developers I can tell.
- calvinmorrison 1mo ago
  it's not an insult. yes. cache heavily. shitty php stacks serving trillions of dollars of ecommerce sales have managed to do it for a long time.
  - zbentley 1mo ago
    Those ecommerce stacks serve a large fixed number of pages. cgit does not. Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages. Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text. Now imagine scrapers are permuting those URLs. How would you cache that? That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
  - inigyou 1mo ago
    Can you explain how a cache lets you avoid serving a request for the first time?
riknos314 1mo ago
The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate. The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """
- riknos314 1mo ago
  I'm not sure if the website allows for diffs against arbitrary tree states, but if it does than the diff space is completely unbounded, and the ram demand is theoretically infinite.
- calvinmorrison 1mo ago
  cache all of them? no... but you could cache some of them, likely the newer are accessed more frequently.
  - zbentley 1mo ago
    Per the article, the vast majority of traffic is scrapers that are generating effectively random, permuted URLs. Even if they're already caching their popular-with-real-humans most recent data, that efficiency gain is a tiny drop in a very large bucket.
  - desterothx 1mo ago
    Yes, and it still wouldnt help you since they are trying to fetch everything for each of the 921 forks, as stated in the article (same commits need to be cached again on a forked branch)

inigyou 1mo ago

It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely? > Us PHP developers I can tell.

calvinmorrison 1mo ago
it's not an insult. yes. cache heavily. shitty php stacks serving trillions of dollars of ecommerce sales have managed to do it for a long time.
- zbentley 1mo ago
  Those ecommerce stacks serve a large fixed number of pages. cgit does not. Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages. Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text. Now imagine scrapers are permuting those URLs. How would you cache that? That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
- inigyou 1mo ago
  Can you explain how a cache lets you avoid serving a request for the first time?

riknos314 1mo ago

The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate. The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """

riknos314 1mo ago
I'm not sure if the website allows for diffs against arbitrary tree states, but if it does than the diff space is completely unbounded, and the ram demand is theoretically infinite.
calvinmorrison 1mo ago
cache all of them? no... but you could cache some of them, likely the newer are accessed more frequently.
- zbentley 1mo ago
  Per the article, the vast majority of traffic is scrapers that are generating effectively random, permuted URLs. Even if they're already caching their popular-with-real-humans most recent data, that efficiency gain is a tiny drop in a very large bucket.
- desterothx 1mo ago
  Yes, and it still wouldnt help you since they are trying to fetch everything for each of the 921 forks, as stated in the article (same commits need to be cached again on a forked branch)