4 ms·
Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.
by znnajdla 2mo ago
Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.
- ninglor 2mo agoThis is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.
- grep_it 2mo agoDid you read the article? “[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
- jwilk 2mo agoFrom the HN guidelines <https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html>: > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
- inigyou 2mo agoWhat if it's really obvious they didn't read the article?
- znnajdla 2mo agoI did read the article. It just didn't occur to me that their combinatorial explosion of diffs was scrape-able. To be honest that sounds like an scrapers tarpit / honeypot now, because there is no value in scraping trillions of diffs. Sounds like the issue could be fixed by putting the diffs in a frontend app, not scrapable by URL, only by clicking around the app.
- jopsen 2mo agoThey allow you to diff commits, which is an awesome feature. But if bots a crawling diffs between all possible commits it's crazy. CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
- inigyou 2mo agoNo CDN blocks bots. Some claim to. None do.