2 ms·
Who's doing this at such a high volume? Most of the data is static enough that there isn't value in frequent crawls, crawls are (probably) more expensive than c
by dehrmann 1y ago
Who's doing this at such a high volume? Most of the data is static enough that there isn't value in frequent crawls, crawls are (probably) more expensive than caching, and small shops and hobbyists don't have the resources to move the needle.
- chneu 1y agoI think it was wikipedia recently that was pissed because bots were crawling the site instead of using the already available datasets. AI bots don't care about caches. That's one of the big issues