3 ms·
Requiring PoW (proof-of-work) could take over for simple requests, rejecting requests until a sufficient nonce is included in the request. Unfortunately, this c
by OutOfHere 1y ago
Requiring PoW (proof-of-work) could take over for simple requests, rejecting requests until a sufficient nonce is included in the request. Unfortunately, this collective PoW could burden power grids even more, wasting energy+money+computation for transmission. Such is life. It would be a lot better to just upgrade the servers, but that's never going to be sufficient.
- Bjartr 1y agoSo, Anubis? https://anubis.techaro.lol/ https://anubis.techaro.lol/
- OutOfHere 1y agoYes, although the concept is simple enough in principle that a homegrown solution also works.
- Zardoz84 1y agoWe are wasting power on feeding statistics parrots, and we need to waste additional power to avoid being DoS by that feeding. We will be better without that useless waste of power.
- treyd 1y agoWhat do you suppose we as website owners do to prevent our websites from being DoSed in the meantime? And how do you suppose we convince/beg the corporations running AI scraping bots to be better users of the web?
- OutOfHere 1y agoThis should be an easy question for an engineer. It depends on whether the constraint is CPU or memory or database or network.
- zihotki 1y agoTechnology can't solve a human problem, the constraints are in budgets and in available time
- OutOfHere 1y agoWhat human problem. Do tell -- how have sites handled search engine crawlers for the past few decades? Why are AI crawlers functionally different? It makes no sense because they aren't functionally different.
- OutOfHere 1y agoAs of this year, AI has given people superpowers, doubling what they can achieve without it. Is this gain not enough? One can use it to run a more efficient web server.
- jaoane 1y agoWrite proper websites that do not choke that easily.
- HumanOstrich 1y agoSo I just need a solution with infinite compute, storage, and bandwidth. Got it.
- jaoane 1y agoThat is not what I said and that is not what is necessary. First of all web developers should use google and learn what a cache is. That way you don’t need compute at all.
- throwawayscrapd 1y agoAnd maybe you could Bing and learn what "cache eviction" is and why that happens when a crawler systematically hits every page on your site.
- OutOfHere 1y agoMaybe because it's an overly simplistic LRU cache, in which case a different eviction algorithm would be better. It's funny really since Google and other search engines have been crawling sites for decades, but now that search engines have competition, sites are complaining.
- OutOfHere 1y agoHow did you manage search engine crawlers for the past few decades? And why are AI crawlers functionally different? They aren't.
- jakderrida 1y agoIf I'm being honest... I expect the websites to keep returning errors and have hopes that those that employ you to at least start to understand what's going on.