2 ms·

I am confused where this traffic is coming from. OP says it's from well funded AI companies. But there are not such a large number of those? Why would they need

by andai 1y ago

I am confused where this traffic is coming from. OP says it's from well funded AI companies. But there are not such a large number of those? Why would they need to scrape the same pages over and over? Or is the scraping happening in real time due to the web search features in AI apps? (Cheaper to load the same page again than to cache it?)

hekkle 1y ago
On is website: https://maurycyz.com/projects/ai-tarpit/ https://maurycyz.com/projects/ai-tarpit/ He mentions that he had a "Chrome" browser send him 20 requests per second from the address: 43.134.189.59. If you look this address up on shodan.io you will see this address is for Tencent, a public company that makes AI, with an annual revenue of $92 Billion USD.
marginalia_nu 1y ago
Crawlers are pretty hard to build, they have an insane number of corner cases they need to deal with if you want them to perform well AND be perceived as respectful, and crawlers (if you go that route) find themselves among the harder problems in distributed computing, with a huge shared mutable state and some very complex shared timers. If you're in a hurry to race to the market, it's very likely you'll run into these issues and find yourself tempted to cut corners, and unfortunately, with nearly unbounded cloud spend, cutting corners in a large scale crawler operation can very believably cause major disruption all over the web.