8 ms·
There is a lot of talk about AI training being a driver of bot activity, but I think AI inference is also a driver, in two ways. (1) It's always been easy to
by PaulHoule 2y ago
There is a lot of talk about AI training being a driver of bot activity, but I think AI inference is also a driver, in two ways.
(1) It's always been easy to write bots [1] [2]. If you knew beautifulsoup well you could often write a scraper in 10 minutes, now people will ask ChatGPT to write a scraper for them and have a scraper ready in 15 minutes so they're discovering how easy it is, how you don't have to limit yourself to public APIs that are usually designed to limit access, not expand it.
(2) Instead of using content to train an AI you can feed it into an AI for inference. For instance, you can tell the AI to summarize pages or to extract specific facts from pages or to classify pages. It's increasingly possible to develop a workflow like: classify 30,000 RSS feed items, select 300 items that the user will probably find interesting, crawl those 300 pages looking for hyperlinks to scientific journal articles or other links that would be better to post, crawl those links to see if the journal articles are open access, weigh various factors to decide what's likely to be the best link, do specialized image extraction so I can make a good social post, etc. It's not too hard to do but it all comes falling down if the bot has to click on fire hydrants endlessly.
[1] Polite crawlers limit how many threads they have running against a single server. If you only have one thread per server you are unlikely to overload it. If you want to make a crawler with a large thread count that is crawling a large number of servers it can be a hassle to implement this, particularly if you want to maximize performance or run a large distributed crawler. However a lot of times I do a crawling project that targets one site or five sites or that maybe crawls 1000 documents a day and in those cases the single-threaded crawler is fine.
[2] For some reason, my management has always overestimated the work of building scrapers, I think because they've been burned by UI development which is always underestimated. The fact that UI development is such a bitch actually helps with crawler development -- you might be afraid that the target site is going to change but between the high cost of making changes and the fact that Google will trash your SEO if you change anything about your site, the target site won't change.
- showerst 2y agoAgreed on all points except [2], I run many scrapers and sites change _all the time_, often changing markup for seemingly random reasons. One government site I scrape changes ids and classes between camel and snake case every couple of weeks, it makes me wonder if it's a developer pulling a fast one on the client.
- dboreham 2y agoHearing this makes me suspect some tool auto-generates the id's and its config is getting changed every couple weeks by some spaces vs tabs battle between devs.