3 ms·
My main concern is that this whole situation will not include only chat bots but it will also trigger a demand for crawling bots which could put a lot of stress
by elorant 10y ago
My main concern is that this whole situation will not include only chat bots but it will also trigger a demand for crawling bots which could put a lot of stress on web sites. Then sites could adopt aggressive captcha schemes which will hammer usability and make the life of legitimate crawlers a lot more difficult. It’s a lose-lose situation , until the industry comes into its senses and realize this whole thing is a fad.
- TeMPOraL 10y agoThat and - assuming new crawlers will even bother to respect robots.txt - this may screw up the archive.org's work even more.
- fizzbatter 10y agoOne solution to this could simply be improving our ability to handle traffic. Bots aren't doing anything bad (under normal condition), they just represent a portion of your view traffic that you don't inherently want. An aggressive user, of sorts. With that said, we have more cycles these days, better web servers, and inherently more traffic capacity than we did 10 years ago. I have a feeling bot traffic will grow faster than our traffic-handling-tech, but regardless, i think this is just going to be the new standard. Especially since bot traffic could represent UX requirements for potential users. Quite an interesting future we have though.
- markbnj 10y agoI don't understand the connection. What do chatbots have to do with scrapers? Are you saying that the chatbots will need to use scrapers to harvest the information they present to users? Might be true in some cases, I guess.