3 ms·
If you're getting a lot of crawler traffic, your site probably has information a lot of people find useful, so you should consider finding a way to monetize it.
by huffmsa 9y ago
If you're getting a lot of crawler traffic, your site probably has information a lot of people find useful, so you should consider finding a way to monetize it.
Otherwise, your best bet (hardest to get around in my experience) is monitoring for actual user I/O. Like if someone starts typing in an input field, real humans have to click on it beforehand, and most bots won't.
Or if a user clicks next-page without the selector being visible or without scrolling the page at all. Not natural behavior.
Think like a human.
- FroshKiller 9y agoYou will create accessibility issues for users if you do this. The bias you'd encode in this idea of "human" behavior doesn't consider assistive software at all. I don't click text inputs when my form-filling plugin enters my personal information on a payment screen. And even if I did, you wouldn't know it if I had JavaScript disabled.
- laughfactory 9y agoAs someone who who does a fair amount of scraping for my job, it's always vastly preferred to simply pay for the data. It's generally more efficient and cost effective then scraping. Scraping is always the last resort: we need the data, it's not available for sale, and it's publicly accessible.