4 ms·
> Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing. I don't know about $10m/year, but many site
by nextaccountic 5y ago
> Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.
I don't know about $10m/year, but many sites block bots just because they don't want competitors to access publicly available data. Which is bullshit.