4 ms·
Distributed bot and scraper networks. Thousands of IPs geographically dispersed throughout the world. There is only so much you can do with rate limiting.
by tyri_kai_psomi 7y ago
Distributed bot and scraper networks. Thousands of IPs geographically dispersed throughout the world. There is only so much you can do with rate limiting.
- StuffedParrot 7y agoProxies can also work well for cheaper than buying distributed compute.
- gdulli 7y agoThey asked about LinkedIn, where the content is gated behind a login. If it was a rate limiting problem, that would be trivial. Needing to be logged in as the same user defeats the purpose of proxying to hide your physical origin. Registering thousands of different users to use in a distributed way is hard now that they require a text message verification for new accounts.
- manigandham 7y agoPublic LinkedIn profiles (which is many of them) are open to scrapers and they lost a court case about it. https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
- gdulli 7y agoI go to LinkedIn without being logged in and nearly always get a login gate instead of the profile. They were ordered to unblock hiQ specifically, they were not ordered to open up content to scrapers generally. They can still throttle high volume traffic and put up captchas. I think the only specific thing the court ordered was for them to unblock hiQ IP ranges.