3 ms·

One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential I

One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publish all those IPs up front?

defrost 1y ago
> But How? Typically by hiring them .. although their provenance might be sketchy.. eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man-botnet-for-hire/758293/ https://www.cybersecuritydive.com/news/us-charges-oregon-man... ^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar device mapping project). More legally there are many "free software" deals that offer services for people via installed software that comes with a side order of background web crawling in the fine print of the WALL-O'-TEXT Terms Of Uses agreement. Enterprising middle people gather up bots and offer them for hire to web crawlers, large scale companies will farm their own bots via their existing user base.
- BrenBarn 1y ago
  Okay, but are OpenAI and Meta straight up buying botnets on the black market?
  - defrost 1y ago
    Unlikely. There are many ways, at their scale they (Meta at least) probably have edge servers in ISP's across the planet and can easily mix their crawlers with residential IP addresses rotationally assigned by domestic ISP's they co-mingle with. Or some other way, legal but somewhat obfuscated.
  - diggan 1y ago
    OpenAI i'm not so sure about, but since Meta already got caught downloading copyrighted material to train LLMs, I think it isn't far fetched for them to also use borderline illegal methods for acquiring IPs to use.
tbonnin 1y ago
Some companies publish the IP ranges of their crawler (OpenAI [1], Mistral [2] for example) but many like Anthropic don’t. Not sure those lists can be fully trusted though. Perplexity, for instance, was caught using IPs outside of their declared list [3]. [1] https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots [2] https://docs.mistral.ai/robots/ https://docs.mistral.ai/robots/ [3] https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ https://blog.cloudflare.com/perplexity-is-using-stealth-unde...

defrost 1y ago

> But How? Typically by hiring them .. although their provenance might be sketchy.. eg: https://www.cybersecuritydive.com/news/us-charges-oregon-man-botnet-for-hire/758293/ https://www.cybersecuritydive.com/news/us-charges-oregon-man... ^ Oregon man arrested for running ~70,000 device DDOS-for-Hire botnet; the distributed attacking computers were mostly compromised IoT gadgets - fridges routers, toasters, doorbells, etc. with weak security that were probably scanned and p0wned via Shodan (or similar device mapping project). More legally there are many "free software" deals that offer services for people via installed software that comes with a side order of background web crawling in the fine print of the WALL-O'-TEXT Terms Of Uses agreement. Enterprising middle people gather up bots and offer them for hire to web crawlers, large scale companies will farm their own bots via their existing user base.

BrenBarn 1y ago
Okay, but are OpenAI and Meta straight up buying botnets on the black market?
- defrost 1y ago
  Unlikely. There are many ways, at their scale they (Meta at least) probably have edge servers in ISP's across the planet and can easily mix their crawlers with residential IP addresses rotationally assigned by domestic ISP's they co-mingle with. Or some other way, legal but somewhat obfuscated.
- diggan 1y ago
  OpenAI i'm not so sure about, but since Meta already got caught downloading copyrighted material to train LLMs, I think it isn't far fetched for them to also use borderline illegal methods for acquiring IPs to use.

tbonnin 1y ago

Some companies publish the IP ranges of their crawler (OpenAI [1], Mistral [2] for example) but many like Anthropic don’t. Not sure those lists can be fully trusted though. Perplexity, for instance, was caught using IPs outside of their declared list [3]. [1] https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots [2] https://docs.mistral.ai/robots/ https://docs.mistral.ai/robots/ [3] https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ https://blog.cloudflare.com/perplexity-is-using-stealth-unde...