3 ms·

Please explain the problem (here or via email to me).

catillac 5y ago
I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1] [1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning
- jgrahamc 5y ago
  I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
  - viraptor 5y ago
    https://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
    - jgrahamc 5y ago
      Thanks. I'll bring this to the attention of the bots and DDoS teams.
      - jimktrains2 5y ago
        Yacy is 18 years old and not exactly obscure. If your bots team is unaware of it it's because they've chosen to be ignorant of it.
  - joepie91_ 5y ago
    A search engine which is not run centrally by one organization on infrastructure in a known network, but rather something like YaCy where individual users run crawler nodes on networks that vary over time. Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.
    - vntok 5y ago
      Yeah, I certainly don't want those crawlers anywhere near my servers. Block by default and allow site admins to unblock should they want to seems like the best way. It is also already how it works with Cloudflare.
SXX 5y ago
It is great that you care and I guess others are already provided some examples, but I'll add my own 2c here. Obvious problem is that centralized service like CloudFlare do create entry barrier and make large players on search and data mining markets even more entrenched than ever. Recently your company announced partnership with Internet Archive, but if CloudFlare want to continue play a role as behevolent party everyone should have equal access to this data. Yeah it means that some bad actors will be able to easily scrap the web too, but... CloudFlare service can't prevent scrapping anyway. There are shady residential proxy networks, services to bypass captcha and scrapping software like Zennoposter. It's possible to make scrapping more expensive, but bad actors don't care because they have money. Unfortunately enthusiasts, open source projects and small companies don't have enough resources to do the same.
- dannyw 5y ago
  Making scraping harder definitely reduces scraping. Some bad actors will get through, but others will be deterred. I think you might not understand that it's site owners like me who want to stop scraping. It usually comes from specific bad incidents, like copycat sites stealing our content and work. Cloudflare wouldn't block scraping if website owners didn't want it. And website owners can easily disable this protection.
  - SXX 5y ago
    Scrapping protection is not a problem: defaults that CloudFlare promote are. Saying that website owners can disable it is akin saying website owners should go and whitelist Tor nodes. Most of website owners don't understand either of issues and they never gonna opt-out. Also I'm talking from experience because I been on both sides of the fence: doing scrapping and implementing protection. So yeah your CloudFlare protection will deter 10% of bad actors, but will also cut off 99% of enthusiast / research efforts or users of niche software or browsers. Still anyone with $1000+ budget will scrap whatever they want.

I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1] [1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning

jgrahamc 5y ago
I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
- viraptor 5y ago
  https://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
  - jgrahamc 5y ago
    Thanks. I'll bring this to the attention of the bots and DDoS teams.
    - jimktrains2 5y ago
      Yacy is 18 years old and not exactly obscure. If your bots team is unaware of it it's because they've chosen to be ignorant of it.
- joepie91_ 5y ago
  A search engine which is not run centrally by one organization on infrastructure in a known network, but rather something like YaCy where individual users run crawler nodes on networks that vary over time. Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.
  - vntok 5y ago
    Yeah, I certainly don't want those crawlers anywhere near my servers. Block by default and allow site admins to unblock should they want to seems like the best way. It is also already how it works with Cloudflare.

SXX 5y ago

It is great that you care and I guess others are already provided some examples, but I'll add my own 2c here. Obvious problem is that centralized service like CloudFlare do create entry barrier and make large players on search and data mining markets even more entrenched than ever. Recently your company announced partnership with Internet Archive, but if CloudFlare want to continue play a role as behevolent party everyone should have equal access to this data. Yeah it means that some bad actors will be able to easily scrap the web too, but... CloudFlare service can't prevent scrapping anyway. There are shady residential proxy networks, services to bypass captcha and scrapping software like Zennoposter. It's possible to make scrapping more expensive, but bad actors don't care because they have money. Unfortunately enthusiasts, open source projects and small companies don't have enough resources to do the same.

dannyw 5y ago
Making scraping harder definitely reduces scraping. Some bad actors will get through, but others will be deterred. I think you might not understand that it's site owners like me who want to stop scraping. It usually comes from specific bad incidents, like copycat sites stealing our content and work. Cloudflare wouldn't block scraping if website owners didn't want it. And website owners can easily disable this protection.
- SXX 5y ago
  Scrapping protection is not a problem: defaults that CloudFlare promote are. Saying that website owners can disable it is akin saying website owners should go and whitelist Tor nodes. Most of website owners don't understand either of issues and they never gonna opt-out. Also I'm talking from experience because I been on both sides of the fence: doing scrapping and implementing protection. So yeah your CloudFlare protection will deter 10% of bad actors, but will also cut off 99% of enthusiast / research efforts or users of niche software or browsers. Still anyone with $1000+ budget will scrap whatever they want.