5 ms·
I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it s
by catillac 5y ago
I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1]
[1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning
- jgrahamc 5y agoI do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
- viraptor 5y agohttps://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
- jgrahamc 5y agoThanks. I'll bring this to the attention of the bots and DDoS teams.
- jimktrains2 5y agoYacy is 18 years old and not exactly obscure. If your bots team is unaware of it it's because they've chosen to be ignorant of it.
- joepie91_ 5y agoA search engine which is not run centrally by one organization on infrastructure in a known network, but rather something like YaCy where individual users run crawler nodes on networks that vary over time. Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.
- vntok 5y agoYeah, I certainly don't want those crawlers anywhere near my servers. Block by default and allow site admins to unblock should they want to seems like the best way. It is also already how it works with Cloudflare.