5 ms·

Using a different definition of Google than TFA : We have discussed recently a huge issue preventing new Google-like web crawling competitors from emerging :

Using a different definition of Google than TFA : We have discussed recently a huge issue preventing new Google-like web crawling competitors from emerging : If you're not GAFAM(-like), and so aren't whitelisted, then anti-spam/DDoS filters like Cloudflare will simply prevent your crawler from working !

easytiger 6y ago
I've been working on a project that involves executing web crawling on a responsible scale but this has concerned me when I ramp to wider/regular fetching how will it be received by the web distribution mafia. I recall last time i set up my own mail server that it was very hard to do it in such a manner that emails from it were not automatically listed as spam
- finnthehuman 6y ago
  >web distribution mafia [...] set up my own mail server One of the tragedies of the tech powerhouses is that their decision making that effects the rest of us is often thinly-veiled power tripping that they've convinced themselves is for the good of everyone. When in reality, they just haven't delivered good enough software to eliminate the problem entirely. It's like a large scale version of a bad corporate computer security team. They can't quite design all the inconveniences out of their "solutions." Which is normally fine (after all what software and/or human-managed process is perfect the first time?), but out of a refusal to admit to their limitations they've convinced themselves that bureaucratic interactions with them to get day-to-day work done are necessary and important features of the design. There are genuinely technical challenges with no good technical solution, so political ones are employed. But sometimes you start getting too many middle managers that think the answer to every problem that requires systems thinking is just more self-assured tough-talking or "we could make more money by NOT fixing this." And then suddenly all the nice properties of the system slowly start disappearing or have conditional access gates.
wslh 6y ago
Are in some way robots.txt and those Cloudflare measures anticompetitive and/or against net neutrality?
- chris_f 6y ago
  >Are in some way robots.txt and those Cloudflare measures anticompetitive? Article by Matt Wells from Gigablast on Cloudflare. Gigablast has one of the largest independent search indexes. https://www.gigablast.com/blog.html https://www.gigablast.com/blog.html
  - kavalg 6y ago
    While I generally agree with Matt about the anti-competitive behavior of said tech giants, I just want to point out that: 1.Contents of robots.txt is created by the site owner, not those giants. 2.robots.txt is generally non-enforceable, unless someone like cloudflare translates it into proxy/DDoS settings (and can also actually distinguish a real google bot from a fake one). 3.Above behavior of cloudflare would be very problematic (in anti-trust terms) if this is not what the site owner wanted to achieve by writing said robots.txt. Now this is a (m/b)illion dollar question: *What is the site owner actually trying to achieve?* Are they only going for the DDoS protection or do they actually want to restrict other crawlers? Is it even legal to restrict other crawlers? From my limited experience I have come to several conclusions. First, there isn't a very big short term incentive for site owners to tolerate an infinite amount of crawlers and they usually don't bother. Heck, most of the times you talk to a SEO specialist, they only talk about Google. It is somehow assumed that if your site is indexed well in Google, all the other search engines (Bing, DDG ...) will work automagically. Second, there is this category of listing sites, where most of the value is in the database itself (e.g. LinkedIn, Craiglist, various property/car/younameit listings). They generally don't want to be crawled by their wannabe competitors. But is it legal to restrict anyone but Google/Bing? From a legal standpoint, one can explore the analogy between physical business and a website. There is public and private access. Public access is considered the case when people can just walk in into your office/store and "look around". You can take some measures against "misbehavers", but not allowed to restrict access to only "good" people. Furthermore, visitors are not bound by anything more than the country's law. On the other hand with private access you require a login and some form of a contract (e.g. ToS). In this case crawling and copying portions of the site may be specifically excluded in such ToS as long as it doesn't contradict the law. There are still some problems left though. Once you are blocked by such a data aggregator then the burden of proof is on your side and since there still isn't a clear ruling about these matters[1], how many average Joes will take it to the court? And even if you file it, how do you know the line between crawling and DDoS? [1] https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data

easytiger 6y ago

I've been working on a project that involves executing web crawling on a responsible scale but this has concerned me when I ramp to wider/regular fetching how will it be received by the web distribution mafia. I recall last time i set up my own mail server that it was very hard to do it in such a manner that emails from it were not automatically listed as spam

finnthehuman 6y ago
>web distribution mafia [...] set up my own mail server One of the tragedies of the tech powerhouses is that their decision making that effects the rest of us is often thinly-veiled power tripping that they've convinced themselves is for the good of everyone. When in reality, they just haven't delivered good enough software to eliminate the problem entirely. It's like a large scale version of a bad corporate computer security team. They can't quite design all the inconveniences out of their "solutions." Which is normally fine (after all what software and/or human-managed process is perfect the first time?), but out of a refusal to admit to their limitations they've convinced themselves that bureaucratic interactions with them to get day-to-day work done are necessary and important features of the design. There are genuinely technical challenges with no good technical solution, so political ones are employed. But sometimes you start getting too many middle managers that think the answer to every problem that requires systems thinking is just more self-assured tough-talking or "we could make more money by NOT fixing this." And then suddenly all the nice properties of the system slowly start disappearing or have conditional access gates.

wslh 6y ago

Are in some way robots.txt and those Cloudflare measures anticompetitive and/or against net neutrality?

chris_f 6y ago
>Are in some way robots.txt and those Cloudflare measures anticompetitive? Article by Matt Wells from Gigablast on Cloudflare. Gigablast has one of the largest independent search indexes. https://www.gigablast.com/blog.html https://www.gigablast.com/blog.html
- kavalg 6y ago
  While I generally agree with Matt about the anti-competitive behavior of said tech giants, I just want to point out that: 1.Contents of robots.txt is created by the site owner, not those giants. 2.robots.txt is generally non-enforceable, unless someone like cloudflare translates it into proxy/DDoS settings (and can also actually distinguish a real google bot from a fake one). 3.Above behavior of cloudflare would be very problematic (in anti-trust terms) if this is not what the site owner wanted to achieve by writing said robots.txt. Now this is a (m/b)illion dollar question: *What is the site owner actually trying to achieve?* Are they only going for the DDoS protection or do they actually want to restrict other crawlers? Is it even legal to restrict other crawlers? From my limited experience I have come to several conclusions. First, there isn't a very big short term incentive for site owners to tolerate an infinite amount of crawlers and they usually don't bother. Heck, most of the times you talk to a SEO specialist, they only talk about Google. It is somehow assumed that if your site is indexed well in Google, all the other search engines (Bing, DDG ...) will work automagically. Second, there is this category of listing sites, where most of the value is in the database itself (e.g. LinkedIn, Craiglist, various property/car/younameit listings). They generally don't want to be crawled by their wannabe competitors. But is it legal to restrict anyone but Google/Bing? From a legal standpoint, one can explore the analogy between physical business and a website. There is public and private access. Public access is considered the case when people can just walk in into your office/store and "look around". You can take some measures against "misbehavers", but not allowed to restrict access to only "good" people. Furthermore, visitors are not bound by anything more than the country's law. On the other hand with private access you require a login and some form of a contract (e.g. ToS). In this case crawling and copying portions of the site may be specifically excluded in such ToS as long as it doesn't contradict the law. There are still some problems left though. Once you are blocked by such a data aggregator then the burden of proof is on your side and since there still isn't a clear ruling about these matters[1], how many average Joes will take it to the court? And even if you file it, how do you know the line between crawling and DDoS? [1] https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data