21 ms·

Only Google is really allowed to crawl the web

• henriquez 6y ago
  I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to restrict bots, it would be much easier to restrict googlebot because they actually follow the rules. I don't disagree in principle that there should be an open index of the web, but for once I don't see Google as a bad actor here.
  • zmarty 6y ago
    A lot of news websites restrict any crawler other than Google. And this does not happen only via robots.txt.
    • simias 6y ago
      Indeed, years ago I had scripts to automatically fetch URLs from IRC and I quickly realized that if I didn't spoof the user agent of a proper web browser many websites would reject the query. Googlebot's UA worked just fine however.
      • judge2020 6y ago
        > Googlebot's UA worked just fine however They obviously don't care enough then - Google says you should use rdns to verify that googlebot crawls are real[0]. Cloudflare does this automatically now as well for customers with WAF (pro plan). 0: https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
  • throwaway_uat 6y ago
    LinkedIn profile/Quora answer are accessible by Google bot without signin
  • malf 6y ago
    What do you think this is used for? https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
  • calimac 6y ago
    The studies and data to support their claim is in the first paragraph of the article you "read" before posting the question.
  • Lammy 6y ago
    Spoofing your user-agent as googlebot is a common way to bypass paywalls, is (was?) a way to read Quora without creating an account, etc. Publishers obviously need to send their page/article to Google if they want it to be indexed but may not want to send the same page content to a normal user: https://www.256kilobytes.com/content/show/1934/spoofing-your-user-agent-as-googlebot-avoid-paywalls-circumvent-log-in-requirements-and-exploit-other-quality-loopholes https://www.256kilobytes.com/content/show/1934/spoofing-your... This was common even back in the mid-2000s: https://www.avivadirectory.com/bethebot/ https://www.avivadirectory.com/bethebot/ https://developers.google.com/search/blog/2006/09/how-to-verify-googlebot https://developers.google.com/search/blog/2006/09/how-to-ver...
  • soheil 6y ago
    It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked. Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.
    • deleted 6y ago
      [deleted]
    • WesolyKubeczek 6y ago
      I don't scrape a website often, but when I do, I'm using a user agent of a major browser.
    • Mauricebranagh 6y ago
      I have never had that problem running screaming frog on big brand sites apart from one or two times.
    • dheera 6y ago
      Do any of them intersect with Google Cloud IP addresses? If so set up a VPN server on Google Cloud.
    • tedunangst 6y ago
      I don't whitelist googlebot, but I don't block them either because their crawler is fairly slow and unobtrusive. Other crawlers seem determined to download the entire site in 60 seconds, and then download it again, and again, until they get banned.
    • quitethelogic 6y ago
      > Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them. Google says[1] they do not do this: "Google doesn't post a public list of IP addresses for website owners to allowlist." [1]https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
  • suicas 6y ago
    A company I worked for ~7 years ago ran its own focused web crawler (fetching ~10-100m pages per month, targeting certain sections of the web). There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time. We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was allowed to crawl them (though having no robots.txt configured to say so).
    • luckylion 6y ago
      I usually recommend setting only Google/Bing/Yandex/Baidu etc to Allow and everything else to Disallow. Yes, the bad bots don't give a fuck, but even the non-malicious bots (ahrefs, moz, some university's search engine etc) don't bring any value to me as a site owner, take up band width and resources and fill up logs. If you can remove them with three lines in your robots.txt, that's less noise. Especially universities do, in my opinion, often behave badly and are uncooperative when you point out their throttling does not work and they're hammering your server. Giving them a "Go Away, You Are Not Wanted Here" in a robots.txt works for most, and the rest just gets blocked.
      • d110af5ccf 6y ago
        > they're hammering your server Why can't you just ratelimit IPs that are "too active" for your server to handle?
        • MinorTom 6y ago
          CGNAT
          • d110af5ccf 6y ago
            Now that's a really good point. I wonder why there isn't a standard protocol for signalling upstream that a particular connection is abusive and to please rate limit the path at the source on your behalf? It would certainly add complexity, but the current situation is hardly better.
  • burkaman 6y ago
    See figure I.4 on page 24 of this UK government report: https://assets.publishing.service.gov.uk/media/5efb1db6e90e075c5674db35/Appendix_I_-_search_quality_v.3.pdf https://assets.publishing.service.gov.uk/media/5efb1db6e90e0... Additional evidence here: https://knuckleheads.club/the-evidence-we-found-so-far/ https://knuckleheads.club/the-evidence-we-found-so-far/
  • jameshart 6y ago
    When you operate commercial sites at scale, bots are a real thing you spend real engineering hours thinking about and troubleshooting and coding to solve for. And yes, that means google gets special treatment. Think about the model for a site like stackoverflow. The longest of long tail questions on that site: what’s the actual lifecycle of that question? - posted by a random user - scraped by google, bing, et al - visited by someone who clicked on a search result on google - eventually, answered - hopefully, reindexed by google, bing et al - maybe never visited again because the answer now shows up on the google SERP In the lifetime of that question how many times is it accessed by a human, compared to the number of times it’s requested and rerequested by an indexing bot? What would be the impact on your site of three more bots as persistent as google bot? Why should you bother with their requests? So yes, sites care about bot traffic and they care about google in particular.
  • noxvilleza 6y ago
    Google aren't the bad actor in the sense that they are actively doing something wrong, but they are definitely benefiting from the monopoly that they created and work on maintaining. If this continues then nobody will really ever be able to challenge them, which means possibly "better" products will fail to penetrate the market.
  • ehsankia 6y ago
    > but for once I don't see Google as a bad actor here. As inflammatory as the headline of the page looks, they literally admit it's not google's fault in the smaller text lower down: "This isn’t illegal and it isn’t Google’s fault, but"
• sxp 6y ago
  https://knuckleheads.club/the-googlebot-monopoly/ https://knuckleheads.club/the-googlebot-monopoly/ has actual details. > Let’s take a look at the robots.txt for census.gov from October of 2018 as a specific example to see how robots.txt files typically work. This document is a good example of a common pattern. The first two lines of the file specify that you cannot crawl census.gov unless given explicit permission. The rest of the file specifies that Google, Microsoft, Yahoo and two other non-search engines are not allowed to crawl certain pages on census.gov, but are otherwise allowed to crawl whatever else they can find on the website. This tells us that there are two different classes of crawlers in the eyes of the operators of census.gov: those given wide access, and those that are totally denied. > And, broadly speaking, when we examine the robots.txt files for many websites, we find two classes of crawlers. There is Google, Microsoft, and other major search engine providers who have a good level of access and then there is anyone besides the major crawlers or crawlers that have behaved badly in the past that are given much less access. Among the privileged, Google clearly stands out as the preferred crawler of choice. Google is typically given at least as much access as every other crawler, and sometimes significantly more access than any other crawler.
  • TheAdamAndChe 6y ago
    Are there any actual repercussions for just ignoring robots.txt?
    • 5560675260 6y ago
      Your crawler's IP might get banned, eventually.
    • the_dege 6y ago
      Sometimes website admins will also try to report your ips to the service provider as a source of attacks (even if not true).
      • DocTomoe 6y ago
        Given how often I've had misbehaving crawlers slow own servers in the early 2000s, I do not see how a crawler that disobeys robots.txt is not an attempted attack.
• sesuximo 6y ago
  Seems like a private cache of the web would solve the problem? Why does it need to be public?
  • lisper 6y ago
    Seriously? Google is a private cache of the web. That is the problem.
    • sesuximo 6y ago
      Google doesn’t give anyone access to said cache. I mean one crawler with a shared api among competitors. So exactly the same as the public cache, but run my a private company and accessed for a small fee
      • ajcp 6y ago
        > run [by] a private company and accessed for a small fee That is exactly the opposite of a public cache.
        • sesuximo 6y ago
          Not really. It serves the same function. Either you pay this hypothetical company or ??? pays to keep up the public one.
          • ajcp 6y ago
            Just because it serves the same function does not mean the implementation is the same. Private military contractors and a US infantry squad serve the same function, but the implementation completely changes their context. That being said what I think you're arguing for would be the implementation of a public utility or private-public business. If that's the case then yes, what you're saying is correct.
      • lisper 6y ago
        I don't think you're quite clear on what the words "public" and "private" mean. "Public" is not a synonym for "run by the government" and "private" is not a synonym for "closed to everyone but the owner". Restaurants, for example, are generally open to the public, but they are not public. A restaurant owner is, with a few exceptions, free to refuse service to anyone at any time. If it's "exactly the same as a public cache" then it's public, even if it is managed by a private company. The difference is not in who has access, the difference is in who decides who has access.
• dleslie 6y ago
  The idea of a public cache available to anyone who wishes to index it is ... kind of compelling. If it was the only indexer allowed, and it was publically governed, then enforcing changes to regulation would be a lot more straightforward. Imagine if indexing public social media profiles was deemed unacceptable, and within days that content disappeared from all search engines. I don't think it'll ever happen, but it's interesting to think about.
  • xtracto 6y ago
    That would be a very cool use case for something like STORJ or IPFS.
  • ISL 6y ago
    Imagine if Donald Trump decided that indexing Joe Biden's campaign site was unacceptable. A mandated singular public cache has potential slippery slopes.
    • whimsicalism 6y ago
      Imagine if Donald Trump decided to tax campaign donations to Joe Biden's campaign at 100%. I am unconvinced by the "slippery slope" argument being deployed by default to any governmental attempt to combat tech monopolies.
      • ISL 6y ago
        This is an argument against centralization more than it is against government. "One index to rule them all" seems more fraught with difficulty than, "large cloud providers are unhappy that crawlers on the open web are crawling the open web".
        • whimsicalism 6y ago
          If the impact stopped at "large cloud providers" being unhappy, I think that you're correct. But I think we've seen considerably downstream "difficulty" for the rest of society from search essentially being consolidated into one private actor.
• wunderflix 6y ago
  Even that won't change much. There is no way Google can be out-googled by other search engines because of its market dominance: more traffic means more clicks, more clicks mean better search results, better search results will drive more traffic. I try bing and DDG for a week or so every 6 months. I always switch back to google eventually because the results are so much better. Google can only be disrupted if something new is invented, something different than search but delivering way better results. I have no clue what that might be. But I hope someone is working on it.
  • Zelphyr 6y ago
    I've had the exact opposite reaction to the comparison between Google and DuckDuckGo. I use the latter daily and only rarely revert to Google. Even then I usually don't find the results to be any better and often find them to be worse. In my estimation, Google's search results have significantly declined in recent years.
    • wunderflix 6y ago
      Ha, maybe I should give it a try again :) My 6 months period is almost over again.
    • rstupek 6y ago
      Agreed. I've fully changed over to DDG on my phone and rarely add the !g to get a google search.
  • internetslave 6y ago
    Yup. My opinion has long been that the only thing that will take down google is a massive increase in NLP, such that the historical click data can be outperformed by a straight up really good NLP model
    • wunderflix 6y ago
      That's interesting. Is anyone working on this already? SV startup? And: don't you think Google is in the best position to build such a thing?
• jrockway 6y ago
  I think there are plenty of other people crawling the web. There's Common Crawl, there's the Wayback machine... it's not just Google. Then there is a very long tail of crawlers that show up in the logs for my small-potatoes personal website. Whatever they're doing, they seem to be existing in peace, at the very least. To some extent, I agree with this site that people are nicer to Google than other crawlers. That's because the crawl consumes their resources but provides benefits -- you show up on Google, the only search engine people actually use. But at the same time, they are happy to drag Google in front of Congress for some general abuse, so... maybe there is actually a little bit of balance there.
• tyingq 6y ago
  The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling. Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from. Airlines are a good example. If you search for flight status for a particular flight, Google presents that flight status in a box. As an end user, that's great. However, that sort of search used to (most times) lead to a visit to the airline web site. The airline web site could then present things Google can't do. Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it". Google took those eyeballs away. Okay, fine, that's their choice. But they don't give anything back, which removes incentives from the actual source to do things better. You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for. In short, I don't think the crawler is the problem. And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.
  • benatkin 6y ago
    That's the result of the crawling, and it preventing competition. Google would much prefer that people complain about the details while ignoring the root cause.
    • tyingq 6y ago
      I don't understand that. The crawling access is mostly the same as it ever was. Google's SERP pages are not. A mutually beneficial search engine that respects it's sources would still crawl the same. Google used to be that. The core problem is incentives: http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html "we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm."
      • 6y ago
• jinseokim 6y ago
  This has been submitted to HN quite a few times. https://news.ycombinator.com/item?id=25426662 https://news.ycombinator.com/item?id=25426662 (Most comments; 11 comments) https://news.ycombinator.com/item?id=25417067 https://news.ycombinator.com/item?id=25417067 (3 comments) https://news.ycombinator.com/item?id=25546867 https://news.ycombinator.com/item?id=25546867 (Most recent; 89 days ago) https://news.ycombinator.com/item?id=25543859 https://news.ycombinator.com/item?id=25543859 https://news.ycombinator.com/item?id=25424852 https://news.ycombinator.com/item?id=25424852
  • deleted 6y ago
    [deleted]
  • Darkphibre 6y ago
    Hooray! Looks like I'm one of today's lucky 10,000. :) https://xkcd.com/1053/ https://xkcd.com/1053/
  • skinkestek 6y ago
    Wasn't aware of that. Resubmitting interesting content that hasn't got traction earlier on is however explicitly allowed in the guidelines IIRC.
    • pessimizer 6y ago
      And linking past threads on the same subject is helpful.
  • monkeybutton 6y ago
    Interesting that the most comments it got before was 11, and today it succeeds and makes it to the front page! This is a good illustration of whether or not submissions get any traction can be fairly stochastic. On topic, stack overflow does exactly what the article is talking about; They lock down their sitemap and make special exceptions for the Google bot: https://meta.stackexchange.com/a/98087 https://meta.stackexchange.com/a/98087 https://meta.stackexchange.com/questions/33965/how-does-stack-overflow-protect-its-sitemap https://meta.stackexchange.com/questions/33965/how-does-stac... I can understand SO's reasoning but it only perpetuates the incumbents' stranglehold on the internet.
• tmcw 6y ago
  I've definitely scraped by this problem on several occasions. Recently I was writing a tool to check outgoing links from my site, to see which sites are offline (it's called notfoundbot). What I found was that many sites have "DDoS Protection" that makes such an effort impossible, other sites whitelist the CuRL headers, others like it when you pretend to be a search engine. Basically writing some code that tests whether "a website is currently online or offline" is much, much harder than you think, because, yep, the only company that can do that is Google.
• topspin 6y ago
  If the shared cache ever became significant enough to matter it would be devastated by marketers, scammers and other abusers. Google employs the groomers that make their index at least tolerable, if still clearly imperfect. Without that cadre of well compensated expertise to win the arms race against such abusers the scheme is not feasible. I suppose this could be crowdsourced if I didn't know about politics and how any attempt at delegating the responsibility for blessing sites and their indexes would become a controversy. Google takes lots of heat about its behavior already, but Google is a private entity and can indulge its private prerogatives for the most part. Without that independence this couldn't function.
  • finnthehuman 6y ago
    I don't really understand your comment. Marketers, scammers and other abusers already publish to the web with the intention to be included in a crawl. Postprocessing crawl data is already a thing. Assuming this hypothetical shared crawl cache were to exist, it does not preclude google (and all consumers of that cache) doing their own processing downstream of that cache. Does it? What's the new attack vector?
    • topspin 6y ago
      > I don't really understand your comment. If you don't then you fail to appreciate the amount of labor it takes to thwart bad actors from ruining indexes. Abusers do publish to the web, and we enjoy not wallowing in their crap because small army of experienced and expensive people at a select few Big Tech companies are actively shielding us from it. It's easy to anticipate the malcontent view; 'Google spends all its resources on ads and ranking and we don't need all that.' That is naïve; if Google completely neglected grooming out the bad actors people wouldn't use Google and Google's business model wouldn't be viable. So the obvious question is; where is this mechanism without Google et. al? Will the published caches be 99% crap (and without an active defense against crap you can bet your life it will) and anything derived from it hopelessly polluted? If so then it isn't viable. Now the instinct will be to find a groomer. Guess what; that's probably doomed too. No selection will be impartial to all, so you get to fight that battle. Good luck.
• mrweasel 6y ago
  While I don’t disagree with the idea that all crawlers should have equal access, we also need to address the quality of many crawlers. Google and Microsoft have never hammered any website I’ve run into the ground. Crawlers from other other, smaller, search engines have, to the point where it was easier to just block them entirely. Part of the problem is that sites want search engine to index their site, but not allow random people just scrapping the entire site. So they do the best they can, and forget that Google isn’t the web. I doubt it’s shady deals with Google, it’s just small teams doing the best they can and sometimes they forget to think ideas through, because it’s good enough.
  • rstupek 6y ago
    We've had the Bing crawler make a obscene number of requests quite often but fortunately it doesn't bring us down.
  • kmeisthax 6y ago
    I think this is a problem which should be solved by automatic rate-limiting and throttling at the application/caching layer (or just individual web server for smaller sites). Requests with a non-browser UA get put into a separate bots-only queue that drains at a rate of ~1/sec or so. If the queue fills up you start sending 429s with random early failures for bots (UA/IP/subnet pairs) that are overrepresented in the traffic flow. I don't know if such software exists, but it should. It would be a hell of a lot healthier for the web than "everyone but Google f*ck off", and it creates an incentive for bots to throttle themselves (as they're more likely to get a faster response than trying to request as fast as possible).
    • NathanKP 6y ago
      I suspect that at least some of the bots use web server response times and response codes as part of the signal for ranking. If your website does not appear capable of handling load then it won't rank as highly, because it is not in their best interests to have search results that don't load.
• soheil 6y ago
  I'm not sure if it is a good thing if there is a public cache of everything that Google has. The issue is websites will simply stop serving content to Google to protect their content from being accessed by their competitors, this in turn will make search much worse and will force us back to the pre-search dark ages of the internet. The sites may even serve an even more crippled version of their content just to get hits but there is no doubt search quality will suffer. We're left with a monopoly that is Google, destroying it now could be foolish.
• AlphaWeaver 6y ago
  This "club" charges a membership fee of $10 a month (or $100 a year) to comment. Does this go to some sort of nonprofit or holding entity that's governed by its members? Or do people have to trust the owner?
• anonu 6y ago
  > There Should Be A Public Cache Of The Web This might be closest to it: https://commoncrawl.org/ https://commoncrawl.org/
• ajcp 6y ago
  They really missed an opportunity to get creative with their own `robots.txt` implementation.
• herewhere 6y ago
  Around a decade ago, I was part of the team responsible for msnbot (a web crawler for bing). There used to be robot.txt (forgot the extension now). Most of the website was giving 10-20x higher limits to googlebot than rest other crawler. Google definitely has unfair advantage there. Bing and duckduckgo still provide very reasonable result with 10-20x less resources but not at par of google.
• staunch 6y ago
  Google makes $150+ billion from Google Search per year. Running Google Search could be operated for likely (much less than) $10 billion per year. So, Google is in effect taxing us all $140 billion per year. It's not dissimilar from how Wall Street effectively taxes us all for an even larger amount. In both cases, we could use some kind of non-profit open system to facilitate web search and stock trading. The Great Lie that Google is doing a good thing by charging money to insert "relevant ads" above the search results is totally wrong. If those ads are the most relevant, they should just be the top organic results, obviously. Google mostly solved search 20 years ago. There's really nothing that impressive about Google Search in 2021. It should be relatively easy to replace it with something open, leveraging the massive improvements in hardware and software. It could operate like Wikipedia or Archive.org. The hard part is probably getting the right team and funding assembled.
• dbsmith83 6y ago
  I just don't see this working out legally. How would it even work? From the "learn more" > Sometime soon we will be publishing what we think should happen and what we think will happen. These two futures diverge and we believe that, while the gap between them exists, it will entrench Google’s control over the internet further. We believe that nothing short of socialization of these resources will work to remove Google’s control over the internet. Our hope is that in publishing this work right now we will let the genie out of the bottle and start a process towards socialization that cannot be undone. Sorry, but I deeply skeptical of this. This sounds like the first step towards a non-free internet. At the end of the day, it is your box on the web, and if you want or don't want someone/something to crawl it, that is your call to make.
• marshmallow_12 6y ago
  I have an idea: remove the art of web crawling from the domain of a single company and instead create a international group of interested parties to run it instead. I'm thinking broadly along the lines of the Bluetooth SIG. Maybe it will be a bit more complicated, and require international political efforts, but it will make the search engine market way more democratic.
• rezonant 6y ago
  > Let’s take a look at the robots.txt for census.gov from October of 2018 as a specific example to see how robots.txt files typically work. This document is a good example of a common pattern. The first two lines of the file specify that you cannot crawl census.gov unless given explicit permission. This was eyebrow-raising. Actually seems like this is not (any longer?) true: https://census.gov/robots.txt https://census.gov/robots.txt: User-agent: * User-agent: W3C-checklink Disallow: /cgi-bin/ Disallow: /libs/ ... That first line wildcards for any user agent but does nothing with it. It should say "Disallow /" on the next line if it blocked all unnamed robots. It looks like someone found out about it and told the operators, rightfully so, that government webpages with public information (especially the census) shouldn't have such restrictions. They then removed only the second line and left the first. Leaving the first line has no impact on the meaning of the file.
  • rezonant 6y ago
    Actually, thinking more about this, I think they might be misconfigured, because they clearly don't want robots touching /cgi-bin etc (reasonable!) but they are actually only asking the named robots to do that, all other bots have no guidance about what not to touch
• dclaw 6y ago
  I can't really trust a website that spells its own name wrong on their homepage. "Knucklesheads’ Club" Edit: https://imgur.com/a/inqYrjV https://imgur.com/a/inqYrjV
  • slenk 6y ago
    Everyone makes mistakes
• lawwantsin17 6y ago
  I'm all for killing Google's monopoly but spiders can ignore robots.txt you know. This just seems like a failure of other companies to effectively ignore those.
• tomc1985 6y ago
  I think the solution here is everybody masquerades as Googlebot so we can render the whole thing moot
  • quantumofalpha 6y ago
    Ignoring robots.txt is trivial, that's why some(many?) sites enforce it by verifying source IP and recognize Googlebot from its IP addresses - how will you get access to one of those?
    • p-sharma 6y ago
      What does "recognize Googlebot from its IP addresses" mean? If I'm a human and I access a site, I have some other IP than Googlebot, how should this side know if I'm a human or knuckleheadsbot?
      • smarx007 6y ago
        https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
      • quantumofalpha 6y ago
        if you're claiming to be User-Agent: Googlebot, but your IP doesn't seem like it belongs to Google, don't you think it's a clear sign that you're FAKING IT? The check itself could be implemented for example with ASN or reverse DNS lookup or hard-coding known Google's IP ranges (though that's prone to become stale)
• samizdis 6y ago
  Coincidentally, this item [1] has just turned up in HN - Common Crawl [1] https://news.ycombinator.com/item?id=26594172 https://news.ycombinator.com/item?id=26594172
• systemBuilder 6y ago
  This is not really about Google. Websites block crawlers because they get abused / crashed by Crawlers. In the early days (2000-2010) Google not only got banned by some websites, it even got DNS-banned for abusing some DNS domains. You see, Google already has already built the "megacrawlers" described in this article, it can melt any website on the Internet, even Facebook - the largest, and they paid a high price for letting the early Google crawlers run free. Google today has a rate-limit for every single website and DNS sub-domain on the internet. For small websites the default is a handful of web pages every few seconds. Google has a very slow (days) algorithm to increase its crawl rate, and a very fast (1d) algorithm to cut the rate limit if it's getting any of the errors likely due to website overload. To summarize, Google has several layers of congestion control custom-designed into the crawl application. Most small web crawlers have zero. None of these other crawlers have figured this out, so they abuse websites, causing all small-scale crawlers to get banned. - ex-Google Crawl SRE
  • ricardo81 6y ago
    Thank you for those insights, it's a topic I'm interested in. Agree with what you're saying about naive bots hitting websites/hosts/subnets too hard, in the context of site owners being hit by multiple bots for multiple reasons and them questioning the return they'll get. I'd be interested to know more info wrt DNS lookups. Did you apply a blanket rate limit on the number of DNS requests you'd make to any particular server? From past experience I know the .uk Nominet servers would temp-ban if you were doing more than a few hundred lookups per second. At the next host level down, was there a blanket limit or was it dependent on the number of domains that nameserver was responsible for?
• hannob 6y ago
  I have seen sites behave differently if you use a Googlebot UA, but am I missing something or does this merely mean that anyone doing something like this curl -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html http://www.google.com/bot.html)' will get Google-level crawler access?
  • kirubakaran 6y ago
    That would work on website that have a naive check for just user agent. Google also publishes the IP address ranges their crawlers run on. Lot of websites check for that, and there's no way around that. https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/...
• villgax 6y ago
  The irony is that they bitch about you not scraping search or other platforms without paid plans & want to do the same to you
• nova22033 6y ago
  This isn’t illegal and it isn’t Google’s fault Right there in the article..
  • WarOnPrivacy 6y ago
    Again, with critical context. This isn’t illegal and it isn’t Google’s fault, but this monopoly on web crawling that has naturally emerged prevents any other company from being able to effectively compete with Google in the search engine market.
• cookiengineer 6y ago
  Can we take a moment to talk about this club's business model? There's not even any information to see what the "private forum access" that you have to pay for is about, what kind of people are in it...or even to know about what happens with the money. For me, this sounds like a scam. I mean, no information about any company. No imprint. No privacy policy. No non-profit organization. And just a copy/paste wordpress instance. I mean, srsly. I am building a peer-to-peer network that tries to liberate the power of google, specifically, and I would not even consider joining this club. And I am the best case scenario of the proposed market fit.
  • adamdusty 6y ago
    They want you to pay them to "research" google's web crawling monopoly. It's really just a donation, but they don't frame it like that. Probably more credible than using a crowd funding website, because it sounds like their pushing for actual legislation. > Meet with legislators and regulators to present our findings as well as the mock legislation and regulations. We can’t expect that we can publish this website or a PDF and then sit back while governments just all start moving ahead on their own. Part of the process is meeting with legislators and regulators and taking the time helping them understand why regulating Google in this way is so important. Showing up and answering legislators’ questions is how we got cited in the Congressional Antitrust report and we intend to keep doing what’s worked so far.
  • judge2020 6y ago
    Not being set up as a 527 nonprofit[0] is the biggest red flag - no donation or membership money has to be spent for political purposes. They also use memberful for their membership/payment system, which doesn't require owning a business, so you might be paying out to the owner directly instead of to a business with its own bank account. Maybe the owner is looking at HN and can clarify. To add, there are a lot of businesses that use the terms 'Knucklehead' so finding their business on secretary of state business searches might be impossible. 0: https://www.irs.gov/charities-non-profits/political-organizations https://www.irs.gov/charities-non-profits/political-organiza...
• p-sharma 6y ago
  Maybe a naïve question but what prevents Knuckleheads’ from ignoring the robots.txt and crawl the side anyway? And if it's so easy to do, how does Google have a monopoly on crawling then?
  • judge2020 6y ago
    It's just rude to do so, and there are some technical issues with doing that as well (such as crawling admin panel which might trigger backend alarms/security alerts). Google also doesn't have a legal monopoly on crawling, only a natural monopoly thanks to a lot of websites independently choose to only allow Google and Bing because of the many issues with third-party crawlers (eg. crawling all pages at once, costing money/slowing down the site[0]). 0: https://news.ycombinator.com/item?id=26593722 https://news.ycombinator.com/item?id=26593722
  • foobar33333 6y ago
    On smaller sites, nothing usually. But on bigger sites you will be blocked. You will probably be blocked even if you do follow robots.txt
  • sn_master 6y ago
    Their IP range can be blocked. Google has known IP ranges and if the website admin wants can allow them based on that rather than robots.txt, and that would really mess up everyone else.
• graiz 6y ago
  Not sure why http://commoncrawl.org/ http://commoncrawl.org/ wasn't mentioned.
• dheera 6y ago
  > Only a select few crawlers are allowed access to the entire web, and Google is given extra special privileges on top of that. Hmm, so set up a VPN on the Google Cloud so you have a Google IP address, use a Google User-Agent, and go!
  • jesboat 6y ago
    https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/... describes the procedure for checking "is this source Google it". You couldn't fake it just by running on gcp
• drivingmenuts 6y ago
  How about a system whereby we tell others whether or not we want to be crawled/not crawled by them? /s
• andrewclunn 6y ago
  How about an opt-in search engine cache? One where a domain needs to agree allow their site to be crawled, but as a result also gives said crawler full access? And then that repository would be made publicly available to all search engines to use. Sort of an AP for searches, that would give a base line that wouldn't preclude search engines from going further, but which would certainly lower the cost and network traffic for the search engines and sites that take advantage of it?
• zepearl 6y ago
  Maybe it would be nice if some sort of simple central index of "URLs + their last updated timestamp/version/eTag/whatever" would exist, updated by the site owners themselves? ("push"-notification) Meaning that whenever a page of a website would be created or updated, that website itself would actively update that central index, basically saying "I just created/deleted page X" or "I just updated the contents of page X". The consequence would be that... 1) ...crawlers would not have anymore to actively (re)scan the whole Internet to find out if anything has changed, but they would only have to query that central index against their own list URLs & timestamps to find out what needs to be (re)scanned. 2) ...websites would not have to just wait&hope that some bot would decide to come by to have a look at their sites, nor they would have to answer over and over again requests that are just meant to check if some content has changed.
• ThePhysicist 6y ago
  On a related note, Cloudflare just introduced "Super Bot Fight Mode" (https://blog.cloudflare.com/super-bot-fight-mode/ https://blog.cloudflare.com/super-bot-fight-mode/) which is basically a whitelisting approach that will block any automated website crawling that doesn't originate from "good bots" (they cite Google & Paypal as examples of such bots). So basically everyone else is out of luck and will be tarpitted (i.e. connections will get slower and slower until pages won't load at all), presented with CAPTCHAs or outright blocked. In my opinion this will turn the part of the web that Cloudflare controls into a walled garden not unlike Twitter or Facebook: In theory the content is "public", but if you want to interact with it you have to do it on Cloudflare's terms. Quite sad really to see this happen to the web.
  • judge2020 6y ago
    On the other hand, I do not want my site to go down thanks to a few bad 'crawlers' that fork() a thousand http requests every second and take down my site, forcing me to do manual blocking or pay for a bigger server/scale-out my infrastructure. Why should I have to serve them?
    • progval 6y ago
      You can use the same rate-limiting for all crawlers, Google or not.
      • dodobirdlord 6y ago
        Googlebot is pretty careful and generally doesn’t cause these problems.
        • spijdar 6y ago
          Right, then they shouldn't be effected by the rate-limiting, as long as its reasonable. If it was applied evenly to all clients/crawlers, it'd at least allow the possibility for a respectful, well designed crawler to compete.
• l72 6y ago
  I tried to set up YaCy [1] at home to index a few of may favorite smaller websites, so I could quickly search just them. That turned out to be a bad idea. Some ended up blocking my home IP address and others reported me to my ISP. None of these sites were that large, and I wasn't continuously crawling them... [1] https://yacy.net/ https://yacy.net/
  • jedimastert 6y ago
    How often were you searching?
    • l72 6y ago
      I was regularly searching, but I was rarely indexing any of the sites. I struggled to even get an initial index of many of the sites, due to being blocked or being reported.
      • jedimastert 6y ago
        If you were getting blocked on an initial load, you were either hit with rate limiting or an unrecognized user agent
  • slenk 6y ago
    I have been running my own Searx instance in AWS for a while and have not gotten blocked yet anywhere
    • lgats 6y ago
      Wow, I've never heard of Searx - https://searx.space/ https://searx.space/ It's privacy friendly proxied search results, a lot like the Nitter project for Twitter. https://github.com/zedeus/nitter/wiki/Instances https://github.com/zedeus/nitter/wiki/Instances
• deleted 6y ago
  [deleted]
• EGreg 6y ago
  Or use MaidSAFE where you get paid to serve your website as opposed to the other way around.
• ChrisArchitect 6y ago
  dupe/posted earlier etc I also got confused about this page as there's another project of theirs around right now about RIP Google Reader that's on a seperate domain... Funny a site that's all about google this and that doesn't have clear URL/pages for their articles that can be linked to easily geez Original post/discussion from the source, 3 months ago: https://news.ycombinator.com/item?id=25417067 https://news.ycombinator.com/item?id=25417067
  • slenk 6y ago
    https://knuckleheads.club/introduction/ https://knuckleheads.club/introduction/ That seems like an easy link?
• sgsvnk 6y ago
  Money earns more money. Privilege begets more privilege. This is not just true in the case of Google but in other other domains as well like the financial markets. Would you blame capitalism?
• varispeed 6y ago
  I disallow scanning on all my projects. After GDPR I also removed all analytics - I realised it is just a time sink - instead of focusing on content I would often focus on getting the bigger numbers. I am not a marketer, so it didn't have much value to me and it would just enlarge Google dataset without any payment. I get that you cannot find my projects in the search engine. I am okay with that :-)
• jeelecali 6y ago
  I'm looking for $ 576
• mancerayder 6y ago
  Any word on or opinions about Brave's initiative to challenge search?
  • aembleton 6y ago
    Yes, there was a large discussion here https://news.ycombinator.com/item?id=26328758 https://news.ycombinator.com/item?id=26328758
• arbirk 6y ago
  so you want money. what about telling who you are and what the money specifically supports. and why not a free join tier?
• ExcavateGrandMa 6y ago
  google is you government... it doesn't need you...
• renonce 6y ago
  I believe there is a lot of hidden fight behind the scenes for Google to monopolize the web. There are a lot of expectations from the public that Google maintains. Apart from delivering the best search results, they have to respect robots.txt, limit crawling frequency, deliver search results really fast, etc. At the same time, lots of people want to game Google and spam search results so they can make money with ads. The competition is not fair at all - the websites can tell which traffic came from GoogleBot and craft legitimate responses while Google is not allowed to publicly crawl the website with a fake User Agent. Many websites are concerned about their data being crawled (like Amazon, they certainly wouldn't be happy if all their price information is dumped as a database), and Google has to make sure that no robot can crawl too much of a website by automatically searching Google, and to that end they invented reCATPCHA. It's not easy at all to build a search engine that behaves responsibly both to websites and to users. It's the ability to deal with all these matters really well that gives Google the monopoly power.
• sn_master 6y ago
  And those spiders can sometimes inadvertently act as a DDoS attack. https://security.stackexchange.com/questions/16609/spider-attack-from-several-search-engines-at-once https://security.stackexchange.com/questions/16609/spider-at...
• jedwhite 6y ago
  I'm working on a search-related project. There is a paradox here. Many content producers hate Google taking their content without giving them a fair share of the revenue that search generates. But they block any competitors to Google from spidering or crawling their websites, or rate-limit programmatic queries. The way things work in practice, much of the web tries to prevent any type of programmatic access though a combination of edge-tech and robots.txt policies. Content producers are addicted to Google and block competition, even though it is killing them. As Shakespeare put it: "Like rats that ravin down their proper bane, A thirsty evil; and when we drink we die."
• kova12 6y ago
  Public cache is an awful idea. It initially sounds like a great public good, but many of us can see where it will go: - google will still have some "extra" crawling to keep his monopoly - everyone else would be fighting for an access to said cache, which will not be able to carry everyone who wishes. So, there will be rationing and favors - that will quickly become a bureaucracy which would envelop every aspect of internet activity - "undesirable" sites will be easily ejected from cache and forgotten forever - you will end up with a single entity paid by public, can't go bankrupt, in charge of whole internet. Google will go bankrupt if they don't satisfy people - these people don't even have to do a good job - there's nobody else on the market (remember, we started with the need to eject google? This will eventually happen).
• londons_explore 6y ago
  This argument would be made a lot stronger with numbers... What percentage of the top million webpages allow Googlebot? What percentage allow other robots? Why not simply pretend to be Googlebot? After all, browsers pretend to be Mozilla...
• Mikhail_Edoshin 6y ago
  A somewhat offtopic question: why isn't there a lighter version of HTTPS that only signs messages, but doesn't encrypt them? This way it would be possible to cache the content and use the cache for indexing.
• sailtronus 6y ago
  If you could understand, that what you see in a Google search is a manipulated reality, an algorithm created to their oun profit and believes, you would never look back, or would, if you was agent Smith. But my friend, all US companies are in the same row, pointing to the same direction, guided by the oracle of Inteligent Board of Psycopath Agents, who thinks are defending US nation. If you want to see what is ouside the chains of FBIntranet, you need be smart, all the easy things (tor, vpn...) are maded to catch lasy people.