14 ms·

Dear Google, I'm Blocking You from My Website

431 points by bradley_taunt 7y ago

notyourday 7y ago
"Market can be irrational for longer than you can remain solvent"
jgmmo 7y ago
yeah but then how do you get organic traffic?
- deleted 7y ago
  [deleted]
- LyndsySimon 7y ago
  On a related note, what does organic search traffic look like these days? I was once personally invested in running several small content sites and at the time organic traffic was pretty much entirely from Google. There were other inbound sources - Bing, Yahoo!, etc. - but they were so negligible that it would have been reasonable to ignore them completely for the purposes of optimization. I've been using DDG as my primary search engine now for several years, and using Google no longer "feels like the Web" for me. I imagine I have one hell of a case of observation bias in that, but it does seem like alternatives have been gaining some traction in recent years.
  - jgmmo 7y ago
    I agree, I use Startpage.com That said, looks like most of the US does not feel the same way about Google. I still see 90-99% of organic traffic coming from Google for my client sites.
  - heyyyouu 7y ago
    It's still 90-95 percent Google, surprisingly. You'd think it would be less with mobile but no. We are seeing more duck duck go, but bing and yahoo have stayed steady for years. Social is a rise, and that's a lot of mobile. Also then just pickups (links from sites like this). But yeah, still very much google dominated.
bhartzer 7y ago
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are removed from the index, block the google IPs.
- joering2 7y ago
  How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
  - edm0nd 7y ago
    You can block Googlebot by User-Agent and IP addresses.
  - bhartzer 7y ago
    There are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
    - LunaSea 7y ago
      Wouldn't that be considered illegal due to the same reasons as scraping?
      - hnick 7y ago
        I don't think scraping is illegal, unless you're hitting protected pages behind a login after agreeing to a Terms of Service, or hitting it so often you'd constitute a DOS attack. An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.
dna_polymerase 7y ago
"I don't like one of their products so I don't want anyone to find the stuff I write online" Unless you are dependent on SEM, AMP doesn't really hurt you. This whole no-index thing is ridiculous and not at all possible for any sufficiently large website/company. Only thing I can agree with is the alternative search engine point. We really need more competition in the field, so Google is forced to cater to it's user's wishes more instead of purely pushing their agenda on them.
ipsin 7y ago
Are they actually blocking Google somehow? No robots.txt, no 403 when using a Googlebot user agent to fetch the site page...
- sieabahlpark 7y ago
  Googlebot uses the Google ip range
- einr 7y ago
  Yes. <meta name="googlebot" content="noindex"> Googlebot looks for this meta tag.
rahuldottech 7y ago
> As fewer results of such not-for-profit websites show up in Google Search, competition will be reintroduced to the search market. I don't think it will. While I personally hate AMP, and disagree with many of Google's practices, this is not the solution. Google is everywhere. It is the default search engine on almost every browser, as well as on most devices. How often do you hear someone say "look it up" as opposed to "Google it"? The latter is much more common in my experience. Google is synonymous with internet searches. Outside of tech circles, no one is going to make an effort to even try other search engines. There's a reason Bing and Yahoo are treated as a joke. Google is also the best at searches. While other search engines are slowly catching up, and Google's own results are becoming worse, it's still way better then most search engines - especially when it comes to specific and local/regional results. Lastly, blocking Google means that you site also won't show up on Startpage.com results, because it sources results from Google. This is unfortunate, since Startpage is definitely one of the search engines that respect your privacy the most.
- allknowingfrog 7y ago
  I'm not saying it proves anything, but Kleenex is synonymous with facial tissue and Velcro is synonymous with hook and loop fasteners. I "google" things on DuckDuckGo. :)
  - rahuldottech 7y ago
    Yeah, I was just pointing out that to most people, ["google" == "web search"]
  - r3bl 7y ago
    Yeah, I've been using DDG for years, but I just can't get "Google it" out of my dictionary. I try to replace it with "Duck it", which feels stupid as fuck to say, but "Google it" doesn't feel any better.
    - lostmsu 7y ago
      I don't think it is a problem. Xerox is a thing.
LyndsySimon 7y ago
> Google needs to stick to being a solid search engine instead of trying to be gate keepers While I agree with the sentiment, I feel like this ship sailed years ago - it's like saying "Amazon should stick to being an ecommerce platform". What's more, Google was never in the business of "being a solid search engine". Google is and always has been in the business of selling access to users' eyeballs. AMP - as much as I hate it - is directly in line with their core business.
- PavlovsCat 7y ago
  > What's more, Google was never in the business of "being a solid search engine". Google is and always has been in the business of selling access to users' eyeballs. That's not how they became the darling of the tech scene though. http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html > Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users [..] we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers. Since it is very difficult even for experts to evaluate search engines, search engine bias is particularly insidious. [..] we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm.
  - liberte82 7y ago
    When Google was first released it was revolutionary. I remember just being amazed at the quality of the search results compared to any other engines. And amazingly, the competitors still haven't caught up, for all of Google's issues.
    - lunchables 7y ago
      So true, the difference was monumental. Pagerank was a huge breakthrough that seems so simple and so obvious in hindsight, it's clear why it changed everything. The popularity of a website shouldn't be based on the number of times they can stuff keywords into a page, it should be based on the number of human beings who used the site and determined it was valuable enough to link to. I don't know if pagerank is used anymore, but at the time, with the state of the web then, it was an incredible breakthrough.
JohnC8 7y ago
It starts with one, Google abused their power too many times. Get a good domain name and direct traffic. That is all.
wil421 7y ago
Can anyone remember if the old AOL landing page linked to AOL created content or content by 3rd parties? I’ve made the argument before that Google is the new AOL homepage. It’s the gateway to the web for a huge majority as AOL was at one time. AMP is concentrating Google’s web just as AOL once did for first timers. Google has much more utility but it’s still very similar. AOL had email, web, chat, news, shopping? (maybe late days of AOL), and let’s not forget search.
lqqkout4elfy 7y ago
This site loads amazingly fast! Proves the point that you really don't need AMP.
- kypro 7y ago
  You do if you want Google to send any traffic to it. Your only option is to bend the knee unless you're site is nothing more than a blog which you do for fun.
- dddddaviddddd 7y ago
  All that AMP does for speed is impose limits on CSS and JS size (plus givs access to Google CDNs). You can simply write small assets without the need for Google to specify this.
  - cameronbrown 7y ago
    I've recently built an AMP version of a marketing website (since AdWords supports AMP cache, thought it would be useful for speed). The truth is, while AMP as a library does provide solid speed improvements on its own by implementing all the best practices, the real speed improvements is the cache with preloading - as well as the other stuff it does like image optimisation.
mateus1 7y ago
Google once was a force for good by indexing pages and rewarding good content with SEO. I deeply hate AMP. It's a monopolistic move to bully companies into handing over Google their content.
- lern_too_spel 7y ago
  > It's a monopolistic move to bully companies into handing over Google their content. That's nonsensical. Google already has their content. If anything, AMP is a democratizing technology. If you want your articles in Apple News, you have to integrate with Apple. Same with Facebook Instant Articles. With AMP, you don't have to integrate directly with anybody. Just publish AMP pages from using standard HTTP, and every link aggregator that implements an AMP cache (including Bing, Baidu, Yandex, Yahoo Japan, etc.) will be able to instantly load your pages for users. If I start my own link aggregator, I don't need to have the clout of a Facebook, Apple, or Google to get publishers to enable instant loading from my site — I can cache the AMP pages they have already published. It's the same as publishing bus routes according to Google Maps's standard feed format (https://developers.google.com/transit/gtfs/ https://developers.google.com/transit/gtfs/) . I publish once in that format, and not only do my bus routes show up better in Google, but they also show up in all other mapping applications that ingest that format, without me having to do any integration with them.
  - thepangolino 7y ago
    AMP is a complicated answer to an easy one. AMP was presented as an answer to webpages not being lightweight anymore. Instead of just pushing for more lightweight webpages, Google cane up with the monstrosity that is AMP.
    - lern_too_spel 7y ago
      > AMP was presented as an answer to webpages not being lightweight anymore. Wrong. AMP is a solution to web pages not loading instantly, not a solution to web pages not being lightweight. If you want to prerender links above the fold without deanonymizing them to the publisher, you will come up with AMP pretty much exactly.
echeese 7y ago
There's an easier way - don't implement AMP.
- t0mbstone 7y ago
  Yeah, I don't understand this post. There are billions of pages out there without AMP and Google indexes them just fine. If you don't wanna implement AMP, then just don't implement it. You don't have to completely block Google over it.
  - deleted 7y ago
    [deleted]
  - heyyyouu 7y ago
    Exactly! I am so confused as well. It's like this person doesn't understand that AMP is optional. And even then there's ways to instruct Google NOT to point to your AMP page. (And no, you don't need AMP for SEO -- it's only a slight boost)
    - bluGill 7y ago
      The problem is if your competitor has an AMP page google will rank them higher than you. That will cost your money.
      - cameronbrown 7y ago
        AMP only puts you in the top stories carousel. Speed is measured neutrally for organic search results.
        heyyyouu 7y ago
        THIS. I don't understand how so many people don't actually get how Google and AMP work. You don't need AMP for organic search results. It's used by Google, yes, but only for certain things AND there's ways to turn it off/point to non-AMP pages. We choose not to use it, other publishers do use it, but everyone looks at the pros and cons and then just decides. It's like any other business decision, but you have to understand how it actually works first and not just go "Oh, AMP's bad" or "Oh, without AMP my site is dead.) I don't believe either of those is true.
TazeTSchnitzel 7y ago
Is it possible to make an AMP page that just redirects to a non-AMP one?
- heyyyouu 7y ago
  That is one technique some people use.
speeder 7y ago
I am currently struggling with Google and been even considering just straight up creating a Brazilian "Yandex/Baidu" clone or something somehow. My current business is heavily Google reliant, and Google been proving to be a very poor business partner: * Keeps punishing my site both in organic rank and AdWords rank (thus price) for not being mobile, despite my site being intended to be viewed by engineers with big screens. * They remove frequently whatever features they had to avoid wasting money, the one that offended me the most is that I can't anymore advertise on mobile websites unless I also allow apps too, and apps are purely fraudulent clicks, babies don't buy industrial parts... * Google refuses to take care of fraud, I've visited the top sources of clicks on my display campaigns and there was comments for example on store pages explicitly complaining it was pure fraud, and Google does nothing about it (example: one source of clicks was a flashlight app on Android that opens ads whenever the user turns the flashlight on, clicking the ad is only way to turn it off again) * Google often randomly stop my ads and search because bogus reasons, requiring me to call them to fix it. * They removed from AdWords interface the max click price when the campaign is in automatic mode, in the past this was very useful to find out if their AI was bugged (like one time it insisted on setting max cpm to 0.01 despite average cost until that day being 1.20...) or oberpricing me (charging ludicrous prices per click) Their product (ad publishing, not search) is crap, but they have literal Monopoly, all other ads publishers I tried gave me almost zero clicks... Only thing I don't tried yet is more traditional ads, but I don't have money for that (for example TV ads) I honestly don't know why so many governments with strong antimonopoly laws tolerate Google shenanigans.
- whatshisface 7y ago
  >I honestly don't know why so many governments with strong antimonopoly laws tolerate Google shenanigans. To build an anti-monopoly case you would have to do more than show that Google was the only good ad vendor. Are they the only ones that can give you clicks because they are smarter and better at their jobs than every other ad vendor, or are they the only ones that can give you clicks because of underhanded market tactics? If I was the best shoemaker in town, to the point where buying shoes from anybody else was a waste of time, that wouldn't be illegal. You'd have to show I was doing more to stop competition.
JohnFen 7y ago
Way ahead of you. I started blocking Googlebot from my websites a few years ago, and never looked back!
rangeofmotion 7y ago
I really wonder about the people who work at Google. Surely they have people whose ideals are higher than this? It seems obvious that Google is a real threat to the freedom the internet originally made possible. But engineers still sign up in droves to work there? What kind of engineers are these? Do all of Google's employees simply disagree that Google is a threat? Do they not care? Are they just financially scared and willing to sell everyone else out so they can pay their mortgage? Are there efforts internally to try and stop them from centralizing everything? What's going on here? I know only a couple of people who don't think Google is a real problem today.
- bsimpson 7y ago
  I don't work on AMP (I've never even written an AMP page); this is purely my personal perspective: The difference is trust: These AMP rants invariably look to the use of Google's CDN, or to the "google.com" domain in the address bar, and infer malintent. I presume the team had good intentions, particularly for end users. The AMP team saw a problem (websites take a stupid amount of time to load on mobile, even on nice phones/networks) and devised a solution: "Google has one of the best network infrastructures on the planet. It serves things quickly. Let's cache pages that don't do slow things on that network, so they're fast for users." They devised some criteria for what they mean by "don't do slow things," and wrote tooling to assert it. I don't believe the intent of the AMP team is to strongarm developers into using their framework, or to funnel all traffic through Google. I trust that they are well-intentioned people who are trying to do the right thing. They're not proud of the limitations of their original solution, and are making progress on fixing them: for instance, they pushed forward a new web standard (packaging) to fix the address bar problem: https://blog.amp.dev/2018/05/08/a-first-look-at-using-web-packaging-to-improve-amp-urls/ https://blog.amp.dev/2018/05/08/a-first-look-at-using-web-pa... --- Many words have been written about the ramifications of echo chambers on our political discourse. Those same ideas apply here on HN too. The more I see hyperbolic comments presuming everyone/thing is evil/bad/stupid, the less interesting this place becomes for me -- the less likely I am to come here. As that culture drives people like me away, the ratio of conspiratorial-armchair-quarterbacking:reasonable-discourse tilts further towards the negative. It drives even more people away, leaving a concentration of negativity. "Don't read the comments" starts to apply here too. That's not a reaction to your comment, in particular: it's how I've started to feel about a lot of Hacker News, especially when my employer is the topic. When people presume the absolute worst -- in spite of more reasonable (and more likely) alternatives -- there's nothing fun to read or interesting to learn. I lose reasons to keep coming back.
heyyyouu 7y ago
I'm very confused by this article -- AMP is an optional framework: https://amp.dev/ https://amp.dev/ and even if you use it you can opt out of having Google load/point to it: https://support.google.com/news/producer/answer/7281462?hl=en https://support.google.com/news/producer/answer/7281462?hl=e... You don't need to no_index your site. What can you do about it? Just don't implement AMP. Yes, sites who do AMP get preferential treatment in search engines, but you can still get excellent results without it; it's a very small boost.
- rahuldottech 7y ago
  It's a matter of principle, AMP is a terrible thing for the web in general.
  - heyyyouu 7y ago
    Yes, a lot of people agree on that. But you don't need to block Google from your site to stop AMP.The way this article is written it makes it seem like the only way to have AMP not affect your site is by blocking Google entirely, and that's simply not the case. I think that's why those of us who understand how AMP is used are commenting here -- and you can see that people are confused because other people are posting questions about blocking amp, when you don't have to block amp, you don't use it in the first place. The author should clarify that they're taking extreme measures, but that this isn't required to not use AMP.
    - anecdotalShit 7y ago
      the article clearly states. The reason to block is to improve competition. The lack of competition is the sole reason why google can stongarm amp instead of an open standard
- playpause 7y ago
  > Just don't implement AMP. ... you can still get excellent results without it; it's a very small boost. What evidence do you have that it's only a small boost? That may be the case for very specific searches. But if you search anything where there is a lot of competition between publishers (coverage of major news stories, product reviews, etc), Google serves almost exclusively AMP results, with the exception of the odd result from Wikipedia or Twitter. Failing to implement AMP would be suicide for many publishers.
meerita 7y ago
Google must offer an option to opt out from those kind of products and at the same time, offer the search result. I am interested in this, I don't want them to replace me for their own UIs.
- heyyyouu 7y ago
  AMP is optional. You don't have to implement it. Even if you do there's ways to instruct Google when to use it/when not to.
  - meerita 7y ago
    Yes, but there are other things that they already display in their search results without opting.
Proven 7y ago
Good move, kudos to Bradley
Circuits 7y ago
Yeah alright I can get behind this. I don't want to support a company looking to monopolize the market. I will start looking into how I can go about cutting Google out of my life.
jammygit 7y ago
If he’s going offline from search, he should create an rss or mail list. I didn’t see where to subscribe, so honestly I will probably forget his site exists in a week...
AlexandrB 7y ago
I like this. It reminds me of the pizza shop that gave 50% off to 1-star Yelp reviewers[1] or the ice cream truck that charges "influencers" double[2]. It's a statement of "I don't like your system, so I'm going to opt out of the whole thing even if it potentially harms me." [1] https://thehustle.co/botto-bistro-1-star-yelp/ https://thehustle.co/botto-bistro-1-star-yelp/ [2] https://metro.co.uk/2019/07/03/ice-cream-seller-charges-double-for-every-influencer-who-asks-for-a-freebie-10109479/ https://metro.co.uk/2019/07/03/ice-cream-seller-charges-doub...
- deminature 7y ago
  It's not going to potentially harm him, not being listed by Google is arguably equivalent to ceasing to exist. It's a nice gesture, but I cannot imagine how this is good idea in practice, regardless of how you view Google ethically.
  - pmlnr 7y ago
    Yeah, no. Ultra-Local business - eg. ice cream truck - really don't need a google presence in any way.
    - alanbernstein 7y ago
      Yeah, no. A programmer's blog is not an ultra-local business.
      - pmlnr 7y ago
        Parent edited.
  - AlexandrB 7y ago
    If your blog is mostly a portfolio, there’s no reason for it to be “discoverable”. You can just link potential employers directly to it. Arguably it’ll be a net benefit because it might cut down on recruiter spam. Alternatively, maybe this problem can be solved by bringing back web rings :P
buboard 7y ago
> Block Google from crawling your websites Maybe also redirect googlebot to a porn site or something. Make them look bad
- jolmg 7y ago
  Wouldn't that actually make you look bad? Your non-porn site would appear to host porn on Google search results.
  - buboard 7y ago
    so? you wouldn't care for google's traffic anyway, and ppl who know your nonprofit would assume that google went bananas
sequoia 7y ago
I will sign on! Someone make a badge so I can put it on my site's footer :)
dmitrygr 7y ago
> I want good placement in their search results (emphasis added) I too want ability to control others
nunez 7y ago
Let's not forget that you can't disable AMP in Chrome
dastx 7y ago
This is great and all, but small time bloggers aren't gonna matter. So how do we convince the big fish that Google cares about so much?
sanbor 7y ago
Go to Google, seach "Dear Google, I'm Blocking You From My Website" (with our without the quotes). You will find this very page and other link-sharing networks. Go to Duck duck go and search "Dear Google, I'm Blocking You From My Website". Without quotes the results are nonsense. With quotes you get 0 results.
- dhimes 7y ago
  That may be partially a timing thing. One hour after your post, this page (HN) turns up if you use quotes. Now, I agree that it would seem that a smart algorithm would automate the case of "if it shows up on HN, add the source link to the results too." Especially since Gabriel is on here IIRC.
user17843 7y ago
Someone please make a googlebot noindex web page with a list of participating websites, in an effort to make resistance against google go viral. If Google is reduced to basically product ads and news it will lose its relevance as a search engine.
runxel 7y ago
> As fewer results of such not-for-profit websites show up in Google Search, competition will be reintroduced to the search market. Sorry, you lost me there. Can anybody explain me please, how this is supposed to work? Because I think that can not work. If you're not on Google, you have no outreach whatsoever... Just the classic Rumsfeld with his "unknowns" comes to my mind here.
MrStonedOne 7y ago
As a user, I fucking hate searching on mobile google now. I was trying to find some information on something, can't remember what, and all I got on the first 3 pages were amp links to news sites talking about the thing, but nothing to do with the info I was looking for. I ended up opening duckduckgo and it was on the first page. Did google just forget that static article content does not exclusively make up the internet? edit: I'm gonna re-post the rant I posted last time this came up: > As a mobile user, I hate it. > I hate that every fucking google search result on mobile web has its stupid little icon > I hate that there is no way for me to disable it as a user > I hate that it has muddied the waters in what the url bar means > I hate that it has trained users to not question fake url bars. > I hate that cloudflare so thoroughly jumped on its dick > I hate that we invented a way to fake the address in the url bar just for this stupid fucking feature. > I hate that we now have a system where somebody can share a page url with a friend, and that friend can view it on the same device model using the same browser with the same settings, and will get a different page because one was viewing an amp page but shared it's real url. > I hate that every fucking amp page is lower featured in some way, and almost never works in desktop mode. > And most of all, I hate that it leads to everybody offloading shit onto google's servers. > AMP is not fast because it's served from google's CDN. AMP is fast because it's incompatible with 99% of the bullshit client cpu heavy tracking and ad libraries, so they don't get included inside AMP pages.
- kspacewalk 7y ago
  I'm with you on some of those, but having one bar for URLs and search is a) great from the point of view of usability and, b) inevitable. There's just no real estate for two perfectly combinable text boxes on mobile devices.
  - gothroach 7y ago
    I don't know, I'm still firmly in the camp that the URL bar and search bar should be separate. When I'm typing in the URL bar, I expect it to attempt to resolve and request whatever I typed in. If I wanted to search, I'd search. This has caused annoyance with some single-word DNS names not working and performing a search instead. I'm still not entirely sure why some words will do DNS resolving and some don't.
kowdermeister 7y ago
Am I missing something? The article is about AMP. It's fine to dislike it, but your blog content is not displayed via AMP if you do not enable so blocking the Googlebot is totally unreasonable to me.
- MrStonedOne 7y ago
  AMP pages get ranked above non-AMP pages, so if you don't support AMP you lose money from the lost ranking, but AMP is bullshit so its a shitty lose-lose google is doing.
oblib 7y ago
I've been fairly well amazed with how far developers have embraced and kowtowed to Google since they first bought "Urchin" and started issuing requirements, guidelines, and standards, for them to "rate" your web page. Going down that road was against the best ideals of how the internet should work and was essentially "giving the farm" to Google, but everyone around me jumped right in and even fought with one another to do it better. I readily admit that crushed my momentum, but I just could not do it. I could not give all my user's data to Google or develop according to their standards and requirements. "Gatekeepers" is accurate. I just cannot follow that route.
Causality1 7y ago
I hate AMP for the simple reason that it hijacks my mobile Chrome browser and forces me to scroll all the way back to the top of the page and drag down again just to get to the menu/address bar so I can leave the page or switch to another tab.
zelphirkalt 7y ago
Kudos to the website author! I've not thought of that before, but next website I create will block Google too. A new way of expressing my dislike for Google in my arsenal.
unnouinceput 7y ago
quote:"If you feel inspired to do the same, simply add the following to the head of your pages to stop Google from crawling your website:<meta name="googlebot" content="noindex">" </endquote> Hahahaha. Hey, I found the joke of the week and is only Wednesday. If you think for one second that a meta or robots.txt is stopping google to crawl your website you're naive.
whalabi 7y ago
Had a chuckle because this article starts with: "I read these points in this other article" ...Which links to an article that says: "I read these points in this other (other) article" ...Which links to the article I wrote where I made the points being referred to :)
zzo38computer 7y ago
I have blocked Google from my website for a while already. (And I don't use AMP either.)
deftturtle 7y ago
Thanks for sharing! After reading I decided to block google as well. Here’s my blog post: https://calebyers.com/blog/i-blocked-googlebot.html https://calebyers.com/blog/i-blocked-googlebot.html

431 points by bradley_taunt 7y ago

notyourday 7y ago

"Market can be irrational for longer than you can remain solvent"

jgmmo 7y ago

yeah but then how do you get organic traffic?

deleted 7y ago
[deleted]
LyndsySimon 7y ago
On a related note, what does organic search traffic look like these days? I was once personally invested in running several small content sites and at the time organic traffic was pretty much entirely from Google. There were other inbound sources - Bing, Yahoo!, etc. - but they were so negligible that it would have been reasonable to ignore them completely for the purposes of optimization. I've been using DDG as my primary search engine now for several years, and using Google no longer "feels like the Web" for me. I imagine I have one hell of a case of observation bias in that, but it does seem like alternatives have been gaining some traction in recent years.
- jgmmo 7y ago
  I agree, I use Startpage.com That said, looks like most of the US does not feel the same way about Google. I still see 90-99% of organic traffic coming from Google for my client sites.
- heyyyouu 7y ago
  It's still 90-95 percent Google, surprisingly. You'd think it would be less with mobile but no. We are seeing more duck duck go, but bing and yahoo have stayed steady for years. Social is a rise, and that's a lot of mobile. Also then just pickups (links from sites like this). But yeah, still very much google dominated.

bhartzer 7y ago

Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive. Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are removed from the index, block the google IPs.

joering2 7y ago
How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
- edm0nd 7y ago
  You can block Googlebot by User-Agent and IP addresses.
- bhartzer 7y ago
  There are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
  - LunaSea 7y ago
    Wouldn't that be considered illegal due to the same reasons as scraping?
    - hnick 7y ago
      I don't think scraping is illegal, unless you're hitting protected pages behind a login after agreeing to a Terms of Service, or hitting it so often you'd constitute a DOS attack. An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.

dna_polymerase 7y ago

"I don't like one of their products so I don't want anyone to find the stuff I write online" Unless you are dependent on SEM, AMP doesn't really hurt you. This whole no-index thing is ridiculous and not at all possible for any sufficiently large website/company. Only thing I can agree with is the alternative search engine point. We really need more competition in the field, so Google is forced to cater to it's user's wishes more instead of purely pushing their agenda on them.

ipsin 7y ago

Are they actually blocking Google somehow? No robots.txt, no 403 when using a Googlebot user agent to fetch the site page...

sieabahlpark 7y ago
Googlebot uses the Google ip range
einr 7y ago
Yes. <meta name="googlebot" content="noindex"> Googlebot looks for this meta tag.

rahuldottech 7y ago

> As fewer results of such not-for-profit websites show up in Google Search, competition will be reintroduced to the search market. I don't think it will. While I personally hate AMP, and disagree with many of Google's practices, this is not the solution. Google is everywhere. It is the default search engine on almost every browser, as well as on most devices. How often do you hear someone say "look it up" as opposed to "Google it"? The latter is much more common in my experience. Google is synonymous with internet searches. Outside of tech circles, no one is going to make an effort to even try other search engines. There's a reason Bing and Yahoo are treated as a joke. Google is also the best at searches. While other search engines are slowly catching up, and Google's own results are becoming worse, it's still way better then most search engines - especially when it comes to specific and local/regional results. Lastly, blocking Google means that you site also won't show up on Startpage.com results, because it sources results from Google. This is unfortunate, since Startpage is definitely one of the search engines that respect your privacy the most.

allknowingfrog 7y ago
I'm not saying it proves anything, but Kleenex is synonymous with facial tissue and Velcro is synonymous with hook and loop fasteners. I "google" things on DuckDuckGo. :)
- rahuldottech 7y ago
  Yeah, I was just pointing out that to most people, ["google" == "web search"]
- r3bl 7y ago
  Yeah, I've been using DDG for years, but I just can't get "Google it" out of my dictionary. I try to replace it with "Duck it", which feels stupid as fuck to say, but "Google it" doesn't feel any better.
  - lostmsu 7y ago
    I don't think it is a problem. Xerox is a thing.

LyndsySimon 7y ago

> Google needs to stick to being a solid search engine instead of trying to be gate keepers While I agree with the sentiment, I feel like this ship sailed years ago - it's like saying "Amazon should stick to being an ecommerce platform". What's more, Google was never in the business of "being a solid search engine". Google is and always has been in the business of selling access to users' eyeballs. AMP - as much as I hate it - is directly in line with their core business.

PavlovsCat 7y ago
> What's more, Google was never in the business of "being a solid search engine". Google is and always has been in the business of selling access to users' eyeballs. That's not how they became the darling of the tech scene though. http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html > Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users [..] we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers. Since it is very difficult even for experts to evaluate search engines, search engine bias is particularly insidious. [..] we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm.
- liberte82 7y ago
  When Google was first released it was revolutionary. I remember just being amazed at the quality of the search results compared to any other engines. And amazingly, the competitors still haven't caught up, for all of Google's issues.
  - lunchables 7y ago
    So true, the difference was monumental. Pagerank was a huge breakthrough that seems so simple and so obvious in hindsight, it's clear why it changed everything. The popularity of a website shouldn't be based on the number of times they can stuff keywords into a page, it should be based on the number of human beings who used the site and determined it was valuable enough to link to. I don't know if pagerank is used anymore, but at the time, with the state of the web then, it was an incredible breakthrough.

JohnC8 7y ago

It starts with one, Google abused their power too many times. Get a good domain name and direct traffic. That is all.

wil421 7y ago

Can anyone remember if the old AOL landing page linked to AOL created content or content by 3rd parties? I’ve made the argument before that Google is the new AOL homepage. It’s the gateway to the web for a huge majority as AOL was at one time. AMP is concentrating Google’s web just as AOL once did for first timers. Google has much more utility but it’s still very similar. AOL had email, web, chat, news, shopping? (maybe late days of AOL), and let’s not forget search.

lqqkout4elfy 7y ago

This site loads amazingly fast! Proves the point that you really don't need AMP.

kypro 7y ago
You do if you want Google to send any traffic to it. Your only option is to bend the knee unless you're site is nothing more than a blog which you do for fun.
dddddaviddddd 7y ago
All that AMP does for speed is impose limits on CSS and JS size (plus givs access to Google CDNs). You can simply write small assets without the need for Google to specify this.
- cameronbrown 7y ago
  I've recently built an AMP version of a marketing website (since AdWords supports AMP cache, thought it would be useful for speed). The truth is, while AMP as a library does provide solid speed improvements on its own by implementing all the best practices, the real speed improvements is the cache with preloading - as well as the other stuff it does like image optimisation.

mateus1 7y ago

Google once was a force for good by indexing pages and rewarding good content with SEO. I deeply hate AMP. It's a monopolistic move to bully companies into handing over Google their content.

lern_too_spel 7y ago
> It's a monopolistic move to bully companies into handing over Google their content. That's nonsensical. Google already has their content. If anything, AMP is a democratizing technology. If you want your articles in Apple News, you have to integrate with Apple. Same with Facebook Instant Articles. With AMP, you don't have to integrate directly with anybody. Just publish AMP pages from using standard HTTP, and every link aggregator that implements an AMP cache (including Bing, Baidu, Yandex, Yahoo Japan, etc.) will be able to instantly load your pages for users. If I start my own link aggregator, I don't need to have the clout of a Facebook, Apple, or Google to get publishers to enable instant loading from my site — I can cache the AMP pages they have already published. It's the same as publishing bus routes according to Google Maps's standard feed format (https://developers.google.com/transit/gtfs/ https://developers.google.com/transit/gtfs/) . I publish once in that format, and not only do my bus routes show up better in Google, but they also show up in all other mapping applications that ingest that format, without me having to do any integration with them.
- thepangolino 7y ago
  AMP is a complicated answer to an easy one. AMP was presented as an answer to webpages not being lightweight anymore. Instead of just pushing for more lightweight webpages, Google cane up with the monstrosity that is AMP.
  - lern_too_spel 7y ago
    > AMP was presented as an answer to webpages not being lightweight anymore. Wrong. AMP is a solution to web pages not loading instantly, not a solution to web pages not being lightweight. If you want to prerender links above the fold without deanonymizing them to the publisher, you will come up with AMP pretty much exactly.

echeese 7y ago

There's an easier way - don't implement AMP.

t0mbstone 7y ago
Yeah, I don't understand this post. There are billions of pages out there without AMP and Google indexes them just fine. If you don't wanna implement AMP, then just don't implement it. You don't have to completely block Google over it.
- deleted 7y ago
  [deleted]
- heyyyouu 7y ago
  Exactly! I am so confused as well. It's like this person doesn't understand that AMP is optional. And even then there's ways to instruct Google NOT to point to your AMP page. (And no, you don't need AMP for SEO -- it's only a slight boost)
  - bluGill 7y ago
    The problem is if your competitor has an AMP page google will rank them higher than you. That will cost your money.
    - cameronbrown 7y ago
      AMP only puts you in the top stories carousel. Speed is measured neutrally for organic search results.
      - heyyyouu 7y ago
        THIS. I don't understand how so many people don't actually get how Google and AMP work. You don't need AMP for organic search results. It's used by Google, yes, but only for certain things AND there's ways to turn it off/point to non-AMP pages. We choose not to use it, other publishers do use it, but everyone looks at the pros and cons and then just decides. It's like any other business decision, but you have to understand how it actually works first and not just go "Oh, AMP's bad" or "Oh, without AMP my site is dead.) I don't believe either of those is true.

TazeTSchnitzel 7y ago

Is it possible to make an AMP page that just redirects to a non-AMP one?

heyyyouu 7y ago
That is one technique some people use.

speeder 7y ago

I am currently struggling with Google and been even considering just straight up creating a Brazilian "Yandex/Baidu" clone or something somehow. My current business is heavily Google reliant, and Google been proving to be a very poor business partner: * Keeps punishing my site both in organic rank and AdWords rank (thus price) for not being mobile, despite my site being intended to be viewed by engineers with big screens. * They remove frequently whatever features they had to avoid wasting money, the one that offended me the most is that I can't anymore advertise on mobile websites unless I also allow apps too, and apps are purely fraudulent clicks, babies don't buy industrial parts... * Google refuses to take care of fraud, I've visited the top sources of clicks on my display campaigns and there was comments for example on store pages explicitly complaining it was pure fraud, and Google does nothing about it (example: one source of clicks was a flashlight app on Android that opens ads whenever the user turns the flashlight on, clicking the ad is only way to turn it off again) * Google often randomly stop my ads and search because bogus reasons, requiring me to call them to fix it. * They removed from AdWords interface the max click price when the campaign is in automatic mode, in the past this was very useful to find out if their AI was bugged (like one time it insisted on setting max cpm to 0.01 despite average cost until that day being 1.20...) or oberpricing me (charging ludicrous prices per click) Their product (ad publishing, not search) is crap, but they have literal Monopoly, all other ads publishers I tried gave me almost zero clicks... Only thing I don't tried yet is more traditional ads, but I don't have money for that (for example TV ads) I honestly don't know why so many governments with strong antimonopoly laws tolerate Google shenanigans.

whatshisface 7y ago
>I honestly don't know why so many governments with strong antimonopoly laws tolerate Google shenanigans. To build an anti-monopoly case you would have to do more than show that Google was the only good ad vendor. Are they the only ones that can give you clicks because they are smarter and better at their jobs than every other ad vendor, or are they the only ones that can give you clicks because of underhanded market tactics? If I was the best shoemaker in town, to the point where buying shoes from anybody else was a waste of time, that wouldn't be illegal. You'd have to show I was doing more to stop competition.

JohnFen 7y ago

Way ahead of you. I started blocking Googlebot from my websites a few years ago, and never looked back!

rangeofmotion 7y ago

I really wonder about the people who work at Google. Surely they have people whose ideals are higher than this? It seems obvious that Google is a real threat to the freedom the internet originally made possible. But engineers still sign up in droves to work there? What kind of engineers are these? Do all of Google's employees simply disagree that Google is a threat? Do they not care? Are they just financially scared and willing to sell everyone else out so they can pay their mortgage? Are there efforts internally to try and stop them from centralizing everything? What's going on here? I know only a couple of people who don't think Google is a real problem today.

bsimpson 7y ago
I don't work on AMP (I've never even written an AMP page); this is purely my personal perspective: The difference is trust: These AMP rants invariably look to the use of Google's CDN, or to the "google.com" domain in the address bar, and infer malintent. I presume the team had good intentions, particularly for end users. The AMP team saw a problem (websites take a stupid amount of time to load on mobile, even on nice phones/networks) and devised a solution: "Google has one of the best network infrastructures on the planet. It serves things quickly. Let's cache pages that don't do slow things on that network, so they're fast for users." They devised some criteria for what they mean by "don't do slow things," and wrote tooling to assert it. I don't believe the intent of the AMP team is to strongarm developers into using their framework, or to funnel all traffic through Google. I trust that they are well-intentioned people who are trying to do the right thing. They're not proud of the limitations of their original solution, and are making progress on fixing them: for instance, they pushed forward a new web standard (packaging) to fix the address bar problem: https://blog.amp.dev/2018/05/08/a-first-look-at-using-web-packaging-to-improve-amp-urls/ https://blog.amp.dev/2018/05/08/a-first-look-at-using-web-pa... --- Many words have been written about the ramifications of echo chambers on our political discourse. Those same ideas apply here on HN too. The more I see hyperbolic comments presuming everyone/thing is evil/bad/stupid, the less interesting this place becomes for me -- the less likely I am to come here. As that culture drives people like me away, the ratio of conspiratorial-armchair-quarterbacking:reasonable-discourse tilts further towards the negative. It drives even more people away, leaving a concentration of negativity. "Don't read the comments" starts to apply here too. That's not a reaction to your comment, in particular: it's how I've started to feel about a lot of Hacker News, especially when my employer is the topic. When people presume the absolute worst -- in spite of more reasonable (and more likely) alternatives -- there's nothing fun to read or interesting to learn. I lose reasons to keep coming back.

heyyyouu 7y ago

I'm very confused by this article -- AMP is an optional framework: https://amp.dev/ https://amp.dev/ and even if you use it you can opt out of having Google load/point to it: https://support.google.com/news/producer/answer/7281462?hl=en https://support.google.com/news/producer/answer/7281462?hl=e... You don't need to no_index your site. What can you do about it? Just don't implement AMP. Yes, sites who do AMP get preferential treatment in search engines, but you can still get excellent results without it; it's a very small boost.

rahuldottech 7y ago
It's a matter of principle, AMP is a terrible thing for the web in general.
- heyyyouu 7y ago
  Yes, a lot of people agree on that. But you don't need to block Google from your site to stop AMP.The way this article is written it makes it seem like the only way to have AMP not affect your site is by blocking Google entirely, and that's simply not the case. I think that's why those of us who understand how AMP is used are commenting here -- and you can see that people are confused because other people are posting questions about blocking amp, when you don't have to block amp, you don't use it in the first place. The author should clarify that they're taking extreme measures, but that this isn't required to not use AMP.
  - anecdotalShit 7y ago
    the article clearly states. The reason to block is to improve competition. The lack of competition is the sole reason why google can stongarm amp instead of an open standard
playpause 7y ago
> Just don't implement AMP. ... you can still get excellent results without it; it's a very small boost. What evidence do you have that it's only a small boost? That may be the case for very specific searches. But if you search anything where there is a lot of competition between publishers (coverage of major news stories, product reviews, etc), Google serves almost exclusively AMP results, with the exception of the odd result from Wikipedia or Twitter. Failing to implement AMP would be suicide for many publishers.

meerita 7y ago

Google must offer an option to opt out from those kind of products and at the same time, offer the search result. I am interested in this, I don't want them to replace me for their own UIs.

heyyyouu 7y ago
AMP is optional. You don't have to implement it. Even if you do there's ways to instruct Google when to use it/when not to.
- meerita 7y ago
  Yes, but there are other things that they already display in their search results without opting.

Proven 7y ago

Good move, kudos to Bradley

Circuits 7y ago

Yeah alright I can get behind this. I don't want to support a company looking to monopolize the market. I will start looking into how I can go about cutting Google out of my life.

jammygit 7y ago

If he’s going offline from search, he should create an rss or mail list. I didn’t see where to subscribe, so honestly I will probably forget his site exists in a week...

AlexandrB 7y ago

I like this. It reminds me of the pizza shop that gave 50% off to 1-star Yelp reviewers[1] or the ice cream truck that charges "influencers" double[2]. It's a statement of "I don't like your system, so I'm going to opt out of the whole thing even if it potentially harms me." [1] https://thehustle.co/botto-bistro-1-star-yelp/ https://thehustle.co/botto-bistro-1-star-yelp/ [2] https://metro.co.uk/2019/07/03/ice-cream-seller-charges-double-for-every-influencer-who-asks-for-a-freebie-10109479/ https://metro.co.uk/2019/07/03/ice-cream-seller-charges-doub...

deminature 7y ago
It's not going to potentially harm him, not being listed by Google is arguably equivalent to ceasing to exist. It's a nice gesture, but I cannot imagine how this is good idea in practice, regardless of how you view Google ethically.
- pmlnr 7y ago
  Yeah, no. Ultra-Local business - eg. ice cream truck - really don't need a google presence in any way.
  - alanbernstein 7y ago
    Yeah, no. A programmer's blog is not an ultra-local business.
    - pmlnr 7y ago
      Parent edited.
- AlexandrB 7y ago
  If your blog is mostly a portfolio, there’s no reason for it to be “discoverable”. You can just link potential employers directly to it. Arguably it’ll be a net benefit because it might cut down on recruiter spam. Alternatively, maybe this problem can be solved by bringing back web rings :P

buboard 7y ago

> Block Google from crawling your websites Maybe also redirect googlebot to a porn site or something. Make them look bad

jolmg 7y ago
Wouldn't that actually make you look bad? Your non-porn site would appear to host porn on Google search results.
- buboard 7y ago
  so? you wouldn't care for google's traffic anyway, and ppl who know your nonprofit would assume that google went bananas

sequoia 7y ago

I will sign on! Someone make a badge so I can put it on my site's footer :)

dmitrygr 7y ago

> I want good placement in their search results (emphasis added) I too want ability to control others

nunez 7y ago

Let's not forget that you can't disable AMP in Chrome

dastx 7y ago

This is great and all, but small time bloggers aren't gonna matter. So how do we convince the big fish that Google cares about so much?

sanbor 7y ago

Go to Google, seach "Dear Google, I'm Blocking You From My Website" (with our without the quotes). You will find this very page and other link-sharing networks. Go to Duck duck go and search "Dear Google, I'm Blocking You From My Website". Without quotes the results are nonsense. With quotes you get 0 results.

dhimes 7y ago
That may be partially a timing thing. One hour after your post, this page (HN) turns up if you use quotes. Now, I agree that it would seem that a smart algorithm would automate the case of "if it shows up on HN, add the source link to the results too." Especially since Gabriel is on here IIRC.

user17843 7y ago

Someone please make a googlebot noindex web page with a list of participating websites, in an effort to make resistance against google go viral. If Google is reduced to basically product ads and news it will lose its relevance as a search engine.

runxel 7y ago

> As fewer results of such not-for-profit websites show up in Google Search, competition will be reintroduced to the search market. Sorry, you lost me there. Can anybody explain me please, how this is supposed to work? Because I think that can not work. If you're not on Google, you have no outreach whatsoever... Just the classic Rumsfeld with his "unknowns" comes to my mind here.

MrStonedOne 7y ago

As a user, I fucking hate searching on mobile google now. I was trying to find some information on something, can't remember what, and all I got on the first 3 pages were amp links to news sites talking about the thing, but nothing to do with the info I was looking for. I ended up opening duckduckgo and it was on the first page. Did google just forget that static article content does not exclusively make up the internet? edit: I'm gonna re-post the rant I posted last time this came up: > As a mobile user, I hate it. > I hate that every fucking google search result on mobile web has its stupid little icon > I hate that there is no way for me to disable it as a user > I hate that it has muddied the waters in what the url bar means > I hate that it has trained users to not question fake url bars. > I hate that cloudflare so thoroughly jumped on its dick > I hate that we invented a way to fake the address in the url bar just for this stupid fucking feature. > I hate that we now have a system where somebody can share a page url with a friend, and that friend can view it on the same device model using the same browser with the same settings, and will get a different page because one was viewing an amp page but shared it's real url. > I hate that every fucking amp page is lower featured in some way, and almost never works in desktop mode. > And most of all, I hate that it leads to everybody offloading shit onto google's servers. > AMP is not fast because it's served from google's CDN. AMP is fast because it's incompatible with 99% of the bullshit client cpu heavy tracking and ad libraries, so they don't get included inside AMP pages.

kspacewalk 7y ago
I'm with you on some of those, but having one bar for URLs and search is a) great from the point of view of usability and, b) inevitable. There's just no real estate for two perfectly combinable text boxes on mobile devices.
- gothroach 7y ago
  I don't know, I'm still firmly in the camp that the URL bar and search bar should be separate. When I'm typing in the URL bar, I expect it to attempt to resolve and request whatever I typed in. If I wanted to search, I'd search. This has caused annoyance with some single-word DNS names not working and performing a search instead. I'm still not entirely sure why some words will do DNS resolving and some don't.

kowdermeister 7y ago

Am I missing something? The article is about AMP. It's fine to dislike it, but your blog content is not displayed via AMP if you do not enable so blocking the Googlebot is totally unreasonable to me.

MrStonedOne 7y ago
AMP pages get ranked above non-AMP pages, so if you don't support AMP you lose money from the lost ranking, but AMP is bullshit so its a shitty lose-lose google is doing.

oblib 7y ago

I've been fairly well amazed with how far developers have embraced and kowtowed to Google since they first bought "Urchin" and started issuing requirements, guidelines, and standards, for them to "rate" your web page. Going down that road was against the best ideals of how the internet should work and was essentially "giving the farm" to Google, but everyone around me jumped right in and even fought with one another to do it better. I readily admit that crushed my momentum, but I just could not do it. I could not give all my user's data to Google or develop according to their standards and requirements. "Gatekeepers" is accurate. I just cannot follow that route.

Causality1 7y ago

I hate AMP for the simple reason that it hijacks my mobile Chrome browser and forces me to scroll all the way back to the top of the page and drag down again just to get to the menu/address bar so I can leave the page or switch to another tab.

zelphirkalt 7y ago

Kudos to the website author! I've not thought of that before, but next website I create will block Google too. A new way of expressing my dislike for Google in my arsenal.

unnouinceput 7y ago

quote:"If you feel inspired to do the same, simply add the following to the head of your pages to stop Google from crawling your website:<meta name="googlebot" content="noindex">" </endquote> Hahahaha. Hey, I found the joke of the week and is only Wednesday. If you think for one second that a meta or robots.txt is stopping google to crawl your website you're naive.

whalabi 7y ago

Had a chuckle because this article starts with: "I read these points in this other article" ...Which links to an article that says: "I read these points in this other (other) article" ...Which links to the article I wrote where I made the points being referred to :)

zzo38computer 7y ago

I have blocked Google from my website for a while already. (And I don't use AMP either.)

deftturtle 7y ago

Thanks for sharing! After reading I decided to block google as well. Here’s my blog post: https://calebyers.com/blog/i-blocked-googlebot.html https://calebyers.com/blog/i-blocked-googlebot.html