4 ms·

One big reason why Pinterest ranks so highly, and why you are seeing different tld's in the results, is because they have a variety of tld's that they use and t

One big reason why Pinterest ranks so highly, and why you are seeing different tld's in the results, is because they have a variety of tld's that they use and those tld's all have separate crawl budgets from each other. So Google crawls Pinterest about an order of magnitude more than most other sites. You can read more about this here: https://medium.com/pinterest-engineering/how-switching-our-domain-structure-unlocked-international-growth-e00c8184d5dd https://medium.com/pinterest-engineering/how-switching-our-d... Source: I run a think tank focused on Google’s web crawling advantage and have been studying stuff like this for a couple years now.

marmaduke 6y ago
Does your think tank come up with any ideas on how to keep companies like Pinterest from peeing in the pool so to speak? They made it impossible to find DIY craft stuff that I used to search for, and I can’t imagine it’s the only site doing it.
- knuckleheads 6y ago
  We are pursuing ideas that we think will increase competition in the search engine market. Our main focus right now is making Google’s index of the web available for use by competitors, since no one other than Google is really allowed to crawl the web. If people were allowed to use the index to build what they want, then you or somebody else would be able to build a search engine that avoids the Pinterest cruft and would be more suited to your needs, instead of Google’s one size fits all approach. So, to answer your question directly, yes we do have some ideas about how to do this.
  - burnthrow 6y ago
    > making Google’s index of the web available for use by competitors What incentive would Google have to continue populating that index? Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? > since no one other than Google is really allowed to crawl the web Maybe this is the problem that needs solving.
    - knuckleheads 6y ago
      > What incentive would Google have to continue populating that index? Presumably they would still want to run google.com and make money off of it. > Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to. Updated based on edit of parent post: > Maybe this is the problem that needs solving. Why have websites waste the money to serve all those requests all over again? Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? I don't think website operators would be happy if there were a hundred more crawlers out there crawling as much as Google does now.
      - deleted 6y ago
        [deleted]
      - lopmotr 6y ago
        Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof GoogleBot. Browsers have being doing that since forever spoofing Netscape (Mozilla), Safari, etc. for the same reason. > Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? This sounds like a common fallacy of people criticizing the free market. Duplicated effort looks wasteful but turns out to be far more productive than the lack of incentive that comes with not being able to profit from your work/investment.
        Someone 6y ago
        I would think the site owner’s cost of being indexed is the same for every search engine that indexes the site. The benefit varies with the quality of the search engines, and that will vary between search engines, but it does get larger the more a search engine is used, so a cost/benefits analysis may show Google and a few other large ones are the only ones worth supporting.
  - aleph_naught 6y ago
    > since no one other than Google is really allowed to crawl the web. ??
    - knuckleheads 6y ago
      There are two main reasons why I say nobody besides Google is really allowed to crawl the web. The first is that Google gets much more access to pages on websites than everybody else. You can see this by examining the robots.txt files of various websites[0]. I've been doing this for several years now and Google has a consistent advantage across many thousands websites that I've looked at. This adds up to a significant advatnage and many search engine operators complain about how it hampers their ability to compete with Google[1]. The second is that Google gets to ignore crawl delay directive in robots.txt while other search engines don't[2]. Website operators cannot tell Google how fast they want their website crawled, they can only request that Google slow down. If another search engine tried to do what Google does, they would likely be blocked by many important websites. If you would like to read more about this, please checkout https://knuckleheads.club/ https://knuckleheads.club/ [0] https://pdf.sciencedirectassets.com/robots.txt https://pdf.sciencedirectassets.com/robots.txt [1] https://www.nytimes.com/2020/12/14/technology/how-google-dominates.html https://www.nytimes.com/2020/12/14/technology/how-google-dom... [2] https://www.seroundtable.com/google-noindex-in-robots-txt-dead-27824.html https://www.seroundtable.com/google-noindex-in-robots-txt-de...
      - grishka 6y ago
        So, uh, don't respect robots.txt in your search engine? It's not like there's a law that you have to, and that you can't pretend you are Googlebot. The only real obstacle I can imagine is that some firewalls might be configured to be more permissive with traffic originating from Google subnets.
        knuckleheads 6y ago
        You would be blocked fairly quickly by many website operators and no longer able to access those websites if you straight up ignored robots.txt files. You also might even end up being served cease and desists by some websites and sued if you continue to persist and try to find ways around it.
  - marmaduke 6y ago
    > making Google’s index of the web available for use by competitors That does seem like a good idea since it amounts essentially to a database of stuff that Google does not own (by construction) and is of public utility.
    - knuckleheads 6y ago
      Exactly. Everybody always talks about turning tech companies into public utilities without really explaining what the utility would be. A public index of the web would be an amazing utility for many of the reasons in this thread and would spur on a ton of new innovation and businesses. If you would like to read more about all this, please checkout https://knuckleheads.club https://knuckleheads.club
      - rarefied_tomato 6y ago
        The pursuit of a public index is excellent. Some feedback: - A cost of membership runs contrary to establishing this group, especially at such a high recurring charge. - I'm not sure what your software/AWS situation looks like, but 20 million robots.txt files acquired from Common Crawl is something I can analyze on my PC. It doesn't seem to presently justify such high costs. - Prioritize building a mockup index with an intuitive frontend. This is essential for non-technical people to understand - Exclusively talk with EU legislators (they are motivated, whereas nothing will happen in the US).
        knuckleheads 6y ago
        Thank you for the feedback! I think the price for membership dues is reasonable and many people agree evidenced by them signing up. I think I might start a petition that is free to sign up on though, thank you for the inspiration! It is possible to analyze those files on the pc, it just takes a much longer time. The analysis is an iterative process and so the faster the computers the faster the iterations and process go. I was analyzing them on my pc with python for the first year until it got too slow and my I am using an aws server with some rust and that is going much better. I also need to increase the number of files analyzed by about two orders of magnitude soon as well. Great idea, very cool. That’s going on the todo list! And I am going to be reaching out to and speaking with whoever is interested. One of the fun things about this is that it is an international dynamic, with some jurisdictions having abilities that others don’t. For example, the UK CMA has subpoena powers that the US Congress lacks and got a ton of information out of Google and Bing that shocked me. The US has the ability to get the CEO’s to show up to hearings while the UK does not in the same way. Why limit ourselves to one government when there are so many to mix and match from here?
  - frakkingcylons 6y ago
    > Our main focus right now is making Google’s index of the web available for use by competitors How would that work? I'm pretty curious about this if there's anything out there to read. EDIT: Ah I saw the link to Knuckleheads' Club :P
    - knuckleheads 6y ago
      :) No worries. If you'd like to chat about it more, shoot me an email at zack@knuckleheads.club, I'm obviously very passionate about the subject and love talking about it. There's still a lot more to write and publish, I won't be done with this hobby horse of mine for a while yet.
  - fsflover 6y ago
    Wow, such index would tremendously help free p2p search YaCy, https://yacy.net https://yacy.net.
Winterflow3r 6y ago
This I did not know about and I'm surprised they don't get penalised by Google in some way at the display of search results stage. I mean sure, I suppose one can do the -pinterest trick, but it's still a bit weird that Google thinks search results that are pretty much the same site but from different tlds are not spammy. Do you think they will change this to prevent Pinterest from essentially spamming some search queries by using the tld hack?
- knuckleheads 6y ago
  Live by the SERP updates, die by the SERP updates. It's been at least two years now and Google seems fine with it (until maybe one day they aren't). I know it might annoy folks like you and I to get reposted pinterest images in our results but it might be that Google has done research and found that people like seeing pinterest in images and so that is why they are keeping them around. It's hard to say why they are there until Google says something about it, Google rankings are a black box at times.
joe_the_user 6y ago
I think others here have alluded this but the thing is that once, doing everything you need to do to get listed but actually being crap was enough to get you de-listed. The criteria wasn't "do what Google wants", the criteria was "be good and also help the algorithm determine that you're good". Only running the right SEO wasn't sufficient then and I certain wish that criteria remained.
- knuckleheads 6y ago
  Who determines what is the best ranking? Who was defining what was "good"? There isn't some abstract, mathematically verifiable best ranking, a platonic ideal that all search engines should aspire to. Google has always made choices about how to rank that trade off the interests of various groups of users, from the very beginning. The complaints that you and I have about pinterest being ranked so highly come because Google has decided that our desires for how the results should look are not important enough compared to how other, perhaps larger, groups of users want them to look. Maybe Google deranks pinterest tomorrow, but they've made the choice to keep it as such for a couple years now. Whether it's because they think they have better ways to spend their time or because they like it the way it is, it's not really possible for us to know without somebody leaking that information. But, I imagine, there would be a group of people who would be as disappointed as we are elated if Google were to change this.
  - joe_the_user 6y ago
    There isn't some abstract, mathematically verifiable best ranking, a platonic ideal that all search engines should aspire to. Of course, "quality" applied is a broad term indeed. But there no being a single certain measure of quality doesn't mean there aren't some things that most people would call crap. Google employs a large number of search quality raters and these folks can likely distinguish the terrible from the OK. Google is far from ideal and getting worse but you seem to be exposing a pure democracy of algorithm conformance, which is guaranteed to result in hot, steaming idiocy.
    - knuckleheads 6y ago
mslate 6y ago
I don’t think making Google’s index public would be sufficient. Their real core asset is not the index but the historical engagement data they have of searchers’ interactions with the index (both on the results page and offsite analytics Google has access to, eg Google Analytics/display ad network). Making that historical engagement data public is not feasible/realistic, IMO.
- knuckleheads 6y ago
  I think it is necessary though. Other search engine operators have cited this as one of the reasons their search engines fail or struggle to provide as good results. As far as the engagement data goes, it’s something that the Competition and Markets Authority in the UK has looked at as part of their investigation into google. See here for a very interesting analysis of Google’s and Bings click and query data: https://assets.publishing.service.gov.uk/media/5efb1db6e90e075c5674db35/Appendix_I_-_search_quality_v.3.pdf https://assets.publishing.service.gov.uk/media/5efb1db6e90e0...
  - mslate 6y ago
    Thanks--I learned some valuable vocabulary today! "Click-and-query data" was the phrase I did not have on hand. I have never worked for a search engine company (or devoted much time to SEO), but I would suggest the click-and-query data advantage is invaluable to cementing Google's supremacy over any newcomers.
    - knuckleheads 6y ago
      It was new to me as well, I think it was a phrase that was invented to describe the data during the CMA investigation but I am not sure. I’ve heard search engine operators complain about the index consistently while they seem spilt about the click and query data. For example, DuckDuckGo and StartPage have made pretty good businesses out of pointedly not collecting click and query data. For my part, I think it’s the dual lock on distribution via exclusive agreements on being the default search engine and the advantages it has when web crawling that cements Google’s dominance over the search engine market.
newbie789 6y ago
Out of curiosity, what exactly is a "think tank"? Is it the sort of thing a person can make money by founding? What makes it distinct in type from other businesses or organizations? I've only ever met one person that had anything to do with a think tank at a party about a decade ago, and I never got her contact details so I'm still very much in the dark.
- knuckleheads 6y ago
  Good question! A think tank is an organization that researches and sometimes advocates for the government to do certain things. They can be a non profit or an llc (knuckleheads’ club is an llc right now because it was the easiest to set up). In theory, you can make money from founding a think tank, but in practice it’s not a lucrative endeavour unless it becomes big enough that the president start earning steady six figures (like at the major ones in dc).

marmaduke 6y ago

Does your think tank come up with any ideas on how to keep companies like Pinterest from peeing in the pool so to speak? They made it impossible to find DIY craft stuff that I used to search for, and I can’t imagine it’s the only site doing it.

knuckleheads 6y ago
We are pursuing ideas that we think will increase competition in the search engine market. Our main focus right now is making Google’s index of the web available for use by competitors, since no one other than Google is really allowed to crawl the web. If people were allowed to use the index to build what they want, then you or somebody else would be able to build a search engine that avoids the Pinterest cruft and would be more suited to your needs, instead of Google’s one size fits all approach. So, to answer your question directly, yes we do have some ideas about how to do this.
- burnthrow 6y ago
  > making Google’s index of the web available for use by competitors What incentive would Google have to continue populating that index? Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? > since no one other than Google is really allowed to crawl the web Maybe this is the problem that needs solving.
  - knuckleheads 6y ago
    > What incentive would Google have to continue populating that index? Presumably they would still want to run google.com and make money off of it. > Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to. Updated based on edit of parent post: > Maybe this is the problem that needs solving. Why have websites waste the money to serve all those requests all over again? Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? I don't think website operators would be happy if there were a hundred more crawlers out there crawling as much as Google does now.
    - deleted 6y ago
      [deleted]
    - lopmotr 6y ago
      Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof GoogleBot. Browsers have being doing that since forever spoofing Netscape (Mozilla), Safari, etc. for the same reason. > Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? This sounds like a common fallacy of people criticizing the free market. Duplicated effort looks wasteful but turns out to be far more productive than the lack of incentive that comes with not being able to profit from your work/investment.
      - Someone 6y ago
        I would think the site owner’s cost of being indexed is the same for every search engine that indexes the site. The benefit varies with the quality of the search engines, and that will vary between search engines, but it does get larger the more a search engine is used, so a cost/benefits analysis may show Google and a few other large ones are the only ones worth supporting.
- aleph_naught 6y ago
  > since no one other than Google is really allowed to crawl the web. ??
  - knuckleheads 6y ago
    There are two main reasons why I say nobody besides Google is really allowed to crawl the web. The first is that Google gets much more access to pages on websites than everybody else. You can see this by examining the robots.txt files of various websites[0]. I've been doing this for several years now and Google has a consistent advantage across many thousands websites that I've looked at. This adds up to a significant advatnage and many search engine operators complain about how it hampers their ability to compete with Google[1]. The second is that Google gets to ignore crawl delay directive in robots.txt while other search engines don't[2]. Website operators cannot tell Google how fast they want their website crawled, they can only request that Google slow down. If another search engine tried to do what Google does, they would likely be blocked by many important websites. If you would like to read more about this, please checkout https://knuckleheads.club/ https://knuckleheads.club/ [0] https://pdf.sciencedirectassets.com/robots.txt https://pdf.sciencedirectassets.com/robots.txt [1] https://www.nytimes.com/2020/12/14/technology/how-google-dominates.html https://www.nytimes.com/2020/12/14/technology/how-google-dom... [2] https://www.seroundtable.com/google-noindex-in-robots-txt-dead-27824.html https://www.seroundtable.com/google-noindex-in-robots-txt-de...
    - grishka 6y ago
      So, uh, don't respect robots.txt in your search engine? It's not like there's a law that you have to, and that you can't pretend you are Googlebot. The only real obstacle I can imagine is that some firewalls might be configured to be more permissive with traffic originating from Google subnets.
      - knuckleheads 6y ago
        You would be blocked fairly quickly by many website operators and no longer able to access those websites if you straight up ignored robots.txt files. You also might even end up being served cease and desists by some websites and sued if you continue to persist and try to find ways around it.
- marmaduke 6y ago
  > making Google’s index of the web available for use by competitors That does seem like a good idea since it amounts essentially to a database of stuff that Google does not own (by construction) and is of public utility.
  - knuckleheads 6y ago
    Exactly. Everybody always talks about turning tech companies into public utilities without really explaining what the utility would be. A public index of the web would be an amazing utility for many of the reasons in this thread and would spur on a ton of new innovation and businesses. If you would like to read more about all this, please checkout https://knuckleheads.club https://knuckleheads.club
    - rarefied_tomato 6y ago
      The pursuit of a public index is excellent. Some feedback: - A cost of membership runs contrary to establishing this group, especially at such a high recurring charge. - I'm not sure what your software/AWS situation looks like, but 20 million robots.txt files acquired from Common Crawl is something I can analyze on my PC. It doesn't seem to presently justify such high costs. - Prioritize building a mockup index with an intuitive frontend. This is essential for non-technical people to understand - Exclusively talk with EU legislators (they are motivated, whereas nothing will happen in the US).
      - knuckleheads 6y ago
        Thank you for the feedback! I think the price for membership dues is reasonable and many people agree evidenced by them signing up. I think I might start a petition that is free to sign up on though, thank you for the inspiration! It is possible to analyze those files on the pc, it just takes a much longer time. The analysis is an iterative process and so the faster the computers the faster the iterations and process go. I was analyzing them on my pc with python for the first year until it got too slow and my I am using an aws server with some rust and that is going much better. I also need to increase the number of files analyzed by about two orders of magnitude soon as well. Great idea, very cool. That’s going on the todo list! And I am going to be reaching out to and speaking with whoever is interested. One of the fun things about this is that it is an international dynamic, with some jurisdictions having abilities that others don’t. For example, the UK CMA has subpoena powers that the US Congress lacks and got a ton of information out of Google and Bing that shocked me. The US has the ability to get the CEO’s to show up to hearings while the UK does not in the same way. Why limit ourselves to one government when there are so many to mix and match from here?
- frakkingcylons 6y ago
  > Our main focus right now is making Google’s index of the web available for use by competitors How would that work? I'm pretty curious about this if there's anything out there to read. EDIT: Ah I saw the link to Knuckleheads' Club :P
  - knuckleheads 6y ago
    :) No worries. If you'd like to chat about it more, shoot me an email at zack@knuckleheads.club, I'm obviously very passionate about the subject and love talking about it. There's still a lot more to write and publish, I won't be done with this hobby horse of mine for a while yet.
- fsflover 6y ago
  Wow, such index would tremendously help free p2p search YaCy, https://yacy.net https://yacy.net.

Winterflow3r 6y ago

This I did not know about and I'm surprised they don't get penalised by Google in some way at the display of search results stage. I mean sure, I suppose one can do the -pinterest trick, but it's still a bit weird that Google thinks search results that are pretty much the same site but from different tlds are not spammy. Do you think they will change this to prevent Pinterest from essentially spamming some search queries by using the tld hack?

knuckleheads 6y ago
Live by the SERP updates, die by the SERP updates. It's been at least two years now and Google seems fine with it (until maybe one day they aren't). I know it might annoy folks like you and I to get reposted pinterest images in our results but it might be that Google has done research and found that people like seeing pinterest in images and so that is why they are keeping them around. It's hard to say why they are there until Google says something about it, Google rankings are a black box at times.

joe_the_user 6y ago

I think others here have alluded this but the thing is that once, doing everything you need to do to get listed but actually being crap was enough to get you de-listed. The criteria wasn't "do what Google wants", the criteria was "be good and also help the algorithm determine that you're good". Only running the right SEO wasn't sufficient then and I certain wish that criteria remained.

knuckleheads 6y ago
Who determines what is the best ranking? Who was defining what was "good"? There isn't some abstract, mathematically verifiable best ranking, a platonic ideal that all search engines should aspire to. Google has always made choices about how to rank that trade off the interests of various groups of users, from the very beginning. The complaints that you and I have about pinterest being ranked so highly come because Google has decided that our desires for how the results should look are not important enough compared to how other, perhaps larger, groups of users want them to look. Maybe Google deranks pinterest tomorrow, but they've made the choice to keep it as such for a couple years now. Whether it's because they think they have better ways to spend their time or because they like it the way it is, it's not really possible for us to know without somebody leaking that information. But, I imagine, there would be a group of people who would be as disappointed as we are elated if Google were to change this.
- joe_the_user 6y ago
  There isn't some abstract, mathematically verifiable best ranking, a platonic ideal that all search engines should aspire to. Of course, "quality" applied is a broad term indeed. But there no being a single certain measure of quality doesn't mean there aren't some things that most people would call crap. Google employs a large number of search quality raters and these folks can likely distinguish the terrible from the OK. Google is far from ideal and getting worse but you seem to be exposing a pure democracy of algorithm conformance, which is guaranteed to result in hot, steaming idiocy.
  - knuckleheads 6y ago

mslate 6y ago

I don’t think making Google’s index public would be sufficient. Their real core asset is not the index but the historical engagement data they have of searchers’ interactions with the index (both on the results page and offsite analytics Google has access to, eg Google Analytics/display ad network). Making that historical engagement data public is not feasible/realistic, IMO.

knuckleheads 6y ago
I think it is necessary though. Other search engine operators have cited this as one of the reasons their search engines fail or struggle to provide as good results. As far as the engagement data goes, it’s something that the Competition and Markets Authority in the UK has looked at as part of their investigation into google. See here for a very interesting analysis of Google’s and Bings click and query data: https://assets.publishing.service.gov.uk/media/5efb1db6e90e075c5674db35/Appendix_I_-_search_quality_v.3.pdf https://assets.publishing.service.gov.uk/media/5efb1db6e90e0...
- mslate 6y ago
  Thanks--I learned some valuable vocabulary today! "Click-and-query data" was the phrase I did not have on hand. I have never worked for a search engine company (or devoted much time to SEO), but I would suggest the click-and-query data advantage is invaluable to cementing Google's supremacy over any newcomers.
  - knuckleheads 6y ago
    It was new to me as well, I think it was a phrase that was invented to describe the data during the CMA investigation but I am not sure. I’ve heard search engine operators complain about the index consistently while they seem spilt about the click and query data. For example, DuckDuckGo and StartPage have made pretty good businesses out of pointedly not collecting click and query data. For my part, I think it’s the dual lock on distribution via exclusive agreements on being the default search engine and the advantages it has when web crawling that cements Google’s dominance over the search engine market.

newbie789 6y ago

Out of curiosity, what exactly is a "think tank"? Is it the sort of thing a person can make money by founding? What makes it distinct in type from other businesses or organizations? I've only ever met one person that had anything to do with a think tank at a party about a decade ago, and I never got her contact details so I'm still very much in the dark.

knuckleheads 6y ago
Good question! A think tank is an organization that researches and sometimes advocates for the government to do certain things. They can be a non profit or an llc (knuckleheads’ club is an llc right now because it was the easiest to set up). In theory, you can make money from founding a think tank, but in practice it’s not a lucrative endeavour unless it becomes big enough that the president start earning steady six figures (like at the major ones in dc).