20 ms·

Ask HN: Why does Pinterest dominate Google text search results?

689 points by Winterflow3r 6y ago

More and more often when I search (using text queries, not image search, which I know has been polluted by Pinterest for years), I get pages upon pages of Pinterest results, sometimes the same Pinterest page but from the different pinterest country domains like pinterest.fi for Finland and pinterest.se for Sweden. Does anyone know if Google gives Pinterest preferential treatement in SEO rankings? Edit: A few comments were asking what my queries were to generate search results where Pinterest dominates, so clarifying that a bit. I run a site that has a colour search engine for lipsticks and since Google is one of the dominant ways in which people land on my site (searching for things like "nyx budapest lipstick dupes"), I was studying various makeup related queries to see which sites ranked highest . Edit2: Edited the title for clarity - I mean text search, not image search

DanielleMolloy 6y ago
Just wanted to come here to agree. I blacklisted pinterest (and some other garbage content with forced registration for the purpose of data collection, like Quora) long time ago using a browser extension. It is particularly annoying when searching for images. I never clicked on pinterest either and wonder why Google is still showing me these results. I wish they would natively allow blacklisting specific domains.
derenge 6y ago
“Deconstructing Pinterest’s reverse-image-search SEO growth hack” https://www.rankscience.com/blog/pinterest-image-seo-growth-hack https://www.rankscience.com/blog/pinterest-image-seo-growth-...
- amelius 6y ago
  Ugh, this is bad. Makes you wonder, why they let this happen. Do people at Google own Pinterest stocks?
- Winterflow3r 6y ago
  Oh wow :( I'm a bit scared by this, because of what it means for smaller competitors, who can't afford to pour investor dollars into hiring a whole team of software engineers just focused on SEO driven growth
hayksaakian 6y ago
There's no intentional or manual effort at Google to promote pinterest. Pinterest shows up because they understand how the Google algorithm works and built their website to display all the signals that Google looks for in relevant image content. They understand user intent and generate URLs that present content in a way that google expects to see. Examples of how they do this from their engineering team: https://medium.com/pinterest-engineering/demystifying-seo-with-experiments-a183b325cf4c https://medium.com/pinterest-engineering/demystifying-seo-wi... More: https://medium.com/pinterest-engineering/tagged/seo https://medium.com/pinterest-engineering/tagged/seo
- blunte 6y ago
  Google should be asking from a product perspective if they are providing the search results the user is most likely seeking. I seriously doubt that most pinterest results are what users are seeking. As such, Google probably should consider changes to their algorithm to compensate (cat and mouse) for pinterest's gaming. That is assuming how pinterest does it is not against any google rules... because if it were against any rules, they should be slapped down hard. When my searches result in mostly pinterest results, I give up more quickly. That means I spend less time using google, and that in theory means less money for google. Unfortunately, I do see other more regular user types operate their computers, and they are less or un-descriminating about the domain name listed under the image. So they'll head directly into the pinterest waste pond anyway.
  - kauju 6y ago
    Just do -pinterest in your search to filter
    - erichocean 6y ago
      It would be great if Google had a little "hide this domain for me permanently" next to every search result. Not only would it improve that user's own search results, it's a very strong signal that other users also might not like it, for de-ranking purposes.
jldugger 6y ago
Are we using different googles?
- PopeDotNinja 6y ago
  I see Pinterest appearing disproportionately in image search.
  - oneplane 6y ago
    I practically never see Pinterest in the results. I suppose it really depends on what you are searching for.
- Winterflow3r 6y ago
  We might be doing very different queries :) If you search for makeup related content, Pinterest comes up a lot because people tend to pin and curate makeup related stuff there
  - jonas21 6y ago
    If lots of people have spent time curating the Pinterest content, and searchers are likely to be looking for curated content, doesn't it make sense that Pinterest would rank highly in the search results?
partingshots 6y ago
Pinterest spends a large percentage of their budget on gaming the search page. Though I’m sure you’ve noticed in recent years that Google has begun to replace many search results with answers directly from them instead of redirecting you to places like Pinterest. For the better I say. Companies like Pinterest are parasitic and degrade the overall ecosystem of search.
- Winterflow3r 6y ago
  Yeah - I've just been reading some of the Pinterest engineering blogs linked above about their sophisticated SEO experimentation framework and ugh, I think it's time to load up my browser with extensions or move away from google completely. Image search has been borked due to Pinterest results for a few years now, but now my text queries have also started being dominated more and more often by their search results. I'm bummed that Google doesn't count the different Pinterest country domains as duplicates and remove them. instead there is like pages upon pages of the same Pinterest result.
- tpxl 6y ago
  > Google has begun to replace many search results with answers directly from them > Companies like Pinterest are parasitic Surely you mean companies like Google are parasitic?
  - 9HZZRfNlpR 6y ago
    Pinterest doesn't provide value with their image search spam hence they are parasites indeed.
    - tpxl 6y ago
      I see my sarcasm didn't go over well. Replacing search results with quips from websites is parasitic as fuck.
underseacables 6y ago
There’s a great Chrome extension that will remove those from search results.
- ffpip 6y ago
  google.*##a[href*=".pinterest."]:upward:(1) google.*##.g:has(a[href*=".pinterest."]) Add to uBlock Origin. in the "My Filters" tab. This will completely block pinterest. No need to use more extensions that can access your cookies and history.
rdlecler1 6y ago
Another example about how google is broken. It has great difficulty distinguishing noise from signal.
- blunte 6y ago
  It's also another example of how modern tech companies are broken (or lacking ethics). Currently, many big tech companies do basically whatever they want. Facebook and other ad companies have been caught knowingly defrauding ad spend customers, Amazon has been knowingly permitting counterfeit sales for years, Uber has been ignoring regional laws globally, Airbnb was (and may still be) ignoring or hiding data that illustrates their connection with increased housing pressures, and on and on. Doing what you can get away with (because the cost of the penalty is lower than the profit that results from the bad behavior) is not ok. It doesn't contribute to an improved civilization, it pushes civilization back down toward a zero-sum, selfish survivalist mentality. I know it sounds corny or hippie, but if people can try to better balance their own desires against the good of the community, the end result will surely be better.
  - ClumsyPilot 6y ago
    The economy is meant to be built so that anything harmful to the rest of society is either unprofitable, or criminal (i.e. theft). Obviously we have a new frontier of digital stuff, but the law and economic incentives are still ignorant of it.
    - blunte 6y ago
      There are plenty in power now who would disagree with your definition of economy. Sometimes law and policy is ignorant of technical concerns, but quite often it is aware and simply has a more selfish agenda. Prime example of this is modern US copyright and patent laws and enforcement. As long as elected officials have to raise funds to campaign and win, elected officials will be beholden to the people who financed them. If politicians didn't write special laws and loopholes for corporate special interests, they wouldn't last more than one term. They would be replaced by the next challenger who would take special interest money and then "pay it back" via special laws.
ffpip 6y ago
google.*##.g:has(a[href*=".pinterest."]) google.*##a[href*=".pinterest."]:nth-ancestor(1) Add to uBlock Origin. in the "My Filters" tab. This will completely block pinterest
- blunte 6y ago
  Unfortunately this did not work for me. I added these two, and then I did a search on images.google.com for "custom carved door frame". 5 of 7 images on the first row of results was some pinterest site.
  - ffpip 6y ago
    Can you try this? google.*##.g:has(a[href*=".pinterest."]) google.*##a[href*=".pinterest."]:nth-ancestor(1)
    - boraoztunc 6y ago
      Thank you that works, eliminated Pinterest results.
    - edoceo 6y ago
      ffpip is the real Santa
    - gregsadetsky 6y ago
      Thank you so much! A very quick copy paste into uBlock and it works wonders for Google & Google Image searches. I wish I had looked into this before! :-)
    - aarchi 6y ago
      nth-ancestor(n) and upward(arg) look handy. They're uBO extensions. https://github.com/gorhill/uBlock/wiki/Procedural-cosmetic-filters#subjectnth-ancestorn https://github.com/gorhill/uBlock/wiki/Procedural-cosmetic-f...
    - blunte 6y ago
      Many thanks, this does work. I do see a momentary flash of pinterest results, but they quickly disappear (and subsequent matches are re-tiled up). Good enough!
perardi 6y ago
I’d love to be able to block a domain from showing up in my search results. Pinterest being an obvious offender, but I can imagine people wanting to block certain news sites, or Fandom/Wikia sites, or all sorts of link farm spam sites.
knuckleheads 6y ago
One big reason why Pinterest ranks so highly, and why you are seeing different tld's in the results, is because they have a variety of tld's that they use and those tld's all have separate crawl budgets from each other. So Google crawls Pinterest about an order of magnitude more than most other sites. You can read more about this here: https://medium.com/pinterest-engineering/how-switching-our-domain-structure-unlocked-international-growth-e00c8184d5dd https://medium.com/pinterest-engineering/how-switching-our-d... Source: I run a think tank focused on Google’s web crawling advantage and have been studying stuff like this for a couple years now.
- marmaduke 6y ago
  Does your think tank come up with any ideas on how to keep companies like Pinterest from peeing in the pool so to speak? They made it impossible to find DIY craft stuff that I used to search for, and I can’t imagine it’s the only site doing it.
  - knuckleheads 6y ago
    We are pursuing ideas that we think will increase competition in the search engine market. Our main focus right now is making Google’s index of the web available for use by competitors, since no one other than Google is really allowed to crawl the web. If people were allowed to use the index to build what they want, then you or somebody else would be able to build a search engine that avoids the Pinterest cruft and would be more suited to your needs, instead of Google’s one size fits all approach. So, to answer your question directly, yes we do have some ideas about how to do this.
    - burnthrow 6y ago
      > making Google’s index of the web available for use by competitors What incentive would Google have to continue populating that index? Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? > since no one other than Google is really allowed to crawl the web Maybe this is the problem that needs solving.
joegahona 6y ago
I do not know whether Google gives Pinterest preferential treatment. That said, I'd like to know: What are you searching? Results will be different depending on if you're searching "cute bedroom ideas" or "epistemology in pre-Socratic Greece."
- Winterflow3r 6y ago
  Ha good question! I run a company that builds visual search engines and one of the sites I build and develop is a colour search engine for lipsticks. One of the main ways people discover it is through Google searches so I was studying which sites rank highest for queries like "nyx budapest lipstick dupes" etc. So yeah, for sure, queries for stuff that lends itself well to visual curation will be full of Pinterest results unlike stuff about Greek philosophy.
- jeffbee 6y ago
  Give it a few days and this page will be the #1 hit for "pre-Socratic bedroom ideas"
  - deleted 6y ago
    [deleted]
- nullsense 6y ago
  "epistemology in pre-Socratic Greece." has a few results from Pinterest.
shmerl 6y ago
You can search with adding `-pinterest` to remove all that noise.
burnthrow 6y ago
And the Pinterest pages have no content! They've beaten Google or Google doesn't care anymore, either way, plummeting result quality is a strong signal that Google Search is ripe for disruption. Ten years ago, Google would've punished all Pinterest properties for this.
jeffbee 6y ago
How much of the bafflement stems from the fact that Pinterest's own audience is overwhelmingly women, while the general search audience is 50/50 and the HN audience is, judging by the character of the comments, 99% men?
udioron 6y ago
I feel like pinterest is being a "placeholder"/"fallback"/"default" of the web. Each time I encounter quora or pinterest search results I am getting the feeling I was looking for something that does not have good online answer/content.
- baobabKoodaa 6y ago
  Did you look beyond the Quora / Pinterest / other SEO spam results? There's good content on the web for virtually any query, it's just often buried under mountains of SEO spam results.
  - udioron 6y ago
    I did. Usually I am not clicking the pinterest/quora links and I am left with nothing.
hftf 6y ago
Some relevant prior discussions: https://news.ycombinator.com/item?id=21622322 https://news.ycombinator.com/item?id=21622322 (Nov 2019) "Tell HN: Google should drop Quora from search results" 1000+ upvotes https://news.ycombinator.com/item?id=16613996 https://news.ycombinator.com/item?id=16613996 (Mar 2018) “Pinterest needs to be removed from Google IMO” 1100+ upvotes https://news.ycombinator.com/item?id=16388833 https://news.ycombinator.com/item?id=16388833 (Feb 2018) And many more: https://www.google.com/search?q=site%3Anews.ycombinator.com+%22pinterest%22+google+images https://www.google.com/search?q=site%3Anews.ycombinator.com+...
bane 6y ago
I solved this nonsense by installing a plugin in chrome that automatically adds -site:pinterest.* to all searches. Almost living in bliss ever since.
- mancerayder 6y ago
  Care to share? A convenient, trustworthy plugin that lets you exclude sounds ideal. Anything for Firefox?
  - bane 6y ago
    Unpinterested
ehsankia 6y ago
This answer has never made sense to me. Pinterest breaks the 2 signals I would assume Google search to use: 1. Often, the image I'm looking for isn't the main image on the page, just one of the small thumbnails and very hard to find on the page 2. Almost every single time I click on a pinterest result, I end up going back because it doesn't help me, i assume others do the same.
bra-ket 6y ago
Facebook users are linking a lot to Pinterest
victor106 6y ago
Wondered the same about Amazon as well. They show up first for most products. Are those organic results? Or do they have a kickass SEO team that games Google’s algorithm.
deleted 6y ago
[deleted]
g051051 6y ago
I can't think of the last time I did a search and got any pinterest pages in my results. It must be more a factor of what you search for.
- pkamb 6y ago
  Are you referring to web/text search, or image search?
  - g051051 6y ago
    Web/text, but on the rare times I do image search, Pinterest results don't "dominate" my results as OP suggested.
bserge 6y ago
Well, Google and Bing do favor big or older/established websites. And Google's search is trying to be smart when it clearly isn't. I'm getting sick of my searches being "corrected", sometimes they even ignore the quotes.
the__alchemist 6y ago
Could you please post some examples of search queries (text or image) that are dominated by Pinterest? I share the sentiment in other comments that search engines are (deliberately?) allowing SEO abusers to degrade the user's experience. I noticed that for some queries (Like "How to brush teeth"), there are more ads on the first page than results! They're marked, but not in an immediately-visible way; I'd be most people integrate them with search results mentally. I started throwing together a search engine about a week ago to address these concerns, and put it online yesterday. (https://www.pageref.org https://www.pageref.org). I'm deliberately penalizing SEO abusers, and promoting websites that have high-quality content. Running custom searches on these sites in some cases based on keyword. In a lot of cases, I'm throwing in search queries, and going through each result one at a time, and categorizing them; eg penalizing sites that use clickbait or scammy ads, that are low-quality but show up high in results etc, or probably aren't relevant, but are highly SEO optimized, like Pinterest.
- CapricornNoble 6y ago
  >>>Could you please post some examples of search queries (text or image) that are dominated by Pinterest? These are all image searches "sci-fi combat exoskeleton" -> 7 of 24 top images are Pinterest "zen minimalist clothing" -> 8 of 24 "grav tank concept art" -> 9 of 19 Like many other users in this topic, I run most of my G-Image searches with "-pinterest". I use DDG as my primary non-technical search engine, typically only switching to Google for images, trying to solve error messages, or researching frameworks/APIs/etc.
  - the__alchemist 6y ago
    Thanks! Can rep
grishka 6y ago
Related: in the last several years, there started appearing these StackOverflow mirrors. It's gotten to the point that you can't google a programming question without running into at least one. Some copy the questions and answers verbatim, some use machine translation to (crappily) translate them into my native language. Sometimes they even rank higher than the real StackOverflow. I really wish there was a non-hacky way to ban sites from Google search results. I also feel like Google's ranking algorithm is utterly broken since it's amenable to this kind of exploitation.
- ratww 6y ago
  They're pretty old. I remember when Google added an option to block domains (the option you want) around 2011 and they were the first thing I removed from my results, along with Wikipedia mirrors. After the option to block domains was removed around 2014-2015, I rarely came across those anymore, but it seems they're back...
  - jjeaff 6y ago
    There are extensions to permanently block sites from results. Like: personal blocklist https://chrome.google.com/webstore/detail/personal-blocklistnot-by/cbbbhelcpfjhdcncigdlkabmjbgokmpg?hl=en https://chrome.google.com/webstore/detail/personal-blocklist...
    - grishka 6y ago
      I know these exist and I could as well write uBO rules to hide unwanted search results, that's why I said non-hacky ;)
- dalmo3 6y ago
  I've even had one of those spammy websites appear as the first result, in a "featured" box. It's just sad. https://news.ycombinator.com/item?id=25437378 https://news.ycombinator.com/item?id=25437378
- alfu 6y ago
  I've been seeing the same even in English language.
somerandom895 6y ago
Pinterest search results are so frustrating. There's a page that's close to what I want - but I can't get to the text ... I don't want a picture, I want to read a document. Seems like something is off, cuz it's crowding out sites with real content.
omaranto 6y ago
For me this is an instance of Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no." :) I very rarely see Pinterest in my search results.
zelly 6y ago
It should violate the duplicate content guideline, since 100% of Pinterest content is not original and lifted from another website. Then again G also ranks those Markov-generated "blogs" with nonsensical text, not sure what to call them but you've seen them. It seems SEO is still alive and well despite all the claims of its death.
intrasight 6y ago
It does dominate image search which sucks. I instinctively add "-pinterest" to my search. I don't have this problem on text search.
tomaszs 6y ago
I read the question as follows: how to rank so high like Pinterest does? The answer is: you should not spend any second on SEO. It is time and money wasted if you are a SMB. Anything else, including taking a walk is better for your business than going into SEO optimization. Your clients are people who buy your stuff. Search engine robot is not your client. After years of working with SEO it becomes clear for me, if you satisfy a robot, you don't satisfy your client. These are two different things. The bright side is that when you will serve your clients, robots will catch up eventually. But it is search engine problem not something you should spend time on. There is no magic bullet method to rank high. No special hack you can use. Everything you will read about SEO is smokes and mirrors. If you have million dollar company and did everything else right - sure, you can throw hundreds thousands dollars into SEO and pray search engine devs won't change their minds. But as a SMB company, counting each dollar, just leave it. You won't win over search engines and people with bulk of money. Spend your time and effort on clients. It is the only way to spend it right. If search engine will change mind - so be it. You won't be dependant on it. And your clients will come to you because they love your product, it is stronger than being number one in a twisted search ranking.
teunispeters 6y ago
I occasionally do copyright searches for owners of images (it's a thing in online groups where "do not post images without permission" is a thing) pinterest is full black hat when it comes to image ownership. It owns nothing it has on its site and without an account, there's no way to trace ownership using its links. And usually there's so many duplicates of the same images (without credit) that it swamps all search results.
systemvoltage 6y ago
Pintrest has the worst UI/UX of any website. It is a cancer of the internet, contributes nothing and allows re-aggregation and front-running in Google search as a feature. It baffles me why people use this piece of shit.
- person_of_color 6y ago
  This. Imagine working for Pinterest.
- RobertKerans 6y ago
  An image bookmarking app that can suggest similar images is a genuinely useful thing. That allows a user to rapidly collate a set of images based on some theme (so for example I use it for design/illustration projects). Despite the UI, it works, and I can't think of any other apps that do the same thing and have a better UI, so :shrug:. However I'd still agree with you 100% on its cancerous nature: it's an app with a limited, specialised use (emphasis on app). That app leaking and pissing all over search results is not at all good and, as you say, contributes nothing for most people. By analogy, if I have some todos saved in a todo app, what would be gained (or more tellingly who would gain) by making my todos appear in public search results? Best answer I've got is simply that advertisers want that data, and want users on that app to advertise to (so, fuck you, users of search who were expecting useful search results).
  - systemvoltage 6y ago
    Wanna start one? There is an interesting overlap between submission of links (HN/Reddit/Lobster), bookmarking them (Del.icio.us), image based collections (Pinterest), and push-pull feeds (RSS/Feedreader).
    - RobertKerans 6y ago
      Yes, I think that would be a good project. I think the major hurdle is that the thing that makes it useful is an extremely large bank of images, and ability to share within the app. So that implies a social aspect, and then that needs to be gated off from search results This has existed several times before -- this is very vague because I'd forgotten about until now, but the one I used to use in the early 2010s (?) was an invitation-only site with very simple UI, almost bare HTML (similar to pinboard), a weird name and a logo that was I think a gorilla. Used to have very high quality results, albeit highly focussed on art/design, with a small, dedicated user base Edit: it was FFFFOUND!, shuttered in 2017 (Pinterest seems to have killed most of those sites, so how does one go about breaking that monopoly?)
noja 6y ago
Why does Google let Pinterest dominate their results?
alkonaut 6y ago
Should be pretty simple to make a browser addin that adds “-pinterest” to search queries (I very often add that and search again, if I get a Pinterest result at the top).
alkonaut 6y ago
If Google decided tomorrow that Quora and Pinterest is spam and should basically never be in the top 10 results pages, couldn’t they just fix it? What’s the worst that could happen?
- mianos 6y ago
  They would be sued. Pinterest and Quora don't actually care that their actions make the web crappy for us all. They would stand on their high horses and complain that google is treating them unfairly. Normally I would assume this problem is because all the cool kids at google are working on more interesting problems but in this case I honestly believe they can't make a change for the betterment of the majority of their users.
  - alkonaut 6y ago
    Does google have a legal obligation to be fair or transparent? I thought what made google tick was the army of manual and arbitrary blacklisting where SEO cheating like e.g showing a full article to the google bot and a “subscribe to read the full article” to everyone else was way over the line that could lead to a ban
- summerlight 6y ago
  DoJ, which is in charge of the on-going antitrust lawsuit against Google, will be very excited about that change.
dwighttk 6y ago
Is there a ddg bang for search without returning Pinterest? (I mean I think -pinterest should work but maybe fewer characters, and I’m not even sure “-“ works anymore.
IgorPartola 6y ago
I am curious how your site accounts for different screens representing colors differently. It seems like it would be a hard problem to solve for this use case.
mancerayder 6y ago
Certainly worth mentioning the WP article from October that highlights Google search's deterioration only in the last few years. https://www.washingtonpost.com/technology/2020/10/19/google-search-results-monopoly/ https://www.washingtonpost.com/technology/2020/10/19/google-... "How does Google’s monopoly hurt you? Try these searches. Right under our noses, the Internet’s most-used website has been getting worse" Yeah, sure, Pinterest knows how to game the results. But the rules of the "game" are in Google's control, so... I hope we don't normalize a deterioriation of a valuable resource under some neutral-sounding algorithmic play.
swyx 6y ago
i specifically hate how they hack slap a login wall on images, and dont allow you to right-click+save on mobile. they're intent on breaking the mobile web for their own self interest. if you're going to SEO yourself to the top of google, good for you, but please dont use google as non-optional lead capture for your app that I don't want.
- swyx 6y ago
  update: i wrote a personal google proxy to tweak google results how i want them. managed to get it recognized as a search engine (on my android chrome) via opensearch. not sharing the code for obvious reasons https://twitter.com/swyx/status/1342625544320339969 https://twitter.com/swyx/status/1342625544320339969
gilbetron 6y ago
You can add: -site:pinterest.com to your search to exclude pinterest.
aerosmile 6y ago
For two reasons: 1. When people see a search result from Pinterest, they tend to click on it more than on the results from competing publishers. 2. After they click on it, they tend to engage with the content more than they do with the content from other publishers. This is not specific to Pinterest, it's how modern SEO works. You'll notice that Pinterest doesn't dominate every search result in the world, but predominantly the ones where the user's search intent is more aligned with Pinterest's strengths (product discovery, visual stimulation, pet/animal pictures, art direction, etc.). I think what people find confusing is how can Pinterest compete with other search results that have far more text? Well, it's not about the amount of text at all, it's about satisfying the search intent. And when someone types in: "what should I wear for my wedding," images are better in answering that question than text (a picture is worth...). As Pinterest's success in SEO teaches us, there are many more similar instances than we would have expected.
stevesearer 6y ago
Pinterest is very good at SEO and does some things that I have found to be quite interesting. I posted this before in another post a while ago, but it is still relevant: “For one of my boards that ranks #1 in Google on some searches I've found that the page Google indexes is quite a bit different than the one I see as a logged-in user. One of the differences is that they display the text content associated with the pin. This is also used as the image alt text, but then appended with a bunch of keywords. They also link to other people's boards which have names related to the images so it looks like "tags", but I have the feeling it is probably a mix of keyword stuffing/linking to other content for Google to follow. The page title is also adjusted to include something like, "237 Best ________ images in 2020" followed by the board name.”
deleted 6y ago
[deleted]
tjpnz 6y ago
My girlfriend and I watched Interstellar the other night. She had a few questions at the end and I remembered there being some great infographics that would help explain. So I reached for Google Image Search and the results were dominated by Pinterest. It wouldn't have been so bad if I could've viewed the original full-size image - but for that I would've needed to create a Pinterest account which I absolutely didn't want to do. After that I gave up and drew my own with pen and paper.
pinterestsucks 6y ago
I heard there was some incestual stuff at medium-high levels of relevant google org/teams and pinterest in terms of people moving between the companies and being friends/favors. Also that these specific teams at google are mediocre at best now and all the good engineers and product focused people have moved on to greener pastures. Probably why google image search has stagnated hard (I find bing and yandex out perform it easily depending on your searches). The explanation that Pinterest is just "good at SEO" never made sense to me because at their scale someone at google obviously would notice and could decide to counter their SEO tricks etc (google has never had problem with doing this before). Not to sound conspiratorial but from what I know I find it more plasubile that mid level decision makers at google can block any change that would nuke pinterest because they or their social circle have personal interest in keeping pinterest up while issue is not close to big enough or impacting larger revenue to attract attention from above.
- 9HZZRfNlpR 6y ago
  Pinterest CEO is an ex Google worker and many investments into Pinterest come from Google related people. They obviously turn the blind eye, anyone in the business knows that since Google is very keen on blackholing people or companies who use tricks like that. Just another Silicon Valley affluent people love story.
bhartzer 6y ago
Google has been rewarding a lot of the top sites that have the highest trust and authority lately. Pinterest being one of them. This has been going on for a while now.
austhrow743 6y ago
If Pinterest is one of your main competitors for search engine results there’s a good chance you’ve clicked on them plenty of times to check out what they’re doing. Google is going to personalise your results on that and they will think you like Pinterest. I personally practically never see them as a result. Been a while since I’ve done any seo but iirc there are tools you can use to try to get the most objective serps.
intricatedetail 6y ago
I always thought Pinterest was Google's site and that's why it was coming up so often.
didip 6y ago
I still remembered Google Panda update in 2011, destroyed many low effort SEO farms. I wonder why Google stopped fighting this good fight anymore.
rukshn 6y ago
I search lot of academic suff like medicine and programing. Sometimes I search for image results, and my search results are polluted by SlideShare and Slide player SidePlayer is an ad infested site which also pirate PDFs by crawling for open PDFs and automatically uploading to their website I don't understand why Google gives priority to such websites. I believe including pinterest and sites like SlideShare and Slide player in our search results is what google should allow us to opt in not something we should opt out
dwd 6y ago
Nice timing on this question - I'm actually deconstructing a website mockup (PDF) this morning and needing to find the originals of the images the graphic designer (who is off on holidays) used. It's not just the dozens of Pinterest copies, but even Medium or private blogs that are using an image and still ranking higher. I find it disappointing that Google doesn't apply more authority to sites like Unsplash when it is clearing the original source.
noomen 6y ago
FWIW I use a greasemonkey script called "google hit hider by Domain". I used it to manually permaban all pinterest domains when they appear and now I almost never see a single pinterest result in my searches.
dazc 6y ago
This complaint features regularly on HN so [my assumption] it has to be wilful blindness on the part of google?
tobyhinloopen 6y ago
I strongly dislike pinterest for this very reason. I wish I could block it from search results
seanwilson 6y ago
For people saying Pinterest has mastered SEO and that's why they're at the top, what exactly are they doing that is so effective? I'm not understanding how individual Pinterest pages or users would gather that many quality backlinks for example. Do they have quality backlinks? If not, why is lack of backlinks not hurting their rankings?
IamZeroBalance 6y ago
I often check my images to see how well there are appearing in the google search. Many times I have found to my delight one of my images clicked on it only to find it leads to another picture on Pinterest and is a completely different product and no link to me or my artwork or product. It is very upsetting that original source takes second place to third parties and these pins are vanishing and being replaced. I have no idea how they are doing this but it is making using Pinterest to show my original art and work a place to consider complete withdrawing from.
whatsmyusername 6y ago
Google hasn't returned the best results for years. They return the results someone paid to put there. Either through direct paid ads or through heavily SEM/SEO influenced results for everything except the most esoteric topics. We actually ended up having to do defensive SEM/SEO on bing simply because of how cheap it is to game results there, just because they're not google (no one does it except scammers and people addressing scammers).
saeedynz 6y ago
John Muller (from Google) says about Pinterest: I'd argue there's a lot of content on Pinterest -- even if it's not a collection of 3000 word blog posts. Sometimes images, even with minimal textual content, can be exactly what people are looking for. Not always, and sometimes we get it wrong, but it's certainly an option. https://twitter.com/JohnMu/status/1260171380580024320 https://twitter.com/JohnMu/status/1260171380580024320 ========= Regarding Pinterest, while I'm personally not a user of the site, there is a ton of content there. It's not 3000 word articles, but you don't need that for search anyway. https://twitter.com/JohnMu/status/1316668506918866944 https://twitter.com/JohnMu/status/1316668506918866944

thaumaturgy 6y ago

User: This service has a problem and it's not working well. Programmer: There's nothing wrong with the service, you just have to use [magic incantation] to make it work right. I've been hoping for 20 years that this kind of exchange would stop happening. Still hoping! :-)

npteljes 6y ago
Services and users have different goals. Why is the helpful technologist the problem?
- grendix 6y ago
  Your second statement is enabling bad behavior, your first statement is insane.
  - TeMPOraL 6y ago
    No. They're exactly right. The root problem here is incentive misalignment. Both Google and Pinterest sacrifice value they provide to extract more revenue from their users.
  - npteljes 6y ago
    Care to elaborate on the insane part? I feel sane.
- matthewdgreen 6y ago
  Real answer? Because helpful technologists are the interface between less knowledgeable users and services like Google. We’re the power users and “mavens” who are both highly exposed to the product and have some understanding of what’s going wrong and how it might be made better. We also spend a lot more time talking to that product’s engineering staff than the typical user (see e.g. the fact that we’re here on HN.) When helpful technologists opt out of the problematic aspects of a product by uttering some magic incantation, we essentially remove ourselves from having to be annoyed by the product’s rough edges. This means we’re less likely to help improve the service for everyone, and we’re probably more likely to incorporate similar carelessness into the things we make ourselves.
  - edoceo 6y ago
    How could a technologist, who's not in charge of the $BigTech be able to fix the problem? They can't. So, they offer the user a work-around. Then, the user never complains to $BigCo because of the Help and still nothing gets done. And later, we complain that we're not fixing $BigCo and we're all mad at "the help". How, really, could you improve the Google service for everyone?
    - wizzwizz4 6y ago
      Acausal decision theory, given the assumption that you have the same decision-making algorithm as the technologists in charge of $BigTech. (The assumption's completely invalid, so it doesn't actually work.)
    - matthewdgreen 6y ago
      How many Google Search* engineers saw this article on HN? I guarantee the number is greater than zero. * Or people who talk to Search engineers regularly, etc.
  - npteljes 6y ago
    I see your point. Working around abusive behaviour, and accepting that as a solution, just enables further abuse. I don't think we can influence said engineers here on HN. I know for a fact that I personally wouldn't be influenced, and I also think that if the product managers will want something, then it will get done, no matter what people write here.
kenniskrag 6y ago
"Programmer" here is wrong. Because we are all user of the product "google search". But yes I see your point. I call it the "permanent workaround"
IanCal 6y ago
And there's always this kind of response. They didn't say the service was fine, they just provided a simple workaround which doesn't require forcing a change to Google's algorithm. Hoping they don't help with this seems odd. So, to counter, I'm glad that they did say this as it regularly annoys me too.
- xg15 6y ago
  It's indeed a useful tip - and no offense to the GP - but this kind of reply always feels like the tech support equivalent of a mandatory arbitration clause: like an attempt to solve an issue individually with each person reporting it instead of giving a systemic solution that would benefit anyone. The blame is shifted from devs providing a good service to individual users not knowing the latest workaround.
  - deleted 6y ago
    [deleted]
  - makapuf 6y ago
    Most probably the devs know how to do it but specs written by someone else told them to obfuscate useful functionality.
tshaddox 6y ago
But no one said that there’s nothing wrong with the service. They merely offered a very simple workaround. I can’t imagine why you’d hope for that to stop happening.
deleted 6y ago
[deleted]
amelius 6y ago
Google: we need your personal information so we can improve your experience User: -pinterest Google: we need your personal information so we can improve your experience User: -pinterest!! et cetera, ad infinitum ...
jedberg 6y ago
I think your premise is wrong. The service isn't broken. I'd assume Google has enough people and data to know that it's doing what most people want. They were even kind enough to put in a workaround for the minority for whom it doesn't work well (which appears to be OP).
- xg15 6y ago
  Most people want to have a full result page of nothing but login walls when searching for any kind of image?
  - jedberg 6y ago
    Most people who click on Pinterest links are already logged into Pinterest...
    - abc-xyz 6y ago
      There’s no way that’s true. I’d be surprised if even 1% of the Google users paid attention to the source site underneath the image. And you’d think people with a Pinterest account would opt to search on their website/app rather than going through Google (thus making it even more unlikely that the Google user has a Pinterest account).
      - alisonkisk 6y ago
        Pinterest has a huge user base. They don't need to look at the source if they have an account; the link will work fine. And, no, most people don't bother picking a site before searching. Why would they artificially limit their results? People don't even visit amazon before searching for Amazon products.
        abc-xyz 6y ago
        Pinterest’s user-base is surely minuscule compared to the amount of users using Google Image Search. I’m not saying that 100% of all users with a Pinterest account use their website/app, but many of them will of course opt to use Pinterest’s website/app for their image searches, which means an even smaller percentage of the Google Image Search users will have a Pinterest account.
njharman 6y ago
User: your thing doesn't work the specific way I want. I haven't read the manual and refuse to learn now to use its features. You should make it magically work the way I imagine. Multiply by 100 users with different opinions, all entitled, on how thing should work.
gsich 6y ago
Why? This is a quick fix. Maybe the problem will get fixed "officially" someday. But I have the problem now, I need a workaround now, not in 5 years.

maya24 6y ago

Stop using Google search. DDG works just as well

fuzxi 6y ago
Would that this were true, but it's not. Searching for specific technical issues especially shows DDG's weakness.
- daveFNbuck 6y ago
  Outside of searching for specific technical issues, DDG is pretty good. I use DDG for most of my searches and this is the first I've heard of search results being dominated by Pinterest, so it's at least better on that front.
_AzMoo 6y ago
I use DDG search most of the time, but I find this to be untrue. DDG is pretty good, but struggles with context a lot more than Google does. For example, when searching for an error in a programming library, DDG is likely to return the library homepage, whereas google is more likely to return the specific github issue where the error is being discussed.
- ratww 6y ago
  For text it depends [1] but for images I find DDG to range from slightly to significantly better than Google. Also, I have direct access to the image file URL without having to rely on browser extensions. [1] I wonder if the meme "DDG is not that good" is so prevalent on Hacker News because it sucks for programming stuff.
  - missedthecue 6y ago
    Yes, Google image searches produce wild results. Far-right groups have had a field day with it.
- the__alchemist 6y ago
  Agreed. I switched to DDG a few weeks ago as a default. I appreciate the privacy, but for some categories (like tech questions) the results are just... not as accurate. For general queries they're comparable. For searching specific issues, Google is more likely to show relevant discussion threads of the exact issue.
- ldng 6y ago
  This used to be true. Regularly I had to use the !g bang to find what I was looking for. Lately, I've realised that I hardly use it anymore and I do results are far worse on Google. Your mileage may vary of course but I don't bother with Google anymore.
  - mancerayder 6y ago
    Same. I went the DDG route a couple of years ago, and searching tech topics made me constantly do g!. And a few other things. Suddenly, Google's results go so much worse, that the duckduckgo offering just proved to be actually functional by comparison. And I think it actually has improved over the years. It's now my default browser search and I'm comfortable not having to worry about Google capturing every feeling or thought I have when I google something, and storing that for eternity. Not to mention the awful vendor blogs and stores and secretive marketing clickbait masquerading as a neutral information totally dominating Google's page 1 and 2 of results.
TaylorAlexander 6y ago
I’ve been using DDG as my default for years and it really fails to provide me helpful results for programming and Linux questions. I know this because I frustratingly have to add !g to queries all the time lately and I’m questioning my choice to default to DDG. That said, I don’t like being tracked. It’s not clear to me if the quality of results I am seeing are due to Google’s tracking or not. (I suppose I could research this but I’ve not done so.)
- ldng 6y ago
  It has improved for me to the point I don't use !g anymore. Maybe subconciously I now do queries that fit DDG better. Anyway, I get the answer I want so can now live without Google.
joe_the_user 6y ago
That's problem, DDG works nearly exactly the same (except a bit worse imo). DDG channels a lot of Bing, which virtually clones Google which makes it not much of an alternative at all.

fiddlerwoaroof 6y ago

Yeah, Pinterest is almost always disappointing as a result: their account requirements are annoying and it tends to obfuscate the source of the image or information you’re looking for.

visarga 6y ago

I think Google collects clickthorughs on the search result page and thus knows how people react to its responses. It also knows when the user is quickly returning to search after checking out a link. Pinterest is probably appreciated by people, probably of a different demographic than you.

cft 6y ago
Here is some background on this: Recently, it has been revealed that Pinterest reverse-image searches Google for uploaded images and adds the Google query result to the Pinterest image page for SEO: https://www.rankscience.com/blog/pinterest-image-seo-growth-hack https://www.rankscience.com/blog/pinterest-image-seo-growth-... This degrades quality of Google image search, for example this Reddit post on removing Pinterest got 77k upvotes: Reddit r/LifeProTips to remove Pinterest from Google searches: https://www.reddit.com/r/LifeProTips/comments/hejbl0/lpt_when_using_google_add_pinterest_sans_quotes/ https://www.reddit.com/r/LifeProTips/comments/hejbl0/lpt_whe... Earlier Reddit petition to Google to remove Pinterest from search results (66k upvotes): https://www.reddit.com/r/google/comments/85atho/pinterest_needs_to_be_removed_from_google_imo/ https://www.reddit.com/r/google/comments/85atho/pinterest_ne... Hacker News discussion of Pinterest's Google scraping: https://news.ycombinator.com/item?id=23701998 https://news.ycombinator.com/item?id=23701998 Earlier discussion on HackerNews (1.1k upvotes): https://news.ycombinator.com/item?id=16613996 https://news.ycombinator.com/item?id=16613996 "Unpinterested" Chrome extension to combat Pinterest spam (10k+ downloads): https://chrome.google.com/webstore/detail/unpinterested/gefaihkenmchjmcpcbpdijpoknfjpbfe?hl=en https://chrome.google.com/webstore/detail/unpinterested/gefa...
- visarga 6y ago
  I agree, but if you take uninformed users (or users at random, that don't subscribe to a Google related forum) they would be happy with the Pinterest results because the results look nice at first sight and they don't have time to learn about the problem. Google using this signal would conclude that Pinterest is OK for the users.

1vuio0pswjnm7 6y ago

"There's no intentional or manual effort at Google to promote pinterest." But there is an effort at Pinterest to promote itself via Google. Pinterest pays Google for online ad aservices. Pinterest is a Google customer. The user is not. The bottom line at Google is not driven by showing users the "best" results for users. The bottom line is driven by customers such as Pinterest who pay Google to assist them with online advertising. Page and Brin identified this conflict of interest in their 1998 paper introducing Google, which was originally intended to be an academic endeavor, not to become an online ad services company.^1 According to the authors, (a) the interests of users and advertisers are not one in the same, (b) there are inherent effects on search results if the search engine is funded by advertising, and (c) the behind-the-scenes operation ("technical details") of search engines funded by advertising become largely non-transparent to users/competitors (a "black art"). I reckon (c) is why we get questions like this one from the OP. I have also seen these inexplicable and useless Pinterest links in Google searches. 1. https://research.google/pubs/pub334.pdf https://research.google/pubs/pub334.pdf Quotes: "To make matters worse, some advertisers attempt to gain people's attention by taking measures meant to mislead automated search engines." "Up until now most search engine development has gone on at companies with little publication of technical details. This causes search engine technology to remain largely a black art and to be advertising oriented (see Appendix A). With Google, we have a strong goal to push more development and understanding into the academic realm." "Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users." "For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers." "Furthermore, advertising income often provides an incentive to provide poor quality search results." "But we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm."

alisonkisk 6y ago
Had anyone ever presented evidence that advertisers manipulate organic results? Former employees blow all sorts of whistles. Why has this one never blown?
1vuio0pswjnm7 6y ago
The term "black art" appears to be a synonym for "black magic". Wikipedia (https://en.wikipedia.org/wiki/Black_magic https://en.wikipedia.org/wiki/Black_magic) describes black magic as "the use of supernatural powers or magic for evil and selfish purposes."^1 1. J. Gordon Melton, ed. (2001). "Black Magic". Encyclopedia of Occultism & Parapsychology. Vol 1: A-L (Fifth ed.). Gale Research Inc. ISBN 0-8103-9488-X.

knuckleheads 6y ago

> What incentive would Google have to continue populating that index? Presumably they would still want to run google.com and make money off of it. > Would I be breaking the law if I independently crawled and hosted an index without publishing an API for it? No. You would not get the advantage that Google gets when it crawls the web and so would not have access to a large amount of data that nobody else has access to. Updated based on edit of parent post: > Maybe this is the problem that needs solving. Why have websites waste the money to serve all those requests all over again? Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? I don't think website operators would be happy if there were a hundred more crawlers out there crawling as much as Google does now.

deleted 6y ago
[deleted]
lopmotr 6y ago
Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? It sound a bit ridiculous unless they actually don't want to be found. Maybe they only allow GoogleBot because that's all they thought of and the extra cost is in researching what all the other search engines call theirs. Perhaps other search engines should spoof GoogleBot. Browsers have being doing that since forever spoofing Netscape (Mozilla), Safari, etc. for the same reason. > Why don't we have Google share the results and we can use that money to do more productive things than recreating that work? This sounds like a common fallacy of people criticizing the free market. Duplicated effort looks wasteful but turns out to be far more productive than the lack of incentive that comes with not being able to profit from your work/investment.
- Someone 6y ago
  I would think the site owner’s cost of being indexed is the same for every search engine that indexes the site. The benefit varies with the quality of the search engines, and that will vary between search engines, but it does get larger the more a search engine is used, so a cost/benefits analysis may show Google and a few other large ones are the only ones worth supporting.
  - knuckleheads 6y ago
    Yes! Exactly!
- loeg 6y ago
  Yes, site operators actually block non-Googlebot crawlers. See the example [0] from https://news.ycombinator.com/item?id=25538842 https://news.ycombinator.com/item?id=25538842 . Spoofing crawler identity completely defeats the point of the honor-system robots.txt.
- knuckleheads 6y ago
  > Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler? Many website operators do actually block crawlers from non Google search engines and it's because the cost of being crawled isn't worth it to them. Here's a good quote from one such webmaster: As a webmaster I get a bit tired of constantly having to deal with the startup crawler du jour. From law firms looking for DMCA violations to verticals search engines, to image aggregators, to company intelligence resellers… It feels to me that everybody and their brother has gotten into spidering sites. With 10,000s of pages that have content that is only relevant to a targeted audience who is perfectly able to find us on the majors, I do not hesitate to block (and possibly ban) when I see an aggressive crawler that does not provide me or my customers with direct benefits. Taken from http://www.skrenta.com/2008/04/cuill_is_banned_on_10000_sites.html http://www.skrenta.com/2008/04/cuill_is_banned_on_10000_site... > Perhaps other search engines should spoof GoogleBot. Browsers have being doing that since forever spoofing Netscape (Mozilla), Safari, etc. for the same reason. People have tried this and it doesn't work. Google provides ways to check to make sure traffic is coming from Google IP addresses and practitioners and academics study how to spot fake Googlebots. https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot?visit_id=637445287636222855-805864839&rd=1 https://developers.google.com/search/docs/advanced/crawling/... https://blogs.akamai.com/2014/07/search-engine-impersonation-the-wolf-in-sheeps-clothing.html https://blogs.akamai.com/2014/07/search-engine-impersonation... https://ieeexplore.ieee.org/document/8421894 https://ieeexplore.ieee.org/document/8421894 > This sounds like a common fallacy of people criticizing the free market. I am asserting that crawling the web is a natural monopoly. This means that the free market has failed and that it is not possible for the market to heal itself in this regard. There is significant evidence that this is the case and I imagine you'll be hearing more and more about it soon.

aleph_naught 6y ago

> since no one other than Google is really allowed to crawl the web. ??

knuckleheads 6y ago
There are two main reasons why I say nobody besides Google is really allowed to crawl the web. The first is that Google gets much more access to pages on websites than everybody else. You can see this by examining the robots.txt files of various websites[0]. I've been doing this for several years now and Google has a consistent advantage across many thousands websites that I've looked at. This adds up to a significant advatnage and many search engine operators complain about how it hampers their ability to compete with Google[1]. The second is that Google gets to ignore crawl delay directive in robots.txt while other search engines don't[2]. Website operators cannot tell Google how fast they want their website crawled, they can only request that Google slow down. If another search engine tried to do what Google does, they would likely be blocked by many important websites. If you would like to read more about this, please checkout https://knuckleheads.club/ https://knuckleheads.club/ [0] https://pdf.sciencedirectassets.com/robots.txt https://pdf.sciencedirectassets.com/robots.txt [1] https://www.nytimes.com/2020/12/14/technology/how-google-dominates.html https://www.nytimes.com/2020/12/14/technology/how-google-dom... [2] https://www.seroundtable.com/google-noindex-in-robots-txt-dead-27824.html https://www.seroundtable.com/google-noindex-in-robots-txt-de...
- grishka 6y ago
  So, uh, don't respect robots.txt in your search engine? It's not like there's a law that you have to, and that you can't pretend you are Googlebot. The only real obstacle I can imagine is that some firewalls might be configured to be more permissive with traffic originating from Google subnets.
  - knuckleheads 6y ago
    You would be blocked fairly quickly by many website operators and no longer able to access those websites if you straight up ignored robots.txt files. You also might even end up being served cease and desists by some websites and sued if you continue to persist and try to find ways around it.
    - grishka 6y ago
      And what if you do respect it but follow Googlebot rules?
      - knuckleheads 6y ago
        Applebot was able to get away with doing exactly this but I imagine that's because it's Apple and websites knew that Apple was about to send them enough traffic via Apple News to make it worth their while. I don't know if other search engine operators have tried this but I would imagine they would get caught by rate limiters set for non Google IP's and then they would be blocked.
        grishka 6y ago
        Still, you keep saying all that as if most websites even notice that they're being crawled, and that their operators are very aware exactly when by whom they're crawled. Like as if the admin gets a notification every time a crawler comes by or something, with precise details about it. I don't think it's nearly as serious as you're trying to make it look.
        knuckleheads 6y ago
        I've been a part of a team that operated a large website and I've been paged before because of the issues that somebody was causing because it was being crawled too much. Many people in the web operations field have had the same experience. Generally speaking, the larger the website, the more sensitive they are about who is crawling and why.
        rewq4321 6y ago
        To add another data point for you: I have had one of my websites brought down by Yandex bots before. There are also dozens of no-name bots (often SEO tools like ahrefs, semrush, etc.) that can sometimes cause troubles. For me it was a problem of having lots of pages, and having a high cost per request (due to the type of website it was). For other websites, it is not necessarily about the volume of traffic from bots, but the risk of web scrapers getting their proprietary data. They're fine with Google scraping their info because that's where their traffic comes from. They're not okay with some random bot scraping them because it could be taking their content and republishing it, or scraping user profile data, or using it for some nefarious/competitive purpose.
  - oh_sigh 6y ago
    Google tells website operators how to verify google bots in a way that can't be spoofed.
  - ehnto 6y ago
    It's a situation where the rules seem obvious but the practical realities of it mean Google has the advantage by being the incumbent. No one would dare block Google for a search traffic reliant business, but some upstart search engine will quickly end up on blacklists even with reasonably slow crawling.
deleted 6y ago
[deleted]

marmaduke 6y ago

> making Google’s index of the web available for use by competitors That does seem like a good idea since it amounts essentially to a database of stuff that Google does not own (by construction) and is of public utility.

knuckleheads 6y ago
Exactly. Everybody always talks about turning tech companies into public utilities without really explaining what the utility would be. A public index of the web would be an amazing utility for many of the reasons in this thread and would spur on a ton of new innovation and businesses. If you would like to read more about all this, please checkout https://knuckleheads.club https://knuckleheads.club
- rarefied_tomato 6y ago
  The pursuit of a public index is excellent. Some feedback: - A cost of membership runs contrary to establishing this group, especially at such a high recurring charge. - I'm not sure what your software/AWS situation looks like, but 20 million robots.txt files acquired from Common Crawl is something I can analyze on my PC. It doesn't seem to presently justify such high costs. - Prioritize building a mockup index with an intuitive frontend. This is essential for non-technical people to understand - Exclusively talk with EU legislators (they are motivated, whereas nothing will happen in the US).
  - knuckleheads 6y ago
    Thank you for the feedback! I think the price for membership dues is reasonable and many people agree evidenced by them signing up. I think I might start a petition that is free to sign up on though, thank you for the inspiration! It is possible to analyze those files on the pc, it just takes a much longer time. The analysis is an iterative process and so the faster the computers the faster the iterations and process go. I was analyzing them on my pc with python for the first year until it got too slow and my I am using an aws server with some rust and that is going much better. I also need to increase the number of files analyzed by about two orders of magnitude soon as well. Great idea, very cool. That’s going on the todo list! And I am going to be reaching out to and speaking with whoever is interested. One of the fun things about this is that it is an international dynamic, with some jurisdictions having abilities that others don’t. For example, the UK CMA has subpoena powers that the US Congress lacks and got a ton of information out of Google and Bing that shocked me. The US has the ability to get the CEO’s to show up to hearings while the UK does not in the same way. Why limit ourselves to one government when there are so many to mix and match from here?
- marmaduke 6y ago
  What kind of research is it? Is it just funding you or members can participate? I do Bayesian and non Bayesian data analysis and modeling, would be potentially fun to poke at the data
  - knuckleheads 6y ago
    Right now the research is two parts, analyzing websites robots.txt files for bias and then talking with search engine operators and website operators to get their stories that validate these ideas. Right now I am the main person working on this research but I would like to get the robots.txt parsing and analysis code open sourced soon to allow people to start digging in. Getting people access to the data is trickier, but it feels doable as well. You'd be welcome to join in!

joe_the_user 6y ago

I think others here have alluded this but the thing is that once, doing everything you need to do to get listed but actually being crap was enough to get you de-listed. The criteria wasn't "do what Google wants", the criteria was "be good and also help the algorithm determine that you're good". Only running the right SEO wasn't sufficient then and I certain wish that criteria remained.

knuckleheads 6y ago
Who determines what is the best ranking? Who was defining what was "good"? There isn't some abstract, mathematically verifiable best ranking, a platonic ideal that all search engines should aspire to. Google has always made choices about how to rank that trade off the interests of various groups of users, from the very beginning. The complaints that you and I have about pinterest being ranked so highly come because Google has decided that our desires for how the results should look are not important enough compared to how other, perhaps larger, groups of users want them to look. Maybe Google deranks pinterest tomorrow, but they've made the choice to keep it as such for a couple years now. Whether it's because they think they have better ways to spend their time or because they like it the way it is, it's not really possible for us to know without somebody leaking that information. But, I imagine, there would be a group of people who would be as disappointed as we are elated if Google were to change this.
- joe_the_user 6y ago
  There isn't some abstract, mathematically verifiable best ranking, a platonic ideal that all search engines should aspire to. Of course, "quality" applied is a broad term indeed. But there no being a single certain measure of quality doesn't mean there aren't some things that most people would call crap. Google employs a large number of search quality raters and these folks can likely distinguish the terrible from the OK. Google is far from ideal and getting worse but you seem to be exposing a pure democracy of algorithm conformance, which is guaranteed to result in hot, steaming idiocy.
  - knuckleheads 6y ago
    I think what I am trying to espouse is two things. The first is that there might be a large group of users unknown to you and I who enjoy looking at pinterest in the image results and that is who Google wants to cater to. And the second is not that Google is or should be a "pure democracy of algorithmic conformance" but that Google has, does and always will make choices about what is best and that the definition of best is up to their discretion. They are making choices about who to cater to with their results and justify those choices with their own logic that we will likely never know. One company having the power to be the only one making all those choices isn't anything I would call a democracy.
    - joe_the_user 6y ago
      The first is that there might be a large group of users unknown to you and I who enjoy looking at pinterest in the image results and that is who Google wants to cater to. The discussion is about Pininterest in text searches. My only objection to Pinterest in image searches is that it's paywalled/login-walled. But larger issue is I'd acknowledge some search results just aren't what I in particular want but I'd claim other results are actually objectively low quality and you seem to want to push things to realm of pure subjectivity, any old crap is something someone might want and who I am to deny to them that pnis enlargement pill. I would say that the results Google returns involve a number of factor/filters. A. What's considered mainstream, what appeals to many B. What's could be more or less objectively called quality. C. SEO, What results just slip through based on the page spending a lot of time and money appearing like A or B to the algorithm (but not being that). And Google spends a huge amount of time and money trying to keep C from being the only thing BUT that's still not enough because there's a lot of time and money spent on the other end. At the time, this tug-of-war serves as a moat keeping competitors out. Any Google competitor would have to invest similar amounts of money. Google has, does and always will make choices about what is best and that the definition of best is up to their discretion.* I'd love to have a decent Google competitor. But I don't think you have won many friend here by implying that Google's discretion is just arbitrary (as a number of your posts here seem to imply to me). Google won, back when they had competition, by caring a lot about the, uh, quality of their search results. Now that they've won, they're slipping into other things and moreover, the SEO trash are nipping their heels. Which is to say you won't build Google alternatives on "democracy" but on some concept of quality people will want.
      - knuckleheads 6y ago
        Many people have been upvoting my comments here so the evidence I am seeing suggests that I haven't lost too many friends here with this line of thinking. Continuing out the logic of your suggestion here, you would seem to imply that every other currently existing search engine besides Google also has bad quality as well in this regard. I say this because if other search engines had better quality that Google, they would be doing better than Google as per your suggestion. But, obviously Google is on top in a big way, so I have to ask, do you think the entire search engine industry, not just Google, has poor quality search results? What do you know that they don't?

grishka 6y ago
> the risk of web scrapers getting their proprietary data That's some weird logic, to me at least. That data is literally given away to everyone but some people or organizations can't have it? If you want to control access to it, maybe at least require people to register before they can see it? Is it even proprietary if it's public with no access control whatsoever? This for-profit internet is just really such a parallel universe to me.
- knuckleheads 6y ago
  This is a question the courts are working through with LinkedIn and HiQ https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l... as well as the Van Buren case https://themarkup.org/news/2020/12/03/why-web-scraping-is-vital-to-democracy https://themarkup.org/news/2020/12/03/why-web-scraping-is-vi... It’s a different world where there are no laws or prices or contracts really.
- ehnto 6y ago
  > This for-profit internet is just really such a parallel universe to me. I know I have been a contrary commentor in this thread, but I hear you with this. What a monster we have built, and what always gets me is how trivial everything is. So much capital is flowing through these ephemeral software systems that, if gone tomorrow, would be ultimately inconsequential to mankind.
  - grishka 6y ago
    I mean it's ridiculous to think about it, but there's this giant, many-billion-dollar online marketing industry that I essentially don't exist for. If it's gone tomorrow, I would indeed not notice, but it'd be the end of the world for some. > and what always gets me is how trivial everything is Whenever I read about corporations and how they work, I always inevitably ask myself the question "where the hell does enough work to keep this many people busy even come from". Everything is ridiculously overengineered to meet imaginary deadlines.
- rewq4321 6y ago
  > That data is literally given away to everyone but some people or organizations can't have it? It's often a question of quantity. LinkedIn probably doesn't care about you scraping a few profiles, but if you're harvesting every bit of their publicly-available data, then they get a little scared that you're building something that's going to compete with them. Same with Instagram, or Facebook, for example. Though in this case it's probably more of a user-privacy issue - at least that's what they say. It's not really weird logic to me - seems to make sense. > If you want to control access to it, maybe at least require people to register Most of the time they can't do this because they need the Google traffic. LinkedIn wants a result in the SERP for Bob Smith when you search for "Bob Smith" because that helps them get signups. Google won't list the page if that content is gated by a sign-in/register page.