22 ms·

The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam a

by birken 5y ago

The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. It's a ridiculously hard problem, especially when people are targeting you directly. But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse. I'm not saying that curated search results for particular verticals is a terrible idea (though I'm sure like anything the devil is in the details), but on the whole Google search is very, very good considering the constant assault they are under from spammers (which most other search engines are not, at least directly).

stoicjumbotron 5y ago
Highly OT, but if a technical person (not at a managerial level) involved in tackling spam at Google were to leave the team, are they allowed to work on the similar problem space at a different company?
goatherders 5y ago
This x100000. There is no scenario - none - where thousands of engineers at Google working on search wake up in the morning and say "we sure have made it good enough wr2 SPAM. I think I'll have another Danish."
- compiler-guy 5y ago
  When the cafes were open, you can bet they said, "I'll have another Danish, and then get back to work on this problem that never seems to go away."
  - techdragon 5y ago
    I sure wish I had problems that were totally unsolvable, they are so easy to measure progress on. /sarcasm I think it’s more likely that because they are just building hundreds of tiny tweak experiments and it’s someone else who desides what to build and if it even worked. Search quality is such a meta-problem that it goes beyond any real hope of simply working on it in anything beyond piecemeal trial and error fashion on their dataset.
    - siva7 5y ago
      What is a danish?
      - compiler-guy 5y ago
        A breakfast pastry something like a donut. https://en.wikipedia.org/wiki/Danish_pastry https://en.wikipedia.org/wiki/Danish_pastry
- ericbarrett 5y ago
  I agree with this and the grandparent comment wholeheartedly. That said, there's a kind of institutional blindness that can build up in companies—especially ones that dominate their sector. It may have roots in intransigent upper management, ossified and inflexible process, wide-scale burnout, a culture of passing the buck, or any number of other pathologies. I don't claim that Google has any of these and certainly have no insight into their search group. But I've personally been at powerful companies with best-of-the-best talent that were blind to the decay in their own living room, so I would caution against immediate dismissal of PG's take.
- UncleMeat 5y ago
  Especially since "made a change that improved search result relevance by X%" is an extremely compelling story for promotions. If indeed there is a launch-driven culture for promos at Google then there'd be extra incentive for new mechanisms to reduce low quality search results.
judge2020 5y ago
Also, i'd be very surprised if they didn't have tens of thousands of workers aiding in spam review already. The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it, but many people would still consider that spam since it adds such an obstacle to getting the information you want. Same for cnet articles like "Best smart home devices to buy in 2022" - it's a reputable brand with a list of smart home devices, but it's hardly a review and exists to funnel people to their Amazon affiliate link.
- awillen 5y ago
  AFAIK the 2000 word essays in recipes are Google's fault - it prioritizes pages with a lot of content, so you have to add that junk to the top in order to rank highly. While I'm sure there's more going on behind the scenes than I'm aware of, it does seem like the rules could be altered on a category-specific basis where a lot of text isn't necessarily a positive.
  - behnamoh 5y ago
    Years ago I wanted to pursue micro blogging, but this "feature" of Google search stopped me from doing it. What's the point of writing succinct, to-the-point mini articles about problems and solutions if nobody finds them on Google?
    - skilled 5y ago
      This is largely because micro-blogging means less content, and less content means you could write five 300-word blog posts instead of one 1,500-word post. I've done blogging for the last 10+ years, and many of those I spent as a freelancer working with startups/brands/editorials. Everyone is after "word count" and I absolutely hate it. Whenever I work on articles for my own blog, I just don't consider word-count at all. I think if your content is great and informative, then readership will be natural.
phsource 5y ago
This 100%. In travel, we see Google constantly tweaking its algorithms, and compared to Bing, Google surfaces a ton more small, well-written travel blogs [1] Not only that, Paul and Michael have seen plenty of startups, and at least in recent memory, the number of vertical search and consumer startups that Y Combinator has funded hasn't been that high As a consumer startup, I know this issue firsthand. Paul and Michael assume that if you build a better product, they will come! That's simply not true these days. Instead, you need to: - Build a better product - Option 1: Figure out a channel with enough growth on an existing platform. This likely means you're doing SEO for your new search engine - Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads As the founder of Wanderlog (YC W19; https://wanderlog.com https://wanderlog.com), a consumer vacation planning app [1], I definitely remember the idealistic days when I thought the best consumer product on its own would win! But growth doesn't just come, and the same can be said of vertical-specific search engines. [1] Try searching "[your city] itinerary" on Google vs. Bing: it's much more likely you'll find a small blog rather than Lonely Planet or the local travel bureau as the top result
- noduerme 5y ago
  >> - Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads nonoonononooonono. No. Don't monetize anything for the first 10 years. That's the only way it can work. Then you can go monetize it and buy an island and not give a shit if you destroy what you created. Oh but don't worry. You'll have investors.
- judge2020 5y ago
  [1]: both signed in, but with the profile image removed Bing: https://i.judge.sh/ShareX/2022/01/www.bing.com_search_q%3Datlanta%2Bitinerary%26form%3DQBLH%26sp%3D-1%26pq%3Datlanta%2Bitinerary.png https://i.judge.sh/ShareX/2022/01/www.bing.com_search_q%3Dat... Google: https://i.judge.sh/ShareX/2022/01/www.google.com_search_q%3Datlanta%2Bitinerary%26oq%3Datlanta%2Biten.png https://i.judge.sh/ShareX/2022/01/www.google.com_search_q%3D... Interestingly Google didn't have a top-result ad and the google.com/travel carousel is 4th from the bottom. For the actual results, both thefearlessforeigner.com and paigemindsthegap.com seem to be actual travel blogs (the pictures didn't appear in a reverse image search, so they are probably organic), but they're clearly geared towards being a 'faq' for visiting the city and have affiliate links where appropriate. Bing went straight for discoveratlanta.com, and frommers.com is well-thought-out but not a personal travel blog.
whiplash451 5y ago
I understood PG's point differently. My understanding is that he is suggesting an angle of attack in which carefully crafted manual reviews (that do dot scale) can be used to bootstrap a product that does scale thanks to something else (e.g. collaborative filtering). All of this being on a niche domain where you can drive a wedge into the mediocre performance of Google (online shopping probably being the worse possible choice, but there are many others).
acdha 5y ago
Your point is good but I'm not sure I'd say very good given how easily the same SEO spam domains can stay at the top of search results for ages simply by scraping someone else's content. What I'd be most interested in knowing is what their success metrics are defined as — for example, how much of a problem does Google's management consider it if someone searches, finds the answer they were looking for on someone's Stack Overflow rip-off, and stops searching? I could easily believe that a significant amount of what we're seeing here is that they're focused on some kind of user frustration metric which doesn't include things like damage to other businesses.
- paulgb 5y ago
  Yes, I've noticed this particularly with technical results. A lot of sites seem to have scraped StackOverflow and GitHub issues, put a crappy ad-loaded interface around them, and somehow out-rank the original SO/GitHub content. It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.
  - coldpie 5y ago
    You just have to look at Google's profit motive here. Their motive isn't to provide quality search results. Their motive is to show users ads, either in the search results themselves or on the destination sites via their ad network. The SEO spam sites aren't a bug, they are a feature of Google's profit algorithm. Google's search quality will never improve so long as their motivation is to show you ads. Why should it? Competition may help here, either by an outsider like the OP suggests, or via breaking Google up with anti-trust enforcement, or both (my preference). As a user, your best personal and ethical move is to install an ad-blocker, to make ad-based business models less viable, which will help promote business models that don't abuse the customer.
    - the_other 5y ago
      The core problem, I guess, is that search engines view all their results as ads. That’s why they got into the ad business in the first place.
Sebb767 5y ago
I agree. You can say a lot of bad things about Google, but they definitely have some of the smartest and highest paid engineers working on their search. Plus there are already a lot of people trying to compete with Google and so far, no one seems to provide consistently better results. The only advantage a startup might have is that they could do completely new concepts, such as specifying what area you search in, allow you to modify their classification of your query and/or moderating sites you include - which is probably necessary anyway, since you'll hardly have the budget to fully index the web. I'm not saying it's impossible, but it's not going to be easy at all. And after all of that, you still need a way to make some money.
Alex3917 5y ago
> But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse. The reality is more that some Google engineer will come up with an algorithm change that makes the result 40% better, but it will come at the expense of making that search 3ms slower so the change won't get merged. Or it will make the results worse for some niche set of queries that the business team really cares about, so again it won't get merged. There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results, so there is lots of room for a startup to compete.
- zozbot234 5y ago
  > There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results Google has a paid-for Search API, so they could do that if they chose to pursue it. And then they could let Google One users opt-in to the same thing via ordinary Search. I'm not sure whether Bing has anything equivalent.
deleted 5y ago
[deleted]
xnx 5y ago
A lot of people forget that one of the inputs to the Google ranking algorithm is input from human quality raters who work off of an extensive, 172 page, guide that Google publishes and updates for anyone to read: https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh...
- visarga 5y ago
  Apparently the "human quality raters" never found the sites reported in this thread.
behnamoh 5y ago
Most comments focus on the technical side of things, whereas I'm sure there are also legal restrictions involved in this. If Google delists a website on the grounds that it's a copycat of stack overflow, or because they have low quality content according to Google's taste, there might be lawsuits filed against Google, claiming that the company is discriminating.
- bell-cot 5y ago
  In which countr[y|ies] does Google not have the discretion to decide that certain sites / pages / etc. "belong considerably further down" in Google's search results pages? Seems to me that sorting the search results to #1, #2, #3, etc. is pretty well baked into their basic product.
noduerme 5y ago
How hard is spam, really. If you're Google? Here's what I would do as a heuristic (uh, not evil?): We know everything about you and everywhere you've visited and everyone you've talked to in the last 60 days. We know all their phone numbers and email addresses. We even know the girl's phone number you met at the bar, who didn't give you her phone number. So if any of those people email you, we'll categorize that as "not spam". Also, if it's your boss or a coworker, "not spam". If it's a major company that's existed for more than ten years, not spam. Everyone else, spam. Done. This is hyperbolic, right? But they can solve spam in a split second, if they just admit they're watching you all the time. [edit] /s thx for reading to the end, folks.
dageshi 5y ago
I said this in a similar thread yesterday, but I think this is an unsolvable problem because much of the content either no longer exists in website form or is old. To put it simply, a new generation of the people who used to make the reliable niche websites that not just answered your questions but also helped you learn a particular topic have moved to youtube instead. Google search is hollowing out as a result with the meat going and the SEO'd fluff that kinda answers the question but ONLY the direct question being asked with none of the wider expertise that more educated people in what they were searching for. Of course google owns youtube as well.. so perhaps they just see it as an inevitable transition.
- neom 5y ago
  Just a note on that, youtube search is finally getting better, yesterday I noticed it was able to find key words in the middle of a lecture that had nothing in the title or comments. I always wonder about their AI transcription service, it's gotten so good, if they're storing all that audio as text, I guess their search is going to get excellent?
- numpad0 5y ago
  Is that...essentially a Proof-of-Work system...
rightbyte 5y ago
Really? I don't think the bureaucratic bloat at Google cares and the original authors of the search engine in its current incarnation are probably long gone. It is maintenance mode and I don't think they dare touch too much. It takes time and effort to build up a spam site's ranking but it is trivial to blacklist those who get to the top.
hardtke 5y ago
The curated search results business model doesn't work. Google gives "aggregators" and other search engines the death sentence for organic search traffic from economically meaningful queries, so you'd get no free traffic. This is one of the major antitrust complaints against Google in the EU. Since you get no organic search traffic, you need to build a brand using advertising, and once you start down that road you need to monetize the first click which compromises the quality of your site.
- magicalist 5y ago
  > This is one of the major antitrust complaints against Google in the EU. The complaints I've read are from exactly the kind of generated content farms people are complaining about in this thread.
throwaway6845 5y ago
> I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence Really? I can point you to Hard Problems that have been solved better at little startups than at Google - or, indeed, at any other bigco. That's why acquisitions happen. Why does Google having 1000 engineers working on a problem automatically mean they are the smartest?
numbsafari 5y ago
The problem isn't that Google doesn't employ these people or invest in their activities. It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities. If Google: - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT) - Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool). - Allowed you to create your own custom search engines, like "Programmable Search Engine". That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable. Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.
- ErikVandeWater 5y ago
  > Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT) Are there any search engines that do this? It's a great, simple idea.
  - mad182 5y ago
    Not really that simple, I see a lot of potential for abuse - using bots and brigading to mass downvote your competitors or political opponents. Couple of positions up or down in google results for somewhat popular and valuable keywords can mean the difference in thousands of dollars per day of ad or affiliate revenue. I suspect it would get pretty wild if google launched something like this. There already are black-hat seo methods and services, but something so simple and direct would turn it up to 11.
    - numbsafari 5y ago
      > I see a lot of potential for abuse They already have the tech to fight this on YT. They, in theory, are supposed to be doing the same thing to detect inauthentic behavior on ad placement and click abuse.
cft 5y ago
The search results markedly worsened in the last 5 years. Why could they keep up with SEO spam until 5 years ago, and now they can't? Their revenue has been growing dramatically, so they could proportionally increase the allocation. It's probably because the focus of their HR/changing workforce is now elsewhere: maybe fighting "disinformation": both COVID and political. Those efforts were non-existent 5 years ago.
- specialp 5y ago
  I think it is also no longer in their interest. If you look at their mobile results now, there are sometimes no search results for webpages, just ads, and their automatically extracted data. So, it is in their interest now to have the search for non-advertisers to be bad. Eventually people will consider those results junk and just use the google extracted data/people who paid to go up.
8ytecoder 5y ago
I agree it’s a hard problem. I don’t agree it’s “really really good”. I regularly encounter obviously scammy websites. With Google’s js execution capabilities I’d assume they can detect that. I’m talking about the VPN install pop ups and so on. Right now there’s a whole bunch of GitHub.Io hosted sites that’s doing that. It’s not even porn. It’s home decoration stuff.
JohnJamesRambo 5y ago
I don’t doubt it is hard but I’m forced to sign into Google now pretty much, just let me rate results and ban domains again etc. You will solve the seo problem really quick and start giving me results I want.
wakiza33 5y ago
Spam, yes, but Google has also made meaningful shifts that are clearly directed from the top-down. It's much harder now (imo) to get specific results, they've overall started looping SERPs into broad answers. This is def a user-engagement strategy -- but it has cons as well. Part of the complaints in the thread were spam related, other were something deeper
zozbot234 5y ago
Legitimate sites could help a lot by adding machine-readable descriptions of their content, per the schema.org spec. The richness of these descriptions means that this is effectively a "hard", non-forgeable claim to being a worthwhile, non-spam source (quite unlike the old META tags that got abused to death pre-Google). Of course spam sites could simply lie in their schema.org tags, but the lies are easy to spot (with combined machine- and human-review) and then they just get banned. It makes it a lot harder (and hopefully infeasible) to SEO-spam by just copying random content.
- frenchyatwork 5y ago
  A lot of what counts as spam these days isn't something like "I search for bicycle reviews and get penis enlargement pills", it's more like "I search for bicycle reviews and get some blog who searched Amazon for the 5 most popular bikes and posted links to them with a little blurb and called it a 'Review'". These sort of things are easy to spot, but only if you actually have a basic amount of familiarly with the topic. It's hard to spot with "AI" or super-cheap labor.
RealityVoid 5y ago
The problem, IMO, might be the monoculture we have around search. Because Google is soo big, it's enough for spammers to target it and they have the vast majority of the search visibility. If we had better, more diverse competition, that might manifest as a tradeoff, presumably, they would have competing and diverse criteria so you would probably not be the top result on _all_ dominant search engines. SEO spam needs upkeep and attention to latest algos, else it decays. Competing algos would yeld better results for everyone. Maybe Google is just ripe for a shakeup.
- jefftk 5y ago
  Doesn't your model predict that Bing would have substantially less SEO-gamed results? (Disclosure: I work at Google, but not on search)
  - RealityVoid 5y ago
    Well... Yes, it should. But, no, it seems it does not. I thought about this when typing it but did it anyway, maybe because I thought there is still something worthwhile there. I still think the model could work if the algorithm is sufficiently different than Google's. Ideally, people would go "I did not find anything I cared about on Google, I know, I'll use Bing!" - but nobody does this, because the results are consistently worse. Don't get me wrong, I like G as a company, I think they do worthwhile things! But they have left things slip and need competition into this field, I mean real competition, then maybe they would actually address issues. Maybe the issue is also on the incentive level as well. I mean more searches means more eyeballs and more money for Google. If someone searches one thing and they are done that is less interaction! I hope they don't work like this, but it's possible. And another possible problem is the opposite. Maybe Google is optimizing search for what it thinks people want, but it uses the wrong metric. Or it gives people what they want but not what they need.
ocdtrekkie 5y ago
I think the problem is just that the solution isn't in Google's wheelhouse: There is no algorithmic ranking system that can't be gamed. Human moderation and curation is the only way to provide true quality, and Google is allergic to solutions that don't automate and scale. I think a really good search engine would still algorithmically search it's index, but the content library should be human-curated with a goal of ingesting content via author, not via platform. Once a given author was human-approved as a quality source of information, content they produce could be automatically ingested going forwards, and conditionally re-reviewed by a human if there were reports the quality had decreased.
- specialp 5y ago
  This was Yahoo in late 90s early 2000s. They had a human curated directory search where one could look up something like "kayaking" and find a bunch of sites on kayaking. Then if you wanted to search on keyword it was outsourced to AltaVista and later Google. Altavista results were terrible and were almost nothing more than a keyword search (IE the word you were searching appeared on this page). Google got much better at the general search and this was history. I think the death of the directory search dramatically dropped the number of self-curated, informative sites from a domain expert that were common in the early internet. Now instead of making a website, many people are on content silos like Reddit/FB
  - ocdtrekkie 5y ago
    I do still think we could adapt this model on top of content silos... assuming we can index them! Consider that one could also, rather than just ingesting Reddit content, we ingest new posts from particular users who write quality posts on Reddit. Assuming a method also existed for an author to authenticate themselves with the search engine, one could also enable an author to help identify their content across multiple platforms, as well as suggest other quality authors to consider.
tablespoon 5y ago
> The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. That may be true, but I think one of the good points made on the OP is that it might actually be cultural constraints that keep them from solving the problem: https://twitter.com/paulg/status/1477761335412809729 https://twitter.com/paulg/status/1477761335412809729: > You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant). Google has some very smart and knowledgeable people, but the things they do have to fit into certain boxes, which means there are some problems they just can't fix, e.g. * Everything has to be automated at scale, which leads to consistent poor user experience (unappealable account closures initiated by inscrutable algorithms, SEO spam). * You get promoted by building new products, not maintaining existing ones, which leads to self-defeating churn outside of core areas (e.g. abandoning Google Talk and squandering their position in the messenger market). * etc.
tibbar 5y ago
But why is Google even dealing with spam? What if they (or someone else) curated top websites for a given category? For instance, when I search for a programming-related term, I already know that I want to see the answer on either Stack Overflow or one of a few reference documentation sites. It is possible that some other site could have the answer instead, but in practice the random sites that often show up at the top of the results are usually SEO spam. A search engine that figured out or let you select the semantic space you are in and then promoted known websites - maybe ones you curate yourself! - would be a big improvement. Of course you can always hardcode the site you want in the Google search results but this is hacky and not very expressive.
dougmwne 5y ago
I'm sure they know all about it, but are prevented from doing anything by the business model. Pinterest has been spamming up my search results for years. Maybe other people find it helpful, but I do not. It's obvious I am never going to get value from Pinterest. Let me click a button to add it to my block list. One single click would have given me years of massively improved results. The fact that this feature does not exist shows that there is something deep within Google's core that is preventing them from addressing SEO spam, just like there is something deep within Airbnb that makes it difficult to filter out Airbnbs with problem reviews. Google has been coasting for a good long time and now major players are realizing they are wide open for disruption.
colordrops 5y ago
You talk about this "constant assault from spammers" like it's not Google's fault and it's an intractable problem. That is not a correct characterization. There are plenty of low hanging fruit that could easily be detected and deranked, for instance scraped stack overflow spam. But google chooses not to deprioritize these results. The reason they don't is that they make money on ad clicks, which many responses have already elaborated on.
prepend 5y ago
I think the issue is that these crappy results are kind of good for revenue. It’s not just organic results impacted but all the affiliate ads. Google is smart so I assume they crunched the numbers and figured out they make more money from people filtering through crappy results that include viewing and clicking ads than by surfacing good content. I think Google is optimizing for ad revenue, not for good search.

by birken 5y ago

stoicjumbotron 5y ago

Highly OT, but if a technical person (not at a managerial level) involved in tackling spam at Google were to leave the team, are they allowed to work on the similar problem space at a different company?

goatherders 5y ago

This x100000. There is no scenario - none - where thousands of engineers at Google working on search wake up in the morning and say "we sure have made it good enough wr2 SPAM. I think I'll have another Danish."

compiler-guy 5y ago
When the cafes were open, you can bet they said, "I'll have another Danish, and then get back to work on this problem that never seems to go away."
- techdragon 5y ago
  I sure wish I had problems that were totally unsolvable, they are so easy to measure progress on. /sarcasm I think it’s more likely that because they are just building hundreds of tiny tweak experiments and it’s someone else who desides what to build and if it even worked. Search quality is such a meta-problem that it goes beyond any real hope of simply working on it in anything beyond piecemeal trial and error fashion on their dataset.
  - siva7 5y ago
    What is a danish?
    - compiler-guy 5y ago
      A breakfast pastry something like a donut. https://en.wikipedia.org/wiki/Danish_pastry https://en.wikipedia.org/wiki/Danish_pastry
ericbarrett 5y ago
I agree with this and the grandparent comment wholeheartedly. That said, there's a kind of institutional blindness that can build up in companies—especially ones that dominate their sector. It may have roots in intransigent upper management, ossified and inflexible process, wide-scale burnout, a culture of passing the buck, or any number of other pathologies. I don't claim that Google has any of these and certainly have no insight into their search group. But I've personally been at powerful companies with best-of-the-best talent that were blind to the decay in their own living room, so I would caution against immediate dismissal of PG's take.
UncleMeat 5y ago
Especially since "made a change that improved search result relevance by X%" is an extremely compelling story for promotions. If indeed there is a launch-driven culture for promos at Google then there'd be extra incentive for new mechanisms to reduce low quality search results.

judge2020 5y ago

Also, i'd be very surprised if they didn't have tens of thousands of workers aiding in spam review already. The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it, but many people would still consider that spam since it adds such an obstacle to getting the information you want. Same for cnet articles like "Best smart home devices to buy in 2022" - it's a reputable brand with a list of smart home devices, but it's hardly a review and exists to funnel people to their Amazon affiliate link.

awillen 5y ago
AFAIK the 2000 word essays in recipes are Google's fault - it prioritizes pages with a lot of content, so you have to add that junk to the top in order to rank highly. While I'm sure there's more going on behind the scenes than I'm aware of, it does seem like the rules could be altered on a category-specific basis where a lot of text isn't necessarily a positive.
- behnamoh 5y ago
  Years ago I wanted to pursue micro blogging, but this "feature" of Google search stopped me from doing it. What's the point of writing succinct, to-the-point mini articles about problems and solutions if nobody finds them on Google?
  - skilled 5y ago
    This is largely because micro-blogging means less content, and less content means you could write five 300-word blog posts instead of one 1,500-word post. I've done blogging for the last 10+ years, and many of those I spent as a freelancer working with startups/brands/editorials. Everyone is after "word count" and I absolutely hate it. Whenever I work on articles for my own blog, I just don't consider word-count at all. I think if your content is great and informative, then readership will be natural.

phsource 5y ago

This 100%. In travel, we see Google constantly tweaking its algorithms, and compared to Bing, Google surfaces a ton more small, well-written travel blogs [1] Not only that, Paul and Michael have seen plenty of startups, and at least in recent memory, the number of vertical search and consumer startups that Y Combinator has funded hasn't been that high As a consumer startup, I know this issue firsthand. Paul and Michael assume that if you build a better product, they will come! That's simply not true these days. Instead, you need to: - Build a better product - Option 1: Figure out a channel with enough growth on an existing platform. This likely means you're doing SEO for your new search engine - Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads As the founder of Wanderlog (YC W19; https://wanderlog.com https://wanderlog.com), a consumer vacation planning app [1], I definitely remember the idealistic days when I thought the best consumer product on its own would win! But growth doesn't just come, and the same can be said of vertical-specific search engines. [1] Try searching "[your city] itinerary" on Google vs. Bing: it's much more likely you'll find a small blog rather than Lonely Planet or the local travel bureau as the top result

noduerme 5y ago
>> - Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads nonoonononooonono. No. Don't monetize anything for the first 10 years. That's the only way it can work. Then you can go monetize it and buy an island and not give a shit if you destroy what you created. Oh but don't worry. You'll have investors.
judge2020 5y ago
[1]: both signed in, but with the profile image removed Bing: https://i.judge.sh/ShareX/2022/01/www.bing.com_search_q%3Datlanta%2Bitinerary%26form%3DQBLH%26sp%3D-1%26pq%3Datlanta%2Bitinerary.png https://i.judge.sh/ShareX/2022/01/www.bing.com_search_q%3Dat... Google: https://i.judge.sh/ShareX/2022/01/www.google.com_search_q%3Datlanta%2Bitinerary%26oq%3Datlanta%2Biten.png https://i.judge.sh/ShareX/2022/01/www.google.com_search_q%3D... Interestingly Google didn't have a top-result ad and the google.com/travel carousel is 4th from the bottom. For the actual results, both thefearlessforeigner.com and paigemindsthegap.com seem to be actual travel blogs (the pictures didn't appear in a reverse image search, so they are probably organic), but they're clearly geared towards being a 'faq' for visiting the city and have affiliate links where appropriate. Bing went straight for discoveratlanta.com, and frommers.com is well-thought-out but not a personal travel blog.

whiplash451 5y ago

I understood PG's point differently. My understanding is that he is suggesting an angle of attack in which carefully crafted manual reviews (that do dot scale) can be used to bootstrap a product that does scale thanks to something else (e.g. collaborative filtering). All of this being on a niche domain where you can drive a wedge into the mediocre performance of Google (online shopping probably being the worse possible choice, but there are many others).

acdha 5y ago

Your point is good but I'm not sure I'd say very good given how easily the same SEO spam domains can stay at the top of search results for ages simply by scraping someone else's content. What I'd be most interested in knowing is what their success metrics are defined as — for example, how much of a problem does Google's management consider it if someone searches, finds the answer they were looking for on someone's Stack Overflow rip-off, and stops searching? I could easily believe that a significant amount of what we're seeing here is that they're focused on some kind of user frustration metric which doesn't include things like damage to other businesses.

paulgb 5y ago
Yes, I've noticed this particularly with technical results. A lot of sites seem to have scraped StackOverflow and GitHub issues, put a crappy ad-loaded interface around them, and somehow out-rank the original SO/GitHub content. It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.
- coldpie 5y ago
  You just have to look at Google's profit motive here. Their motive isn't to provide quality search results. Their motive is to show users ads, either in the search results themselves or on the destination sites via their ad network. The SEO spam sites aren't a bug, they are a feature of Google's profit algorithm. Google's search quality will never improve so long as their motivation is to show you ads. Why should it? Competition may help here, either by an outsider like the OP suggests, or via breaking Google up with anti-trust enforcement, or both (my preference). As a user, your best personal and ethical move is to install an ad-blocker, to make ad-based business models less viable, which will help promote business models that don't abuse the customer.
  - the_other 5y ago
    The core problem, I guess, is that search engines view all their results as ads. That’s why they got into the ad business in the first place.

Sebb767 5y ago

I agree. You can say a lot of bad things about Google, but they definitely have some of the smartest and highest paid engineers working on their search. Plus there are already a lot of people trying to compete with Google and so far, no one seems to provide consistently better results. The only advantage a startup might have is that they could do completely new concepts, such as specifying what area you search in, allow you to modify their classification of your query and/or moderating sites you include - which is probably necessary anyway, since you'll hardly have the budget to fully index the web. I'm not saying it's impossible, but it's not going to be easy at all. And after all of that, you still need a way to make some money.

Alex3917 5y ago

> But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse. The reality is more that some Google engineer will come up with an algorithm change that makes the result 40% better, but it will come at the expense of making that search 3ms slower so the change won't get merged. Or it will make the results worse for some niche set of queries that the business team really cares about, so again it won't get merged. There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results, so there is lots of room for a startup to compete.

zozbot234 5y ago
> There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results Google has a paid-for Search API, so they could do that if they chose to pursue it. And then they could let Google One users opt-in to the same thing via ordinary Search. I'm not sure whether Bing has anything equivalent.

deleted 5y ago

[deleted]

xnx 5y ago

A lot of people forget that one of the inputs to the Google ranking algorithm is input from human quality raters who work off of an extensive, 172 page, guide that Google publishes and updates for anyone to read: https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf https://static.googleusercontent.com/media/guidelines.raterh...

visarga 5y ago
Apparently the "human quality raters" never found the sites reported in this thread.

behnamoh 5y ago

Most comments focus on the technical side of things, whereas I'm sure there are also legal restrictions involved in this. If Google delists a website on the grounds that it's a copycat of stack overflow, or because they have low quality content according to Google's taste, there might be lawsuits filed against Google, claiming that the company is discriminating.

bell-cot 5y ago
In which countr[y|ies] does Google not have the discretion to decide that certain sites / pages / etc. "belong considerably further down" in Google's search results pages? Seems to me that sorting the search results to #1, #2, #3, etc. is pretty well baked into their basic product.

noduerme 5y ago

How hard is spam, really. If you're Google? Here's what I would do as a heuristic (uh, not evil?): We know everything about you and everywhere you've visited and everyone you've talked to in the last 60 days. We know all their phone numbers and email addresses. We even know the girl's phone number you met at the bar, who didn't give you her phone number. So if any of those people email you, we'll categorize that as "not spam". Also, if it's your boss or a coworker, "not spam". If it's a major company that's existed for more than ten years, not spam. Everyone else, spam. Done. This is hyperbolic, right? But they can solve spam in a split second, if they just admit they're watching you all the time. [edit] /s thx for reading to the end, folks.

dageshi 5y ago

I said this in a similar thread yesterday, but I think this is an unsolvable problem because much of the content either no longer exists in website form or is old. To put it simply, a new generation of the people who used to make the reliable niche websites that not just answered your questions but also helped you learn a particular topic have moved to youtube instead. Google search is hollowing out as a result with the meat going and the SEO'd fluff that kinda answers the question but ONLY the direct question being asked with none of the wider expertise that more educated people in what they were searching for. Of course google owns youtube as well.. so perhaps they just see it as an inevitable transition.

neom 5y ago
Just a note on that, youtube search is finally getting better, yesterday I noticed it was able to find key words in the middle of a lecture that had nothing in the title or comments. I always wonder about their AI transcription service, it's gotten so good, if they're storing all that audio as text, I guess their search is going to get excellent?
numpad0 5y ago
Is that...essentially a Proof-of-Work system...

rightbyte 5y ago

Really? I don't think the bureaucratic bloat at Google cares and the original authors of the search engine in its current incarnation are probably long gone. It is maintenance mode and I don't think they dare touch too much. It takes time and effort to build up a spam site's ranking but it is trivial to blacklist those who get to the top.

hardtke 5y ago

The curated search results business model doesn't work. Google gives "aggregators" and other search engines the death sentence for organic search traffic from economically meaningful queries, so you'd get no free traffic. This is one of the major antitrust complaints against Google in the EU. Since you get no organic search traffic, you need to build a brand using advertising, and once you start down that road you need to monetize the first click which compromises the quality of your site.

magicalist 5y ago
> This is one of the major antitrust complaints against Google in the EU. The complaints I've read are from exactly the kind of generated content farms people are complaining about in this thread.

throwaway6845 5y ago

> I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence Really? I can point you to Hard Problems that have been solved better at little startups than at Google - or, indeed, at any other bigco. That's why acquisitions happen. Why does Google having 1000 engineers working on a problem automatically mean they are the smartest?

numbsafari 5y ago

The problem isn't that Google doesn't employ these people or invest in their activities. It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities. If Google: - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT) - Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool). - Allowed you to create your own custom search engines, like "Programmable Search Engine". That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable. Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.

ErikVandeWater 5y ago
> Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT) Are there any search engines that do this? It's a great, simple idea.
- mad182 5y ago
  Not really that simple, I see a lot of potential for abuse - using bots and brigading to mass downvote your competitors or political opponents. Couple of positions up or down in google results for somewhat popular and valuable keywords can mean the difference in thousands of dollars per day of ad or affiliate revenue. I suspect it would get pretty wild if google launched something like this. There already are black-hat seo methods and services, but something so simple and direct would turn it up to 11.
  - numbsafari 5y ago
    > I see a lot of potential for abuse They already have the tech to fight this on YT. They, in theory, are supposed to be doing the same thing to detect inauthentic behavior on ad placement and click abuse.

cft 5y ago

The search results markedly worsened in the last 5 years. Why could they keep up with SEO spam until 5 years ago, and now they can't? Their revenue has been growing dramatically, so they could proportionally increase the allocation. It's probably because the focus of their HR/changing workforce is now elsewhere: maybe fighting "disinformation": both COVID and political. Those efforts were non-existent 5 years ago.

specialp 5y ago
I think it is also no longer in their interest. If you look at their mobile results now, there are sometimes no search results for webpages, just ads, and their automatically extracted data. So, it is in their interest now to have the search for non-advertisers to be bad. Eventually people will consider those results junk and just use the google extracted data/people who paid to go up.

8ytecoder 5y ago

I agree it’s a hard problem. I don’t agree it’s “really really good”. I regularly encounter obviously scammy websites. With Google’s js execution capabilities I’d assume they can detect that. I’m talking about the VPN install pop ups and so on. Right now there’s a whole bunch of GitHub.Io hosted sites that’s doing that. It’s not even porn. It’s home decoration stuff.

JohnJamesRambo 5y ago

I don’t doubt it is hard but I’m forced to sign into Google now pretty much, just let me rate results and ban domains again etc. You will solve the seo problem really quick and start giving me results I want.

wakiza33 5y ago

Spam, yes, but Google has also made meaningful shifts that are clearly directed from the top-down. It's much harder now (imo) to get specific results, they've overall started looping SERPs into broad answers. This is def a user-engagement strategy -- but it has cons as well. Part of the complaints in the thread were spam related, other were something deeper

zozbot234 5y ago

Legitimate sites could help a lot by adding machine-readable descriptions of their content, per the schema.org spec. The richness of these descriptions means that this is effectively a "hard", non-forgeable claim to being a worthwhile, non-spam source (quite unlike the old META tags that got abused to death pre-Google). Of course spam sites could simply lie in their schema.org tags, but the lies are easy to spot (with combined machine- and human-review) and then they just get banned. It makes it a lot harder (and hopefully infeasible) to SEO-spam by just copying random content.

frenchyatwork 5y ago
A lot of what counts as spam these days isn't something like "I search for bicycle reviews and get penis enlargement pills", it's more like "I search for bicycle reviews and get some blog who searched Amazon for the 5 most popular bikes and posted links to them with a little blurb and called it a 'Review'". These sort of things are easy to spot, but only if you actually have a basic amount of familiarly with the topic. It's hard to spot with "AI" or super-cheap labor.

RealityVoid 5y ago

The problem, IMO, might be the monoculture we have around search. Because Google is soo big, it's enough for spammers to target it and they have the vast majority of the search visibility. If we had better, more diverse competition, that might manifest as a tradeoff, presumably, they would have competing and diverse criteria so you would probably not be the top result on _all_ dominant search engines. SEO spam needs upkeep and attention to latest algos, else it decays. Competing algos would yeld better results for everyone. Maybe Google is just ripe for a shakeup.

jefftk 5y ago
Doesn't your model predict that Bing would have substantially less SEO-gamed results? (Disclosure: I work at Google, but not on search)
- RealityVoid 5y ago
  Well... Yes, it should. But, no, it seems it does not. I thought about this when typing it but did it anyway, maybe because I thought there is still something worthwhile there. I still think the model could work if the algorithm is sufficiently different than Google's. Ideally, people would go "I did not find anything I cared about on Google, I know, I'll use Bing!" - but nobody does this, because the results are consistently worse. Don't get me wrong, I like G as a company, I think they do worthwhile things! But they have left things slip and need competition into this field, I mean real competition, then maybe they would actually address issues. Maybe the issue is also on the incentive level as well. I mean more searches means more eyeballs and more money for Google. If someone searches one thing and they are done that is less interaction! I hope they don't work like this, but it's possible. And another possible problem is the opposite. Maybe Google is optimizing search for what it thinks people want, but it uses the wrong metric. Or it gives people what they want but not what they need.

ocdtrekkie 5y ago

I think the problem is just that the solution isn't in Google's wheelhouse: There is no algorithmic ranking system that can't be gamed. Human moderation and curation is the only way to provide true quality, and Google is allergic to solutions that don't automate and scale. I think a really good search engine would still algorithmically search it's index, but the content library should be human-curated with a goal of ingesting content via author, not via platform. Once a given author was human-approved as a quality source of information, content they produce could be automatically ingested going forwards, and conditionally re-reviewed by a human if there were reports the quality had decreased.

specialp 5y ago
This was Yahoo in late 90s early 2000s. They had a human curated directory search where one could look up something like "kayaking" and find a bunch of sites on kayaking. Then if you wanted to search on keyword it was outsourced to AltaVista and later Google. Altavista results were terrible and were almost nothing more than a keyword search (IE the word you were searching appeared on this page). Google got much better at the general search and this was history. I think the death of the directory search dramatically dropped the number of self-curated, informative sites from a domain expert that were common in the early internet. Now instead of making a website, many people are on content silos like Reddit/FB
- ocdtrekkie 5y ago
  I do still think we could adapt this model on top of content silos... assuming we can index them! Consider that one could also, rather than just ingesting Reddit content, we ingest new posts from particular users who write quality posts on Reddit. Assuming a method also existed for an author to authenticate themselves with the search engine, one could also enable an author to help identify their content across multiple platforms, as well as suggest other quality authors to consider.

tablespoon 5y ago

> The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. That may be true, but I think one of the good points made on the OP is that it might actually be cultural constraints that keep them from solving the problem: https://twitter.com/paulg/status/1477761335412809729 https://twitter.com/paulg/status/1477761335412809729: > You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant). Google has some very smart and knowledgeable people, but the things they do have to fit into certain boxes, which means there are some problems they just can't fix, e.g. * Everything has to be automated at scale, which leads to consistent poor user experience (unappealable account closures initiated by inscrutable algorithms, SEO spam). * You get promoted by building new products, not maintaining existing ones, which leads to self-defeating churn outside of core areas (e.g. abandoning Google Talk and squandering their position in the messenger market). * etc.

tibbar 5y ago

But why is Google even dealing with spam? What if they (or someone else) curated top websites for a given category? For instance, when I search for a programming-related term, I already know that I want to see the answer on either Stack Overflow or one of a few reference documentation sites. It is possible that some other site could have the answer instead, but in practice the random sites that often show up at the top of the results are usually SEO spam. A search engine that figured out or let you select the semantic space you are in and then promoted known websites - maybe ones you curate yourself! - would be a big improvement. Of course you can always hardcode the site you want in the Google search results but this is hacky and not very expressive.

dougmwne 5y ago

I'm sure they know all about it, but are prevented from doing anything by the business model. Pinterest has been spamming up my search results for years. Maybe other people find it helpful, but I do not. It's obvious I am never going to get value from Pinterest. Let me click a button to add it to my block list. One single click would have given me years of massively improved results. The fact that this feature does not exist shows that there is something deep within Google's core that is preventing them from addressing SEO spam, just like there is something deep within Airbnb that makes it difficult to filter out Airbnbs with problem reviews. Google has been coasting for a good long time and now major players are realizing they are wide open for disruption.

colordrops 5y ago

You talk about this "constant assault from spammers" like it's not Google's fault and it's an intractable problem. That is not a correct characterization. There are plenty of low hanging fruit that could easily be detected and deranked, for instance scraped stack overflow spam. But google chooses not to deprioritize these results. The reason they don't is that they make money on ad clicks, which many responses have already elaborated on.

prepend 5y ago

I think the issue is that these crappy results are kind of good for revenue. It’s not just organic results impacted but all the affiliate ads. Google is smart so I assume they crunched the numbers and figured out they make more money from people filtering through crappy results that include viewing and clicking ads than by surfacing good content. I think Google is optimizing for ad revenue, not for good search.