5 ms·

I think that's a failure of imagination. In the early days of search engines and web crawling companies paid subject matter experts such as librarians and othe

by lumost 5y ago

I think that's a failure of imagination. In the early days of search engines and web crawling companies paid subject matter experts such as librarians and other experts to manually grade and categorize web pages. Spam pages tend to be filled with Ads and derivative content, owned by spam company domains, and do not provide unique information relative to other sources. If we as humans can recognize a spam page, why can't the machines? At the very least Google can penalize repeated spam domain owners/domains to help reduce the problem.

CWuestefeld 5y ago
If we as humans can recognize a spam page, why can't the machines? I think that many of these automated aggregators have come a long way. On the better sites, it's not immediately obvious that the page is spam, until I've read some way into it and notice patterns in the language and a lack of "meat".
- marlowe221 5y ago
  My dad can't seem to recognize spam pages 99% of the time...
- bjowen 5y ago
  I find the advertising to content ratio is a decent clue as well.
rchaud 5y ago
Can humans recognize spam pages though? Back in the day, such pages would be created on blogspot or other spammy domains, so there was at least a visual signal of some sort. Now the spam is on Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin.....the list goes on. Online misinformation would not be as destructive as it is if people could tell the difference between say a credible news site and a Macedonian troll farm-run 'news' site written in broken English.
- PaulDavisThe1st 5y ago
  > Now the spam is on Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin.....the list goes on Of that list, Medium is the only one I would ever expect/want a search engine to even consider returning results for searches on "how to buy a bicycle for under $500". If I was King, you'd need to add allow:social to any google search to get any results at all from anything remotely like a social media or messaging application. I'd include LinkedIn and Pinterest and similar sites in that exclusion. If you want that stuff, you need to ask for it.
- cratermoon 5y ago
  > Can humans recognize spam pages though? Probably not, but they can recognize valuable non-spam. It's called by many names, but "reputation" is as good a term as any. Yes, I'm saying that "Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin" are all low reputation.
at_a_remove 5y ago
Ah, the old "Step one: create a general purpose artificial intelligence."
- lumost 5y ago
  No general AI required. Simply count the number of ads in the text. For a top ten site I’d also expect a certain degree of interesting vocabulary which loops in industry jargon. I’d also expect spam sites to use similar syntax/sentence structure for all products. Lastly if I can follow the links and get references to the final products, then I can estimate how similar the product description is to the product. I’d expect most customers looking at top tens want something more than what they get from searching a retailers site. Heck, many legit top lists include dedicated YouTube videos visually inspecting the products and reviewing them - cross referencing blogs to YouTube channels and validating that the YouTube video is of some production quality (likes/views/youtube Spam detection) could strip down a ton of spam.
  - cratermoon 5y ago
    > Simply count the number of ads in the text. What is an ad?
    - lumost 5y ago
      Something served from double click or other major ad network
      - cratermoon 5y ago
        That does absolutely nothing to address the affiliate problem, nor does it catch advertorial or "native ads" as they sometimes called. Ignoring all content served by ad brokers is no more sufficient than browser ad blocking extensions. It helps, but you don't get an ad- and distraction-free result.
        FabHK 5y ago
newsclues 5y ago
I don’t think there is a financial incentive to clean up the internet. Too much money to be paid with shady practices that no one in power wants to do it.

by lumost 5y ago

CWuestefeld 5y ago

If we as humans can recognize a spam page, why can't the machines? I think that many of these automated aggregators have come a long way. On the better sites, it's not immediately obvious that the page is spam, until I've read some way into it and notice patterns in the language and a lack of "meat".

marlowe221 5y ago
My dad can't seem to recognize spam pages 99% of the time...
bjowen 5y ago
I find the advertising to content ratio is a decent clue as well.

rchaud 5y ago

Can humans recognize spam pages though? Back in the day, such pages would be created on blogspot or other spammy domains, so there was at least a visual signal of some sort. Now the spam is on Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin.....the list goes on. Online misinformation would not be as destructive as it is if people could tell the difference between say a credible news site and a Macedonian troll farm-run 'news' site written in broken English.

PaulDavisThe1st 5y ago
> Now the spam is on Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin.....the list goes on Of that list, Medium is the only one I would ever expect/want a search engine to even consider returning results for searches on "how to buy a bicycle for under $500". If I was King, you'd need to add allow:social to any google search to get any results at all from anything remotely like a social media or messaging application. I'd include LinkedIn and Pinterest and similar sites in that exclusion. If you want that stuff, you need to ask for it.
cratermoon 5y ago
> Can humans recognize spam pages though? Probably not, but they can recognize valuable non-spam. It's called by many names, but "reputation" is as good a term as any. Yes, I'm saying that "Facebook groups, Whatsapp group chats, TikTok, Twitter, Medium, Linkedin" are all low reputation.

at_a_remove 5y ago

Ah, the old "Step one: create a general purpose artificial intelligence."

lumost 5y ago
No general AI required. Simply count the number of ads in the text. For a top ten site I’d also expect a certain degree of interesting vocabulary which loops in industry jargon. I’d also expect spam sites to use similar syntax/sentence structure for all products. Lastly if I can follow the links and get references to the final products, then I can estimate how similar the product description is to the product. I’d expect most customers looking at top tens want something more than what they get from searching a retailers site. Heck, many legit top lists include dedicated YouTube videos visually inspecting the products and reviewing them - cross referencing blogs to YouTube channels and validating that the YouTube video is of some production quality (likes/views/youtube Spam detection) could strip down a ton of spam.
- cratermoon 5y ago
  > Simply count the number of ads in the text. What is an ad?
  - lumost 5y ago
    Something served from double click or other major ad network
    - cratermoon 5y ago
      That does absolutely nothing to address the affiliate problem, nor does it catch advertorial or "native ads" as they sometimes called. Ignoring all content served by ad brokers is no more sufficient than browser ad blocking extensions. It helps, but you don't get an ad- and distraction-free result.
      - FabHK 5y ago

newsclues 5y ago

I don’t think there is a financial incentive to clean up the internet. Too much money to be paid with shady practices that no one in power wants to do it.