4 ms·

Many moons ago, I put a ton of work into a (now defunct) content site that provided specific information in a small vertical. Each post took hours, but I enjoye

Many moons ago, I put a ton of work into a (now defunct) content site that provided specific information in a small vertical. Each post took hours, but I enjoyed writing the articles, and the revenue from the ads was enough to enjoy a little extra cash every month (we're talking just about enough to cover a car payment, for context). Google decided they liked a lot of the content and started providing it in their knowledge graph. It was initially great to be validated and the link to my content at the top of the search result was pretty cool to see. But it did tank traffic, as Google scraping my content and giving it away at the top of their search result page meant people didn't need to navigate to my site. Felt pretty terrible, if I'm being honest, but I also have always subscribed to understanding the risks for relying on other networks for my own benefit. BUT!!! As a regular user of search engines, I love getting the answer super fast without necessarily having to guess what sort of ad-trap I'll have to navigate to get the answer I'm looking for on some random content site. As a user, when I'm asking some non-critical question, it's nice to just get a snappy answer. I appreciate it. The problem of course arises when content creators stop making the content for Google to scrape and index, what then?

irrational 6y ago
Did they scrape your content and put it into their own database? That is, if you had deleted all of your content, would it still have appeared in the Google results?
- Red_Leaves_Flyy 6y ago
  I'm pretty sure Google is rather quick to delete dead listings, but their is a twilight period where pages persist in results. This is especially annoying with forums hosting dead and un-archived links.
  - joshspankit 6y ago
    I’m pretty sure Google actually deletes nothing. The original goal was to have all data digitized in 200 years. Of course, they remove things from results and from public view, but delete? I suspect no.
caoilte 6y ago
"The problem of course arises when content creators stop making the content for Google to scrape and index disappear, what then?" communist revolution?
- adav 6y ago
  Sounds like the plot of Atlas Shrugged.
shadowgovt 6y ago
Will content creators stop, or will a new payment model that isn't "pay for eyeballs" arise? The financial sources for information-collection-and-creation undertakings throughout human history have been manifold (wealthy patron and collective financing, to name two). It is entirely possible, if Google finds its data sources drying up, that Google itself will pay for data collection (they already do this with several categories of data they host).
germanjoey 6y ago
In my experience, "Google Instant Answers" are more often than not factually incorrect. It makes me queasy every time I see that... it speaks volumes that's so much more important to them that they keep you on the page that they'd rather give you wrong information than let you go.
- 1f60c 6y ago
  A classic example is the instant answer to "who invented running?"
  - monkeybutton 6y ago
    What I find entertaining is that while answers may be wrong, you can see from the question statement and the returned sample text that it is working perfectly in a mechanical sense. The parsing of the question is correct, the searching works, and so does the ranking. But at the end of the day, it is just applied NLP at a large scale, not AGI, so it doesn't know that it is returning a wrong answer. It doesn't know anything at all!
  - screaminghawk 6y ago
    For those that are curious it displays a Quora answer that states: >Thomas Running >Running was invented in 1612 by Thomas Running when he tried to walk twice at the same time. It's actually quite and interesting post if you follow the link. Obviously it's not correct in the purest form but there is justification for this answer.
    - notahacker 6y ago
      On the other hand "who discovered running?" quotes a tweet telling a variant of the same joke. I wonder if GPT-3 has learned about Mr Running yet?
      - joshspankit 6y ago
        At what point do the layers of abstraction and sarcasm become indistinguishable from truth? I don’t know, but I feel like our societal perfect storm + AI + time will get us there.
mnd999 6y ago
It’s hard to see how this isn’t anticompetitive. Using their market power in search to show their own ads but prevent anyone navigating to view any others seems like something that should end up in court.
- thephyber 6y ago
  Google has always respected robots.txt and has well documented how to restrict crawling. I don’t see it as anticompetitive so much as copyright violations. The fact that Google makes advert revenue using this strategy just contributes to damages. Anticompetitive would likely require Alphabet (and potentially other companies) from preventing users from accessing the original web page (which might be possible but more difficult to prove).
  - neolog 6y ago
    IIRC you can't copyright facts
    - cj 6y ago
      You can absolutely copyright the packaging of facts (see any textbook ever printed). Google often has entire paragraphs ripped from the underlying page in their knowledge graph. Would it be copyright infringement if Google displayed paragraphs of text from copyrighted textbook? If so, is there a difference between doing that and doing the same with, say, "facts" from a blog post?
    - thephyber 6y ago
      IANAL, but my understanding is that an individual fact can’t be copyrighted, but an organized collection absolutely can be. The example I remember is that A baseball game score can’t be copyrighted, but if you run a website that lists all of the baseball games throughout history and you gathered that data and did work to organize it (other than to scrape it from another collection), you have a claim to a copyright.
  - joshspankit 6y ago
    Now I’m curious: what would happen if someone had no idea that Google existed (insert wild reason here. I’m leaning towards lived in the woods for 40 years and self-hosts content without reading anything, like a hermited author)? Does Google still get to say “everyone knows we scrape your content” and “We have instructions about opting out”?
jart 6y ago
Knowing the way they are, you might want to consider applying for a job and telling that story during the interviews.
rchaud 6y ago
> The problem of course arises when content creators stop making the content for Google to scrape and index, what then? This has already happened. People put content on their sites only long enough to figure out what works, then turn that content into a book that can be sold. Every Cal Newport book came out of his blog posts. Others are turning content into paid newsletters, courses and Patreon-only walled content. Wrote something 15 years ago? Just set the "Date Modified" to last week and Google will think it's been updated. The 'content' you and I are seeing are just second and third-hand summarizations of popular topics written by content marketers.
- Red_Leaves_Flyy 6y ago
  >Wrote something 15 years ago? Just set the "Date Modified" to last week and Google will think it's been updated. ...which is awful! Try searching for historic information on a viral topic and the results are littered with incorrectly dated listings that may satisfy the search criteria but are from the wrong time span.
  - joshspankit 6y ago
    “Show me results from the last 24 hours” “Ok: here are results dated 4 years ago, but I gotchu. The ‘other articles like this’ changed last time I crawled it!”

irrational 6y ago

Did they scrape your content and put it into their own database? That is, if you had deleted all of your content, would it still have appeared in the Google results?

Red_Leaves_Flyy 6y ago
I'm pretty sure Google is rather quick to delete dead listings, but their is a twilight period where pages persist in results. This is especially annoying with forums hosting dead and un-archived links.
- joshspankit 6y ago
  I’m pretty sure Google actually deletes nothing. The original goal was to have all data digitized in 200 years. Of course, they remove things from results and from public view, but delete? I suspect no.

caoilte 6y ago

"The problem of course arises when content creators stop making the content for Google to scrape and index disappear, what then?" communist revolution?

adav 6y ago
Sounds like the plot of Atlas Shrugged.

shadowgovt 6y ago

Will content creators stop, or will a new payment model that isn't "pay for eyeballs" arise? The financial sources for information-collection-and-creation undertakings throughout human history have been manifold (wealthy patron and collective financing, to name two). It is entirely possible, if Google finds its data sources drying up, that Google itself will pay for data collection (they already do this with several categories of data they host).

germanjoey 6y ago

In my experience, "Google Instant Answers" are more often than not factually incorrect. It makes me queasy every time I see that... it speaks volumes that's so much more important to them that they keep you on the page that they'd rather give you wrong information than let you go.

1f60c 6y ago
A classic example is the instant answer to "who invented running?"
- monkeybutton 6y ago
  What I find entertaining is that while answers may be wrong, you can see from the question statement and the returned sample text that it is working perfectly in a mechanical sense. The parsing of the question is correct, the searching works, and so does the ranking. But at the end of the day, it is just applied NLP at a large scale, not AGI, so it doesn't know that it is returning a wrong answer. It doesn't know anything at all!
- screaminghawk 6y ago
  For those that are curious it displays a Quora answer that states: >Thomas Running >Running was invented in 1612 by Thomas Running when he tried to walk twice at the same time. It's actually quite and interesting post if you follow the link. Obviously it's not correct in the purest form but there is justification for this answer.
  - notahacker 6y ago
    On the other hand "who discovered running?" quotes a tweet telling a variant of the same joke. I wonder if GPT-3 has learned about Mr Running yet?
    - joshspankit 6y ago
      At what point do the layers of abstraction and sarcasm become indistinguishable from truth? I don’t know, but I feel like our societal perfect storm + AI + time will get us there.

mnd999 6y ago

It’s hard to see how this isn’t anticompetitive. Using their market power in search to show their own ads but prevent anyone navigating to view any others seems like something that should end up in court.

thephyber 6y ago
Google has always respected robots.txt and has well documented how to restrict crawling. I don’t see it as anticompetitive so much as copyright violations. The fact that Google makes advert revenue using this strategy just contributes to damages. Anticompetitive would likely require Alphabet (and potentially other companies) from preventing users from accessing the original web page (which might be possible but more difficult to prove).
- neolog 6y ago
  IIRC you can't copyright facts
  - cj 6y ago
    You can absolutely copyright the packaging of facts (see any textbook ever printed). Google often has entire paragraphs ripped from the underlying page in their knowledge graph. Would it be copyright infringement if Google displayed paragraphs of text from copyrighted textbook? If so, is there a difference between doing that and doing the same with, say, "facts" from a blog post?
  - thephyber 6y ago
    IANAL, but my understanding is that an individual fact can’t be copyrighted, but an organized collection absolutely can be. The example I remember is that A baseball game score can’t be copyrighted, but if you run a website that lists all of the baseball games throughout history and you gathered that data and did work to organize it (other than to scrape it from another collection), you have a claim to a copyright.
- joshspankit 6y ago
  Now I’m curious: what would happen if someone had no idea that Google existed (insert wild reason here. I’m leaning towards lived in the woods for 40 years and self-hosts content without reading anything, like a hermited author)? Does Google still get to say “everyone knows we scrape your content” and “We have instructions about opting out”?

jart 6y ago

Knowing the way they are, you might want to consider applying for a job and telling that story during the interviews.

rchaud 6y ago

> The problem of course arises when content creators stop making the content for Google to scrape and index, what then? This has already happened. People put content on their sites only long enough to figure out what works, then turn that content into a book that can be sold. Every Cal Newport book came out of his blog posts. Others are turning content into paid newsletters, courses and Patreon-only walled content. Wrote something 15 years ago? Just set the "Date Modified" to last week and Google will think it's been updated. The 'content' you and I are seeing are just second and third-hand summarizations of popular topics written by content marketers.

Red_Leaves_Flyy 6y ago
>Wrote something 15 years ago? Just set the "Date Modified" to last week and Google will think it's been updated. ...which is awful! Try searching for historic information on a viral topic and the results are littered with incorrectly dated listings that may satisfy the search criteria but are from the wrong time span.
- joshspankit 6y ago
  “Show me results from the last 24 hours” “Ok: here are results dated 4 years ago, but I gotchu. The ‘other articles like this’ changed last time I crawled it!”