5 ms·

"With the rise of AI, photos of the exterior of your business are suddenly controversial" Many revenue-based websites tried to have it both ways with web crawl

"With the rise of AI, photos of the exterior of your business are suddenly controversial" Many revenue-based websites tried to have it both ways with web crawlers wherein they wanted to block automated access or repeat viewers while letting first time viewers get a free taste. Others have noted that basically Google gets a free pass for all the traffic it brings in but everyone else has to respect robots declarations. It seems like a no brainer - if your web server is configured to reply to GET requests with a 200 status and some content then they get to do pretty much whatever they want with it. Don't want to give access to everyone? Stop sending your content for free and get them to agree to some contract and authorize/license their access to your stuff.

spencerflem 3y ago
I don't think this is that cut and dry. Its perfectly reasonable to be fine with your content being linked to in an index like google search does, but not fine with the content being read by humans for free or used to train AI. I'm sure the sites would rather that google paid them in licencing too, but without changes to our laws that's not going to happen
- matthewrobertso 3y ago
  I want to research a topic, so I do a google search about it. If 100% of the search results are readable to google, so that they are indexed, but unreadable to me, due to paywalls, the google search is useless. I'm not sure what the solution is, but paywalled articles in search results are bad. If they want to be indexed they should have to offer that same indexed content to anyone browsing the index.
  - carlosjobim 3y ago
    If you're doing serious research you pay for the paywall. It's not unreadable to you, just like a coke isn't undrinkable to you because you have to pay for it.
    - hobs 3y ago
      It's a bait and switch - you just offered me free cokes and then let me know that its after I sign up for a subscription service, no thanks!
      - spencerflem 3y ago
        It's your assumption that everything behind a google link ought to be 100% free (ad supported). Other people disagree, and Google does not advertise anywhere that their list is free content only.
        lmm 3y ago
        > Google does not advertise anywhere that their list is free content only. Google does advertise that they index based on the same content that's available to anyone viewing the page, and has policies against presenting a different version of the page to their crawler versus what you're showing to visitors.
        carlosjobim 3y ago
        It's splitting hairs at this point, but anyone visiting the page can view the same content as the crawler – if they pay. Should Google also stop indexing Facebook, since Facebook puts a login wall for people to access their content? Should YouTube (ie Google) ban movie trailers, since it's just a tease for paywalled movies? The iTunes store let people listen to 30 seconds of a song before purchasing at the paywall. Was that wrong?
        lmm 3y ago
        > Should Google also stop indexing Facebook, since Facebook puts a login wall for people to access their content? Yes - I thought they already did? (I know LinkedIn edges around this by putting up a login wall only if you have a cookie showing that you'd logged in previously). > Should YouTube (ie Google) ban movie trailers, since it's just a tease for paywalled movies? The iTunes store let people listen to 30 seconds of a song before purchasing at the paywall. Was that wrong? A free sample of a paid thing is fine if everyone knows that's what it is. It's when you bait-and-switch by offering something that seems like it's free to start with that it's a problem. Like imagine showing a movie in the town square and then 10 minutes in you pause it and tell everyone they need to buy a ticket or leave.
  - spencerflem 3y ago
    i agree, it would be nice for google to have an option to avoid paywalled articles, or to specify to it what accounts you pay for and allow those only getting all journalism for free isn't sustainable though
    - LeroyRaz 3y ago
      While I believe in journalism, I'm pretty appalled at the state of modern journalism. There have been a few big fully televised court cases recently, e.g., Depp v. Heard, and I was stunned at how poor the media coverage of them was. As I was intrigued by the legal system, I watched tens of hours of raw footage of witnesses, lawyers and judges, and I was amazed at how watching the raw footage revealed how incredibly biased and superficial the journalism coverage was (on all sides). As this experience showed how untrustworthy newspapers can be, I'm really not aware of any newspaper/journalism (maybe Private Eye? Or Bellingcat?) that is worth reading, let alone paying for. I guess it might be a catch twenty two. Low quality -> low income -> low quality, but the sadness of such a dynamic does not make me want to pay for an inferior service.
  - layer8 3y ago
    I don’t like paywalls as well, but in principle payed content is justified, and if one is willing to pay for relevant content, isn’t it better that Google allows one to find it? Maybe Google Search should have a switch “show only non-paywalled results” (paraphrased) (I’m sure they could figure out which content is paywalled if they wanted to), but personally I would probably still prefer seeing which sources exist even if they are paywalled.
- postalrat 3y ago
  It is also reasonable to serve the same content to regular visitors as you do to Google? Is ok to serve a full article to Google and put visitors behind wall?
  - spencerflem 3y ago
    yes, because google is only using that article to decide what searches should link to it
    - AnthonyMouse 3y ago
      Given two pages that both contain the information the user wants, they prefer the one that isn't behind a paywall, and so they prefer a search engine that puts those results first. Giving the search engine the full content so it will rank higher is lying, because that information isn't actually there unless you pay, which most users aren't going to do. In theory you could solve this with a requirement for the site to disclose that it's doing this, so the search engine can have a box that says "exclude paywalled content", but then everybody would check the box.
      - spencerflem 3y ago
        Google wants you to be happy with your search, and doesn't care at all about journalists. If they thought people would prefer paywalled content gone they could do that anytime. I agree the option would be nice, but I think you are wrong that everyone wants paywalled articles excluded
        AnthonyMouse 3y ago
        > If they thought people would prefer paywalled content gone they could do that anytime. Which is what they do by penalizing sites for showing something different to Googlebot than the user. > I agree the option would be nice, but I think you are wrong that everyone wants paywalled articles excluded What they want is for non-paywalled links that include the relevant information to be listed first, which is de facto equivalent to excluding paywalled links (because they'll be on page 75) in the vast majority of search results.
- hobs 3y ago
  I deeply 100% disagree with you - if you want to show google something else that is a spam tactic that only ends in bad outcomes. If your content is good enough to pay for its good enough to hide from everyone. A search based on the results being something otherwise than the contents of the search is misleading, a spam tactic, and bad.
CharlesW 3y ago
> Stop sending your content for free and get them to agree to some contract and authorize/license their access to your stuff. Agreed, we need to wrap the public web with DRM immediately. We can't expect companies like OpenAI to waste time worrying about pesky legalities like copyright and content licensing.
- spencerflem 3y ago
  I don't think there's a technical solution that would work here. This is a law & enforcement problem
  - ducttapecrown 3y ago
    That's what the above comment implies if you read it sarcastically, they might be in agreement with you!
    - spencerflem 3y ago
      whoops, duh you're right
  - AnthonyMouse 3y ago
    It depends what your goal is. If you're trying to prevent anyone from getting a copy of what's on your site, you're probably screwed, because technical solutions are hard and legal solutions are only going to be violated by The Internet since it isn't all in your jurisdiction. If you're just trying to keep them from putting an excessive load on your servers, technical solutions are easy. Just give them an API or some other efficient way to receive the data.
- BeFlatXIII 3y ago
  Perhaps this can accelerate DRM hacking if multiple billions of startup money are put behind breaking it to stealthily add to the data sets. All the more fun if it's done somewhere extradition-proof from the US.
- flir 3y ago
o11c 3y ago
There's a different between "I allow people to access my data" and "I allow people to create products based on my data"

spencerflem 3y ago

I don't think this is that cut and dry. Its perfectly reasonable to be fine with your content being linked to in an index like google search does, but not fine with the content being read by humans for free or used to train AI. I'm sure the sites would rather that google paid them in licencing too, but without changes to our laws that's not going to happen

matthewrobertso 3y ago
I want to research a topic, so I do a google search about it. If 100% of the search results are readable to google, so that they are indexed, but unreadable to me, due to paywalls, the google search is useless. I'm not sure what the solution is, but paywalled articles in search results are bad. If they want to be indexed they should have to offer that same indexed content to anyone browsing the index.
- carlosjobim 3y ago
  If you're doing serious research you pay for the paywall. It's not unreadable to you, just like a coke isn't undrinkable to you because you have to pay for it.
  - hobs 3y ago
    It's a bait and switch - you just offered me free cokes and then let me know that its after I sign up for a subscription service, no thanks!
    - spencerflem 3y ago
      It's your assumption that everything behind a google link ought to be 100% free (ad supported). Other people disagree, and Google does not advertise anywhere that their list is free content only.
      - lmm 3y ago
        > Google does not advertise anywhere that their list is free content only. Google does advertise that they index based on the same content that's available to anyone viewing the page, and has policies against presenting a different version of the page to their crawler versus what you're showing to visitors.
        carlosjobim 3y ago
        It's splitting hairs at this point, but anyone visiting the page can view the same content as the crawler – if they pay. Should Google also stop indexing Facebook, since Facebook puts a login wall for people to access their content? Should YouTube (ie Google) ban movie trailers, since it's just a tease for paywalled movies? The iTunes store let people listen to 30 seconds of a song before purchasing at the paywall. Was that wrong?
        lmm 3y ago
        > Should Google also stop indexing Facebook, since Facebook puts a login wall for people to access their content? Yes - I thought they already did? (I know LinkedIn edges around this by putting up a login wall only if you have a cookie showing that you'd logged in previously). > Should YouTube (ie Google) ban movie trailers, since it's just a tease for paywalled movies? The iTunes store let people listen to 30 seconds of a song before purchasing at the paywall. Was that wrong? A free sample of a paid thing is fine if everyone knows that's what it is. It's when you bait-and-switch by offering something that seems like it's free to start with that it's a problem. Like imagine showing a movie in the town square and then 10 minutes in you pause it and tell everyone they need to buy a ticket or leave.
- spencerflem 3y ago
  i agree, it would be nice for google to have an option to avoid paywalled articles, or to specify to it what accounts you pay for and allow those only getting all journalism for free isn't sustainable though
  - LeroyRaz 3y ago
    While I believe in journalism, I'm pretty appalled at the state of modern journalism. There have been a few big fully televised court cases recently, e.g., Depp v. Heard, and I was stunned at how poor the media coverage of them was. As I was intrigued by the legal system, I watched tens of hours of raw footage of witnesses, lawyers and judges, and I was amazed at how watching the raw footage revealed how incredibly biased and superficial the journalism coverage was (on all sides). As this experience showed how untrustworthy newspapers can be, I'm really not aware of any newspaper/journalism (maybe Private Eye? Or Bellingcat?) that is worth reading, let alone paying for. I guess it might be a catch twenty two. Low quality -> low income -> low quality, but the sadness of such a dynamic does not make me want to pay for an inferior service.
- layer8 3y ago
  I don’t like paywalls as well, but in principle payed content is justified, and if one is willing to pay for relevant content, isn’t it better that Google allows one to find it? Maybe Google Search should have a switch “show only non-paywalled results” (paraphrased) (I’m sure they could figure out which content is paywalled if they wanted to), but personally I would probably still prefer seeing which sources exist even if they are paywalled.
postalrat 3y ago
It is also reasonable to serve the same content to regular visitors as you do to Google? Is ok to serve a full article to Google and put visitors behind wall?
- spencerflem 3y ago
  yes, because google is only using that article to decide what searches should link to it
  - AnthonyMouse 3y ago
    Given two pages that both contain the information the user wants, they prefer the one that isn't behind a paywall, and so they prefer a search engine that puts those results first. Giving the search engine the full content so it will rank higher is lying, because that information isn't actually there unless you pay, which most users aren't going to do. In theory you could solve this with a requirement for the site to disclose that it's doing this, so the search engine can have a box that says "exclude paywalled content", but then everybody would check the box.
    - spencerflem 3y ago
      Google wants you to be happy with your search, and doesn't care at all about journalists. If they thought people would prefer paywalled content gone they could do that anytime. I agree the option would be nice, but I think you are wrong that everyone wants paywalled articles excluded
      - AnthonyMouse 3y ago
        > If they thought people would prefer paywalled content gone they could do that anytime. Which is what they do by penalizing sites for showing something different to Googlebot than the user. > I agree the option would be nice, but I think you are wrong that everyone wants paywalled articles excluded What they want is for non-paywalled links that include the relevant information to be listed first, which is de facto equivalent to excluding paywalled links (because they'll be on page 75) in the vast majority of search results.
hobs 3y ago
I deeply 100% disagree with you - if you want to show google something else that is a spam tactic that only ends in bad outcomes. If your content is good enough to pay for its good enough to hide from everyone. A search based on the results being something otherwise than the contents of the search is misleading, a spam tactic, and bad.

CharlesW 3y ago

> Stop sending your content for free and get them to agree to some contract and authorize/license their access to your stuff. Agreed, we need to wrap the public web with DRM immediately. We can't expect companies like OpenAI to waste time worrying about pesky legalities like copyright and content licensing.

spencerflem 3y ago
I don't think there's a technical solution that would work here. This is a law & enforcement problem
- ducttapecrown 3y ago
  That's what the above comment implies if you read it sarcastically, they might be in agreement with you!
  - spencerflem 3y ago
    whoops, duh you're right
- AnthonyMouse 3y ago
  It depends what your goal is. If you're trying to prevent anyone from getting a copy of what's on your site, you're probably screwed, because technical solutions are hard and legal solutions are only going to be violated by The Internet since it isn't all in your jurisdiction. If you're just trying to keep them from putting an excessive load on your servers, technical solutions are easy. Just give them an API or some other efficient way to receive the data.
BeFlatXIII 3y ago
Perhaps this can accelerate DRM hacking if multiple billions of startup money are put behind breaking it to stealthily add to the data sets. All the more fun if it's done somewhere extradition-proof from the US.
flir 3y ago

o11c 3y ago

There's a different between "I allow people to access my data" and "I allow people to create products based on my data"