10 ms·

Major Sites Are Saying No to Apple's AI Scraping

86 points by thm 2y ago

josephd79 2y ago
yet some of these companies use your personal data to train ai, which you can't opt of.
- jprete 2y ago
  Yes, I agree and this annoys me. From a privacy perspective, it's still better to have fifty siloed data stores that aren't allowed to mix than for your data to be in a single large store that can do whatever it wants.
- commandlinefan 2y ago
  And when you consider the comparison, it's even worse: the data they're refusing to allow Apple to train on is _public_ data. There were several court cases in the early 1900's where owners of big buildings tried to sue photographers for selling pictures of their buildings (or pictures that included their buildings) and it was finally agreed that since the building was in public, pictures of it were fair game. If you're putting an article on the internet for anybody to read, "anybody" seems like it would include an AI as well.
  - JohnFen 2y ago
    > If you're putting an article on the internet for anybody to read, "anybody" seems like it would include an AI as well. Do you think there's something wrong with having a ToS that specifically excludes AI?
  - BehindBlueEyes 2y ago
    Interesting! There's a difference though to me: if the building has a giant poster of a popular IP on it, afaik it doesn't give the photographer any rights to use that IP in their creations. I see enough series shot in Vancouver that avoid including things like that or keep it blurred out if it can't be out of frame... Content on websites has copyright in some form, just because it isn't behind a paywall/accountwall doesn't make it public domain.
    - commandlinefan 2y ago
      But the AI didn't reproduce it, the AI "read" it and incorporated it into its knowledge base. I can read & cite all the academic papers I want - I can even reproduce their text verbatim, as long as I properly cite them.
teqsun 2y ago
I've been wondering: the efforts to block AI scrapers many sites have implemented, are they because AI scrapers hit the sites too hard and cause hosting costs to increase, or because of a potential value in licensing of their data (i.e. "If you want this data you gotta pay me some big AI money for it").
- brigadier132 2y ago
  All of it. Site scraping for search used to actually be beneficial for content creators but now it’s detrimental to them
  - HenryBemis 2y ago
    "This is why we can't have good things". Something that was created for the benefit, has been mis/ab-used and now is being removed. I am also thinking (more and more lately) the movie The Congress (https://www.imdb.com/title/tt1821641/ https://www.imdb.com/title/tt1821641/) and how easy it will be very soon to make a company without human actors. The mega-big studios can train AI by feeding it all their movies, then feed it with the scenario (this is where we - the humans come in - the famous "prompt engineers"), and a movie will be created. Now, to the above scenario there will be a lot of corrections (can't walk in the air/water/nails)(can't open with killing a dog unless "John Wick" or "Sacred Games")(Sandra Bullock can levitate in "Gravity" but not on "Speed")(etc.) Eventually if you hire 1000 people to watch 2 iterations of the new movie per day and make corrections to each (100025 = ), the AI will learn and improve on these unrealistic outputs and eventually you can produce movies weekly. Yo Netflix people, if you haven't started doing this already, START! :) (oh and make a follow-up series on Sacred Games while at it) EDIT: that's quite a tangent, but I can imagine the day where I will be asking my Paramount-AI "hey, make an original Star Trek series based on this-and-that, I want to binge it next weekend!" (or show me the one you created for someone else)
    - deleted 2y ago
      [deleted]
  - dageshi 2y ago
luma 2y ago
The same sites whose “journalism” mostly involves rephrasing reports they picked off of the news wire. They just don’t want the competition.
- 2OEH8eoCRo0 2y ago
  Apple doesn't compete
  - spogbiper 2y ago
    ...yet
  - layer8 2y ago
    Apple News exists [0]. While the content is externally sourced at present, it certainly competes in that space. [0] https://www.apple.com/apple-news/ https://www.apple.com/apple-news/
  - seydor 2y ago
    Why do they train their models on that data then?
digitalsushi 2y ago
hollywood makes a 100 million dollar movie. a company sends a robot to watch it. 7 dollars for a single ticket. the world spends their ticket money on hanging out with the robot to hear about the movie. i can't make heads or tails of what's next. i am as fascinated as i am petrified.
- sva_ 2y ago
  It would be very speciest to say that a human is allowed to buy a ticket and watch a movie, but a robot is not.
  - raincole 2y ago
    Human is already the special case in every single aspect of legislation. If the laws are written in code (pun intended), they would look like: if typeof(subject) == Human then ...
  - falcolas 2y ago
    A robot (nor the software that runs it) has no inherent rights, as it's a non-sentient creation. That's why they can't go into a movie theater without the theater's explicit permission. Plus, the whole "companies can chose who they associate with" thing means not all humans can even go into a movie theater. And if we want to jog down this rabbit hole, I suggest starting with existing animals which we're realizing are sentient, but still prevent them from going into movie theaters. Once octopi can visit the movie theater, let's revisit the the robot argument.
    - ben_w 2y ago
      > A robot (nor the software that runs it) has no inherent rights, as it's a non-sentient creation. That's not why. We don't know what test could be performed to determine if some future (or present) AI is or isn't sentient, we just assert it and get on with our lives. No, the reason robots and software have no inherent rights is because the law says so. As a demonstration proof that this is a purely legal status that has nothing to do with underlying nature, the co-inventors of the automobile did not have many rights afforded to her husband: https://en.wikipedia.org/wiki/Bertha_Benz https://en.wikipedia.org/wiki/Bertha_Benz And conversely many such rights do exist for corporations, which are not themselves sentient.
dagmx 2y ago
A big chunk of their list have signed agreements with OpenAI, or in Meta’s case, has a competitor. I’d argue the way Wired is reporting this as a gotcha is also a conflict of interest because of their deal with an effective competitor. https://www.wired.com/story/conde-nast-openai-deal/ https://www.wired.com/story/conde-nast-openai-deal/ > “WIRED can confirm that Facebook, Instagram, Craigslist, Tumblr, The New York Times, The Financial Times, The Atlantic, Vox Media, the USA Today network, and WIRED’s parent company, Condé Nast, are among the many organizations opting to exclude their data from Apple’s AI training” Versus > “The media company joins The Atlantic, Axel Springer, Vox Media, and a host of other publishers who have partnered with OpenAI.”
- nerdjon 2y ago
  I find it quite concerning that we are going into a situation where some things can crawl some sites but not others. I have my own serious ethical concerns about AI, but from a pure searchability standpoint it is likely to cause more problems than solve. Information is going to start being silo'd. It would be like if I had to go to Google for one thing and Bing for something else. That being said, I am also quite concerned about what you mentioned. I was not happy when I found out that ArsTechnica signed a deal with OpenAI (or I assume their parent company did). Did that deal include any requirements of how they talk about OpenAI, AI in General, or their competitors? The concerns over AI are real, but they are putting a short term gain over the long term damage by somehow thinking that licensing data is going to save them. The tone of this is a bit weird, do we have similar "blocking google", "blocking anthropic", "blocking meta" articles? Instead of what is really happening is those partnerships were formed so everything not that partnership is being blocked.
  - pjc50 2y ago
    The Internet has long existed in a state of tension over where value gets created, how it is consumed, and who pays for it. This had evolved into an uneasy truce of ad-supported big sites, making the web almost (but not quite) unusable for most people except a few adblocking freeloaders who get the good experience. AI just blows all that up by providing a general-purpose plagiarism engine. It offers a substitute for any content, with a similar-looking but inauthentic version. People might accept not paywalling in order to reach human users, but nobody wants to be fodder for the plagiarism bot. In some ways this is the "Can Googlebot scrape your content and display it on the home page but without generating any ad views for you?" debate we've seen with news publishers before.
markx2 2y ago
Does wordpress.com count as a major site? They host a huge amount of sites. They are saying Yes to everyone. There is no indication anywhere that they block AI scrapers. In fact, they let a blogger 'profit'. https://wordpress.com/blog/2024/07/30/perplexity-partnership/ https://wordpress.com/blog/2024/07/30/perplexity-partnership...
goles 2y ago
https://archive.is/Zz5r1 https://archive.is/Zz5r1
aucisson_masque 2y ago
> Robots.txt allows website owners to block or permit bots on a case-by-case basis. While there’s no legal obligation for bots to adhere. I don't know how long it's going to work like that, internet is basically the farewest. If you can't paywall your data, someone is free to take it. Beside I bet most article written on these website used artificial intelligence to do so.
spacebanana7 2y ago
News and media sites are not very high value data sources for AI models. There's not that much additional value from having 100 news sites in your data or 100k because so much content is generic with a high degree of overlap. It'd be much more concerning if academic publishers, code repositories and wikipedia refused scraping.
- deleted 2y ago
  [deleted]
overstay8930 2y ago
This has gotta be the dumbest “debate” (at least in the USA). If you put something on the internet without protection (i.e. DRM), it is public. Enforcing a data cartel like apple is doing has gotta be the absolute worse outcome, only the ultra rich can afford to get the good and correct information while the poor get whatever scraps they can get. Apple drank the kool-aid and now has to deal with the fact that those bad faith arguments make no sense in reality.
- JumpCrisscross 2y ago
  > If you put something on the internet without protection (i.e. DRM), it is public This is a meaningless semantic punt to the word public. At the end of the day, there is a real debate around how ownership works in relation to AI.
- tdb7893 2y ago
  Data being viewable but being subject to some restrictions isn't some new thing, it's how it has always worked. E.g. if I can view a website it's public but copying/pasting it somewhere else is a violation of copyright. We've never lived in a world where you can just do whatever you want with someone else's content and there's on fact a whole huge portion of the law around intellectual property for this exact sort of thing.
MetaverseClub 2y ago
I now need to deliberately choose the sites I browse to avoid those that work with the stupid closeAI?
elena4151 2y ago
[flagged]

86 points by thm 2y ago

josephd79 2y ago

yet some of these companies use your personal data to train ai, which you can't opt of.

jprete 2y ago
Yes, I agree and this annoys me. From a privacy perspective, it's still better to have fifty siloed data stores that aren't allowed to mix than for your data to be in a single large store that can do whatever it wants.
commandlinefan 2y ago
And when you consider the comparison, it's even worse: the data they're refusing to allow Apple to train on is _public_ data. There were several court cases in the early 1900's where owners of big buildings tried to sue photographers for selling pictures of their buildings (or pictures that included their buildings) and it was finally agreed that since the building was in public, pictures of it were fair game. If you're putting an article on the internet for anybody to read, "anybody" seems like it would include an AI as well.
- JohnFen 2y ago
  > If you're putting an article on the internet for anybody to read, "anybody" seems like it would include an AI as well. Do you think there's something wrong with having a ToS that specifically excludes AI?
- BehindBlueEyes 2y ago
  Interesting! There's a difference though to me: if the building has a giant poster of a popular IP on it, afaik it doesn't give the photographer any rights to use that IP in their creations. I see enough series shot in Vancouver that avoid including things like that or keep it blurred out if it can't be out of frame... Content on websites has copyright in some form, just because it isn't behind a paywall/accountwall doesn't make it public domain.
  - commandlinefan 2y ago
    But the AI didn't reproduce it, the AI "read" it and incorporated it into its knowledge base. I can read & cite all the academic papers I want - I can even reproduce their text verbatim, as long as I properly cite them.

teqsun 2y ago

I've been wondering: the efforts to block AI scrapers many sites have implemented, are they because AI scrapers hit the sites too hard and cause hosting costs to increase, or because of a potential value in licensing of their data (i.e. "If you want this data you gotta pay me some big AI money for it").

brigadier132 2y ago
All of it. Site scraping for search used to actually be beneficial for content creators but now it’s detrimental to them
- HenryBemis 2y ago
  "This is why we can't have good things". Something that was created for the benefit, has been mis/ab-used and now is being removed. I am also thinking (more and more lately) the movie The Congress (https://www.imdb.com/title/tt1821641/ https://www.imdb.com/title/tt1821641/) and how easy it will be very soon to make a company without human actors. The mega-big studios can train AI by feeding it all their movies, then feed it with the scenario (this is where we - the humans come in - the famous "prompt engineers"), and a movie will be created. Now, to the above scenario there will be a lot of corrections (can't walk in the air/water/nails)(can't open with killing a dog unless "John Wick" or "Sacred Games")(Sandra Bullock can levitate in "Gravity" but not on "Speed")(etc.) Eventually if you hire 1000 people to watch 2 iterations of the new movie per day and make corrections to each (100025 = ), the AI will learn and improve on these unrealistic outputs and eventually you can produce movies weekly. Yo Netflix people, if you haven't started doing this already, START! :) (oh and make a follow-up series on Sacred Games while at it) EDIT: that's quite a tangent, but I can imagine the day where I will be asking my Paramount-AI "hey, make an original Star Trek series based on this-and-that, I want to binge it next weekend!" (or show me the one you created for someone else)
  - deleted 2y ago
    [deleted]
- dageshi 2y ago

luma 2y ago

The same sites whose “journalism” mostly involves rephrasing reports they picked off of the news wire. They just don’t want the competition.

2OEH8eoCRo0 2y ago
Apple doesn't compete
- spogbiper 2y ago
  ...yet
- layer8 2y ago
  Apple News exists [0]. While the content is externally sourced at present, it certainly competes in that space. [0] https://www.apple.com/apple-news/ https://www.apple.com/apple-news/
- seydor 2y ago
  Why do they train their models on that data then?

digitalsushi 2y ago

hollywood makes a 100 million dollar movie. a company sends a robot to watch it. 7 dollars for a single ticket. the world spends their ticket money on hanging out with the robot to hear about the movie. i can't make heads or tails of what's next. i am as fascinated as i am petrified.

sva_ 2y ago
It would be very speciest to say that a human is allowed to buy a ticket and watch a movie, but a robot is not.
- raincole 2y ago
  Human is already the special case in every single aspect of legislation. If the laws are written in code (pun intended), they would look like: if typeof(subject) == Human then ...
- falcolas 2y ago
  A robot (nor the software that runs it) has no inherent rights, as it's a non-sentient creation. That's why they can't go into a movie theater without the theater's explicit permission. Plus, the whole "companies can chose who they associate with" thing means not all humans can even go into a movie theater. And if we want to jog down this rabbit hole, I suggest starting with existing animals which we're realizing are sentient, but still prevent them from going into movie theaters. Once octopi can visit the movie theater, let's revisit the the robot argument.
  - ben_w 2y ago
    > A robot (nor the software that runs it) has no inherent rights, as it's a non-sentient creation. That's not why. We don't know what test could be performed to determine if some future (or present) AI is or isn't sentient, we just assert it and get on with our lives. No, the reason robots and software have no inherent rights is because the law says so. As a demonstration proof that this is a purely legal status that has nothing to do with underlying nature, the co-inventors of the automobile did not have many rights afforded to her husband: https://en.wikipedia.org/wiki/Bertha_Benz https://en.wikipedia.org/wiki/Bertha_Benz And conversely many such rights do exist for corporations, which are not themselves sentient.

dagmx 2y ago

A big chunk of their list have signed agreements with OpenAI, or in Meta’s case, has a competitor. I’d argue the way Wired is reporting this as a gotcha is also a conflict of interest because of their deal with an effective competitor. https://www.wired.com/story/conde-nast-openai-deal/ https://www.wired.com/story/conde-nast-openai-deal/ > “WIRED can confirm that Facebook, Instagram, Craigslist, Tumblr, The New York Times, The Financial Times, The Atlantic, Vox Media, the USA Today network, and WIRED’s parent company, Condé Nast, are among the many organizations opting to exclude their data from Apple’s AI training” Versus > “The media company joins The Atlantic, Axel Springer, Vox Media, and a host of other publishers who have partnered with OpenAI.”

nerdjon 2y ago
I find it quite concerning that we are going into a situation where some things can crawl some sites but not others. I have my own serious ethical concerns about AI, but from a pure searchability standpoint it is likely to cause more problems than solve. Information is going to start being silo'd. It would be like if I had to go to Google for one thing and Bing for something else. That being said, I am also quite concerned about what you mentioned. I was not happy when I found out that ArsTechnica signed a deal with OpenAI (or I assume their parent company did). Did that deal include any requirements of how they talk about OpenAI, AI in General, or their competitors? The concerns over AI are real, but they are putting a short term gain over the long term damage by somehow thinking that licensing data is going to save them. The tone of this is a bit weird, do we have similar "blocking google", "blocking anthropic", "blocking meta" articles? Instead of what is really happening is those partnerships were formed so everything not that partnership is being blocked.
- pjc50 2y ago
  The Internet has long existed in a state of tension over where value gets created, how it is consumed, and who pays for it. This had evolved into an uneasy truce of ad-supported big sites, making the web almost (but not quite) unusable for most people except a few adblocking freeloaders who get the good experience. AI just blows all that up by providing a general-purpose plagiarism engine. It offers a substitute for any content, with a similar-looking but inauthentic version. People might accept not paywalling in order to reach human users, but nobody wants to be fodder for the plagiarism bot. In some ways this is the "Can Googlebot scrape your content and display it on the home page but without generating any ad views for you?" debate we've seen with news publishers before.

markx2 2y ago

Does wordpress.com count as a major site? They host a huge amount of sites. They are saying Yes to everyone. There is no indication anywhere that they block AI scrapers. In fact, they let a blogger 'profit'. https://wordpress.com/blog/2024/07/30/perplexity-partnership/ https://wordpress.com/blog/2024/07/30/perplexity-partnership...

goles 2y ago

https://archive.is/Zz5r1 https://archive.is/Zz5r1

aucisson_masque 2y ago

> Robots.txt allows website owners to block or permit bots on a case-by-case basis. While there’s no legal obligation for bots to adhere. I don't know how long it's going to work like that, internet is basically the farewest. If you can't paywall your data, someone is free to take it. Beside I bet most article written on these website used artificial intelligence to do so.

spacebanana7 2y ago

News and media sites are not very high value data sources for AI models. There's not that much additional value from having 100 news sites in your data or 100k because so much content is generic with a high degree of overlap. It'd be much more concerning if academic publishers, code repositories and wikipedia refused scraping.

deleted 2y ago
[deleted]

overstay8930 2y ago

This has gotta be the dumbest “debate” (at least in the USA). If you put something on the internet without protection (i.e. DRM), it is public. Enforcing a data cartel like apple is doing has gotta be the absolute worse outcome, only the ultra rich can afford to get the good and correct information while the poor get whatever scraps they can get. Apple drank the kool-aid and now has to deal with the fact that those bad faith arguments make no sense in reality.

JumpCrisscross 2y ago
> If you put something on the internet without protection (i.e. DRM), it is public This is a meaningless semantic punt to the word public. At the end of the day, there is a real debate around how ownership works in relation to AI.
tdb7893 2y ago
Data being viewable but being subject to some restrictions isn't some new thing, it's how it has always worked. E.g. if I can view a website it's public but copying/pasting it somewhere else is a violation of copyright. We've never lived in a world where you can just do whatever you want with someone else's content and there's on fact a whole huge portion of the law around intellectual property for this exact sort of thing.

MetaverseClub 2y ago

I now need to deliberately choose the sites I browse to avoid those that work with the stupid closeAI?

elena4151 2y ago

[flagged]