7 ms·

Scraping public web content is a really confusing situation. My company, Datastreamer, has been in business for ten years indexing public web content (news + b

Scraping public web content is a really confusing situation. My company, Datastreamer, has been in business for ten years indexing public web content (news + blogs). We focus primarily on "live" content. Content that publishes often. The main challenge we've always had is that just because the content CAN be indexed doesn't necessarily mean you MAY index it. A recent situation was around Craigslist vs 3taps: https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc. Basically the issue doesn't evolve around WHO has copyright but who has copyaccess to the content. So if you create an account on Acme.com... You still own the copyright to the content you post but Acme controls access. Not only that but the ToS that you sign gives them rights to your content including bulk sales. This means that Acme can monetize the content that YOU create while actively preventing people from indexing it even that may be your intention. This means that in 2018 a company like Google COULD NOT get started because websites would just not allow them to access your content. I believe that when most people post public content on the Internet they intend it to be public including accessed by other search engines crawlers, etc. Now we're in a horrible situation where just a few companies essentially own the Internet. This is why Google can't index Facebook content or Twitter content even though it's public - they can't access it.

meritt 8y ago
> Google can't index Facebook content or Twitter content even though it's public They index plenty of their content [1][2]. What they don't index is content not explicitly marked as "Public". e.g. Facebook posts with visibility settings or protected tweets. FB, Twitter, and now LinkedIn have plenty of content that's not publicly accessible: that is, content which doesn't require a logged-in account that explicitly agreed to TOS/EULA, but they have tons of publicly-accessible content, too. The latter is fair-game. That said, the Craigslist lawsuit is still bewildering to me. That content is explicitly public, does not require a login, and the only agreements are the automatic unforceable browsewrap ones. The LinkedIn v. HiQ case is very similar to the Craigslist v. 3Taps, however the decisions are in opposite directions. [1] https://www.google.com/search?q=site%3Afacebook.com https://www.google.com/search?q=site%3Afacebook.com [2] https://www.google.com/search?q=site%3Atwitter.com https://www.google.com/search?q=site%3Atwitter.com
- fjsolwmv 8y ago
  What's bewildering? Published does not mean uncopyrightable. A radio station broadcasts a song; that doesn't remove the song's copyright protection. Receiving a copy is not making a copy. 3Taps settled put of court; there was no decision.
  - 7j 8y ago
    I'm not sure the radiostation analogy holds. Radiostations can't claim copyright to the song.
mc32 8y ago
There are a few forums where I don’t expect nor would prefer indexing. They may be niche forums, etc. Forums like reddit, of course I expect to be indexed. So, basically, it depends.
- jimktrains2 8y ago
  Isn't this what robots.txt was created to convey?
  - mc32 8y ago
    Yes. But not all scrapers and indexers respect that? So it’s still a problem.
    - baroffoos 8y ago
      Require a login/invite then
alkonaut 8y ago
What part of the indexing isn’t allowed, is it the scraping or the publishing of the scraped material? Or something else such as a “use” of the material? Example: Does it matter whether the data is ever stored? Say I make a listing notifier that pings people when a house is listed in their desired area. I’d need to crawl real estate listings and but I’d never store or republish any content from the site.
- bigiain 8y ago
  The hardline copyright view is that to "index" content from a website, you need to make a copy of that content in your indexing server, which is an action copyright law legally prohibits without an explicit exception. (The people who argue that somehow also seem to argue that publishing the webpage legally gives "explicit permission" for your web browser to make a copy, without also giving permission for your web scraping software to do so. Not sure how they get to _that_ position logically...)
  - alkonaut 8y ago
    Yeah but to e.g. make an app that notifies people when a house is published in area X really doesn't need to index anything. I can walk through all the listings once per week and all that is in my database is the table of what user wants what notification. If the data on the site matches my notification table I send an email (or whatever) and after the weekly run is done I still have zero data copied or "indexed" from the target site.
    - jmiserez 8y ago
      Makes sense from a technical standpoint, but it's the legal people you'll need to convince.
    - jscissr 8y ago
      Actually, you still have a copy of the data, in memory.
      - 8y ago
bduerst 8y ago
Yep - I specifcially remember using Padmapper (3Taps) around the time that Craigslist yanked their content. Padmapper is just a much better interface for craigslist data. Even though craigslist pulled their data, craigslist didn't create a new interface to match or even bother to improve their experience, even though they have the user mindshare for posting this type of data. Essentially it's as you said - innovation is being killed to preserve walled gardens like Craigslist, and users are the ones being hurt here.
- kapitalx 8y ago
  > craigslist didn't create a new interface to match or even bother to improve their experience, That's not completely true. After lots of bashing online [1], craigslist started adding some features such as searches on maps[2]. Even though it's not as nice and easy to use as padmapper was, it would still be considered an improvement. [1] - https://www.inc.com/abigail-tracy/craigslist-quietly-makes-changes.html https://www.inc.com/abigail-tracy/craigslist-quietly-makes-c... [2] - https://sfbay.craigslist.org/d/apts-housing-for-rent/search/apa https://sfbay.craigslist.org/d/apts-housing-for-rent/search/...
hummingurban 8y ago
post 3taps, craigslist won't be able to stop scrapers. https://www.eff.org/deeplinks/2018/04/dc-court-accessing-public-information-not-computer-crime https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...
- marcoperaza 8y ago
  That is about the CFAA, not copyright.
  - hummingurban 8y ago
    how do you claim ownership of data you release publicly at no charge to the users? how do you enforce anyone who copies it by hand or programmatically, does the minimum work to derive some insight and then copyrighting that? You can't. Once it's on the internet, anything you release will simply be relinquishing control. It's foolish to assume that the lawyers and the justice system will help you out. This is simply not a case of somebody pirating a movie and slapping on their own copyright. CFAA was how craigslist won but that's been thrown out thanks to the EFF lawyers. Going forward, there will be little to no recourse for "web scraping public data".
DoctorOetker 8y ago
I would like to know your opinion regarding the content problem that people prefer sharing their content on the established platform (simply for widest reach) which prevents new players from emerging (since they are intimidated from scraping said content, even if the original poster intended the widest reach). Suppose there was an intermediary platform so that the user uploads his content/video once to the intermediary platform, and can then select which platforms may use the content under what conditions (i.e. perhaps for free, perhaps for a user-set minimal amount of remuneration, ...), or perhaps instead of selecting which platforms, selecting the rules to which a platform must adhere in order to distribute the content (so that new players can instantly start hosting a diverse set of content as long as they adhere to these rules). The user thus only needs to upload once. Obviously it would require youtube etc, to provide API access for the intermediary so it can post the content without being able to hijack a user's account on youtube etc... This would not be in their interest of course, so anyone attempting to create this intermediary upload service would face a never-ending cat and mouse game, somewhat reminescent of youtube-dl... Perhaps instead of silly link tax rules, it would be better for governments to force platforms above a certain size/usage to provide API access such that these intermediary upload services can flourish. They might effectively become some kind of consumer/producer protection organizations, by proposing multiple usage standards (like my video MUST NOT have ads, or perhaps the opposite for another user, my video MUST be remunerated at x cents per y views, otherwise the platform may no longer distribute my content, ...) from which their users can select... apart from "pure format" content like music, videos, ... as soon as there is more compound structured content, like a blog post with images, it would be hard to standardize the format. then there is the problem of authentication, especially if a user uploads a video through the intermediary, and 1 year later a new platform wishes to distribute the content on its network too, and after another year the user wants be active on this platform, there is already an account, and how does he get his login credentials? In theory the intermediary could have a messagebox for each user, and the platform can send the credentials to the user in this messagebox, so that if interested a producer can decide to make use of his account on a new platform... Edit: so while currently all the platforms act like publisher and printer separate the market into publishers (intermediary rights managers) and printers/distributors (big platforms like YT etc). Similar to how humans decided it was better to split up doctors and pharmacists to prevent de facto quackery of selling whatever you happen to have on hand, or cheaper deals for...

meritt 8y ago

> Google can't index Facebook content or Twitter content even though it's public They index plenty of their content [1][2]. What they don't index is content not explicitly marked as "Public". e.g. Facebook posts with visibility settings or protected tweets. FB, Twitter, and now LinkedIn have plenty of content that's not publicly accessible: that is, content which doesn't require a logged-in account that explicitly agreed to TOS/EULA, but they have tons of publicly-accessible content, too. The latter is fair-game. That said, the Craigslist lawsuit is still bewildering to me. That content is explicitly public, does not require a login, and the only agreements are the automatic unforceable browsewrap ones. The LinkedIn v. HiQ case is very similar to the Craigslist v. 3Taps, however the decisions are in opposite directions. [1] https://www.google.com/search?q=site%3Afacebook.com https://www.google.com/search?q=site%3Afacebook.com [2] https://www.google.com/search?q=site%3Atwitter.com https://www.google.com/search?q=site%3Atwitter.com

fjsolwmv 8y ago
What's bewildering? Published does not mean uncopyrightable. A radio station broadcasts a song; that doesn't remove the song's copyright protection. Receiving a copy is not making a copy. 3Taps settled put of court; there was no decision.
- 7j 8y ago
  I'm not sure the radiostation analogy holds. Radiostations can't claim copyright to the song.

mc32 8y ago

There are a few forums where I don’t expect nor would prefer indexing. They may be niche forums, etc. Forums like reddit, of course I expect to be indexed. So, basically, it depends.

jimktrains2 8y ago
Isn't this what robots.txt was created to convey?
- mc32 8y ago
  Yes. But not all scrapers and indexers respect that? So it’s still a problem.
  - baroffoos 8y ago
    Require a login/invite then

alkonaut 8y ago

What part of the indexing isn’t allowed, is it the scraping or the publishing of the scraped material? Or something else such as a “use” of the material? Example: Does it matter whether the data is ever stored? Say I make a listing notifier that pings people when a house is listed in their desired area. I’d need to crawl real estate listings and but I’d never store or republish any content from the site.

bigiain 8y ago
The hardline copyright view is that to "index" content from a website, you need to make a copy of that content in your indexing server, which is an action copyright law legally prohibits without an explicit exception. (The people who argue that somehow also seem to argue that publishing the webpage legally gives "explicit permission" for your web browser to make a copy, without also giving permission for your web scraping software to do so. Not sure how they get to _that_ position logically...)
- alkonaut 8y ago
  Yeah but to e.g. make an app that notifies people when a house is published in area X really doesn't need to index anything. I can walk through all the listings once per week and all that is in my database is the table of what user wants what notification. If the data on the site matches my notification table I send an email (or whatever) and after the weekly run is done I still have zero data copied or "indexed" from the target site.
  - jmiserez 8y ago
    Makes sense from a technical standpoint, but it's the legal people you'll need to convince.
  - jscissr 8y ago
    Actually, you still have a copy of the data, in memory.
    - 8y ago

bduerst 8y ago

Yep - I specifcially remember using Padmapper (3Taps) around the time that Craigslist yanked their content. Padmapper is just a much better interface for craigslist data. Even though craigslist pulled their data, craigslist didn't create a new interface to match or even bother to improve their experience, even though they have the user mindshare for posting this type of data. Essentially it's as you said - innovation is being killed to preserve walled gardens like Craigslist, and users are the ones being hurt here.

kapitalx 8y ago
> craigslist didn't create a new interface to match or even bother to improve their experience, That's not completely true. After lots of bashing online [1], craigslist started adding some features such as searches on maps[2]. Even though it's not as nice and easy to use as padmapper was, it would still be considered an improvement. [1] - https://www.inc.com/abigail-tracy/craigslist-quietly-makes-changes.html https://www.inc.com/abigail-tracy/craigslist-quietly-makes-c... [2] - https://sfbay.craigslist.org/d/apts-housing-for-rent/search/apa https://sfbay.craigslist.org/d/apts-housing-for-rent/search/...

hummingurban 8y ago

post 3taps, craigslist won't be able to stop scrapers. https://www.eff.org/deeplinks/2018/04/dc-court-accessing-public-information-not-computer-crime https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...

marcoperaza 8y ago
That is about the CFAA, not copyright.
- hummingurban 8y ago
  how do you claim ownership of data you release publicly at no charge to the users? how do you enforce anyone who copies it by hand or programmatically, does the minimum work to derive some insight and then copyrighting that? You can't. Once it's on the internet, anything you release will simply be relinquishing control. It's foolish to assume that the lawyers and the justice system will help you out. This is simply not a case of somebody pirating a movie and slapping on their own copyright. CFAA was how craigslist won but that's been thrown out thanks to the EFF lawyers. Going forward, there will be little to no recourse for "web scraping public data".

DoctorOetker 8y ago

I would like to know your opinion regarding the content problem that people prefer sharing their content on the established platform (simply for widest reach) which prevents new players from emerging (since they are intimidated from scraping said content, even if the original poster intended the widest reach). Suppose there was an intermediary platform so that the user uploads his content/video once to the intermediary platform, and can then select which platforms may use the content under what conditions (i.e. perhaps for free, perhaps for a user-set minimal amount of remuneration, ...), or perhaps instead of selecting which platforms, selecting the rules to which a platform must adhere in order to distribute the content (so that new players can instantly start hosting a diverse set of content as long as they adhere to these rules). The user thus only needs to upload once. Obviously it would require youtube etc, to provide API access for the intermediary so it can post the content without being able to hijack a user's account on youtube etc... This would not be in their interest of course, so anyone attempting to create this intermediary upload service would face a never-ending cat and mouse game, somewhat reminescent of youtube-dl... Perhaps instead of silly link tax rules, it would be better for governments to force platforms above a certain size/usage to provide API access such that these intermediary upload services can flourish. They might effectively become some kind of consumer/producer protection organizations, by proposing multiple usage standards (like my video MUST NOT have ads, or perhaps the opposite for another user, my video MUST be remunerated at x cents per y views, otherwise the platform may no longer distribute my content, ...) from which their users can select... apart from "pure format" content like music, videos, ... as soon as there is more compound structured content, like a blog post with images, it would be hard to standardize the format. then there is the problem of authentication, especially if a user uploads a video through the intermediary, and 1 year later a new platform wishes to distribute the content on its network too, and after another year the user wants be active on this platform, there is already an account, and how does he get his login credentials? In theory the intermediary could have a messagebox for each user, and the platform can send the credentials to the user in this messagebox, so that if interested a producer can decide to make use of his account on a new platform... Edit: so while currently all the platforms act like publisher and printer separate the market into publishers (intermediary rights managers) and printers/distributors (big platforms like YT etc). Similar to how humans decided it was better to split up doctors and pharmacists to prevent de facto quackery of selling whatever you happen to have on hand, or cheaper deals for...