4 ms·

The rule of thumb seems to be: - If the website offers the data publicly (without authentication), it's free to scrape. - If the data isn't protected by copyr

• skrebbel 7y ago
  > If the data isn't protected by copyright or trademark, (e.g. public data, such as an address of a house), it's free to reuse. At the risk of stating the obvious, most data you'll find is protected by copyright. Eg this comment is written by me so according to nearly all jurisdictions in the world, I own the copyright (unless HN has a clause that I agreed to when I signed up that I sign it away, like stack overflow has). Most forums, blogs, essays, articles, news sites, recipes and song lyrics are covered by copyright. I'm pretty sure that a webshop's blarb about why product x is good is covered by copyright.
  • toast0 7y ago
    Copyright protection on recipies is much more limited than the other creative works you mentioned. If you're scraping for more factual information, in some juristdictions, such as the US, there's a good chance those aren't subject to copyright. Things like addresses, opening hours, prices, inventory (but not a description of the inventory), etc can be very useful to scrape and present in different ways.
• tzs 7y ago
  > If the data isn't protected by copyright or trademark, (e.g. public data, such as an address of a house), it's free to reuse Careful with this one. It's possible that it could be copyrighted in both the US and Europe (and also have some beyond copyright protection in Europe--more on the European situation later). In the US, a collection of data might count as a "compilation", defined in 17 USC 101: > A “compilation” is a work formed by the collection and assembling of preexisting materials or of data that are selected, coordinated, or arranged in such a way that the resulting work as a whole constitutes an original work of authorship. The term “compilation” includes collective works In the case of a compilation like a collection of house addresses, the important thing is whether the selection and arrangement of the data was sufficiently creative. The big case on this was decided by the Supreme Court in 1991. The cite is Feist Publications, Inc., v. Rural Telephone Service Co., 499 U.S. 340 (1991). Briefly, the compilation in that case was a book of telephone listing. There was no doubt that it had taken a lot of work to produce, and up to this point copyright law followed the "sweat of the brow" doctrine, which basically means that if you put a lot of time and effort into making something in a category that can be covered by category, you could get copyright. In Feist, the Court said that it is a Constitutional requirement for copyright that the work must actually be creative. It didn't take much creativity to qualify, but there had to be a spark of creativity in there. In the Feist case, they found that the telephone book in question was just an alphabetical lists of all phone users in a region, which the telephone company was required by law to make. There was no creativity in either the selection or arrangement of the data, so no compilation copyright. Based on Feist, then, a list of all house addresses in a region, sorted by address, or owner, or something like that, would probably be up for grabs. If it is a subset of the houses, then it is possible that selection was sufficiently creative to allow copyright. Same goes for a clever arrangement or presentation of the data, although if what you are using it for doesn't copy the arrangement or presentation they compilation copyright might not cover your copying. BTW, in the particular case of address data, if your application doesn't actually need specific house addresses but instead just needs to know all the valid streets in a US state, and the address ranges on those streets, look at how that state handles sales tax. Sales taxes are usually based on street address, and the states make available databases that list all streets and the tax rates for each address range within the street. If the state is one of the states that have joined the Streamlines Sales Tax arrangement, you can get their data here [1]. All the states part of the SST group (around half of the states) agreed to a common format for the data. I think most non-SST states also make the data available in a reasonable form, so the approach of using tax data to get address information works in them, too, just not as conveniently. Most of the rest of the world also recognizes some kind of copyright on data collections, similar to the compilation copyright in the US, for data collections that are selected or arranged with sufficient creativity. This is part of the TRIPS trade agreement. In the case of scraping for academic purposes, it might be OK even if it would otherwise be a copyright violation due to fair use. If it is a state owned school, it might not matter because of sovereign immunity which greatly limits the ability of citizens to sue a state government for violations of Federal laws. Some places, including most of Europe, also have a sui generis database right that creates a property right separate from copyright in databases, based on the effort to put together the database (i.e., the old "sweat of the brow" theory). I'll just point to Wikipedia for those who want more on the sui generis database right [2]. Oh, I suppose if the house addresses were for houses in Europe, then besides copyright and the sui generis database right, you might also want to consider whether or not scraping and using the data might have GDPR implications for you. [1] https://www.streamlinedsalestax.org/Shared-Pages/rate-and-boundary-files https://www.streamlinedsalestax.org/Shared-Pages/rate-and-bo... [2] https://en.wikipedia.org/wiki/Sui_generis_database_right https://en.wikipedia.org/wiki/Sui_generis_database_right