3 ms·
Ask HN: Databases, Scraping, and Copyright Question
Hope you guys can settle an argument I am having with another HNer. I'm going to make up a hypothetical question because I don't want to discuss the real one in detail.
Let's say I have a site that is all about politics and stocks. I list the stock and the board of directors, and site members can submit their political opinions about how they think the corporate officers are acting. In addition, I write a program to go out and find the FaceBook, Twitter, etc accounts for these officers and I list them with the stocks.
Not wanting to do a lot of data entry, I find a database on an ftp site somewhere that has stocks and the corporate officers that are related to them. It is part of some huge system dedicated to do something completely different, like a brokerage system or a system for investing. Although they provide the data openly, on each dump there is a notice about how you can't use the information for commercial purposes (but you are free to use for personal purposes and/or download, copy, and give to friends) Their price is $50K. For that amount of money, you might be able to use part of the data, depending on what your actual needs are (translated: we want to know how much we can get out of you, so we're not committing to anything)
Do I have to physically type in each stock and the directors? And then update it all the time? When there are multiple sources of publicly-available information that gives me the same thing? That sounds crazy.
I was speaking with another HNer just now. He said if I downloaded the database tables I needed directly that was bad. But if I crawled a site created from the database from the static information located on each stock's page, then that was different. I say that matters of fact -- which people are officers for which company -- cannot be copyrighted. It would be different if I were pulling down big hunks of the database and reusing/re-purposing them, but I'm only asking a simple question about information that is very widely distributed.
So who is more right here? If you were trying out an idea for a website, what would you do?
- Vivtek 16y agoPhonebooks can't be copyrighted, and databases can't be copyrighted (exactly) but if their terms of service state that usage is restricted, and they're the kind of company I would expect them to be, you can expect a lawyergram as soon as they find out. Unless your pockets are deep, this will be a Bad Thing. Unfortunately, the exact same thing applies to scraping their site for that data. If you scrape somebody else's site generated from a paid use of that database, then ... maybe. Maybe. It's an awful risk. It would be better if you could figure out a way to scrape other publicly available data sources - even using that database as a guide. Don't get me wrong; there are really three questions here: the moral, the legal, and the practical. Morally, I consider scraping a site to be OK because it's really available to the public; just grabbing the database from a site that says, "don't do this" - not so much. Practically, I just went over that - it's a risk. Legally? I'm not a lawyer, but I suspect that the legality is more or less cognate with my moral feelings above - but a lawyer might be able to tell you. If I were trying out an idea, though, which was your original question, I'd download the damn database and get it running. Then I'd try to find a more plausible data source, but that is a task that could be assigned a couple of months and be somewhat lower priority.
- pedalpete 16y agoActually my understanding is that your first sentence is a fallacy. Each individual entry in a phonebook cannot be copyrighted, as that is public information. However, the collection as a whole can have a copyright. It's like not being able to copyright a music note, but being able to copyright how those pieces of information (notes) are put together into a song.
- DanielBMarkham 16y agoExcept you can't copyright a musical scale, one note sequentially after the other. But you can copyright a phonebook, one name and number sequentially after another? My conclusion is much the same as Vivtek's, although heck if I understand how information that is public very widespread can somehow become special because it is in a text file made by somebody else. I mean we're literally talking about an alphabetical list of names in a text file that contains public information available in hundreds or thousands of various places. But as my HN friend pointed out, databases can be marked, and if you lift them it's very possible to get the nasty gram.
- Vivtek 16y agoBecause it's on a Website that may impose terms of service in a defensible way. And much more saliently, because it is provided by people with more money than you. (Probably.)
- coderdude 16y agoI don't get this. I hear it all the time, but how does simply having more money mean that they will win? Is there an assumption that they will try to litigate until the end of time, thus forcing you to pay for a lawyer that whole time?
- Vivtek 16y agoYeah. Basically, that's it. They don't even have to litigate until the end of time - just until you run out of money, and unless you're made of it, that won't be all that long. Meanwhile, while you're defending yourself, you're not running your business, or being a parent, or really anything. Maybe you'll settle out of court rather than go through that - but the key is, they don't really care. I sometimes think I should go to law school in order to figure out a way to get rich breaking that system.
- KingOfB 16y agoMy understanding is that most "public" data is not copyrightable, so scraping it is fine. For instance, scientific data is public, but access to databases can be charged for. What Macy's phone number is is public information, but Yellow Pages can charge for access to it. In these cases, scraping is fine unless they have a 'Terms of service' that states otherwise. Sites often have terms of service that you wouldn't notice unless you cared to write a scraper for them. Google local for instance has a terms of service that makes it clear it's not for scraping. I have scraped a number of sites to 'repurpose' in other ways. If you're betting the farm on it, talk to a copyright lawyer. If there are lots of sites with the same data, I don't think anyone will notice or care.