4 ms·
I wrote and posted a statistical analysis of the OKCupid dataset, but took it down immediately once the information about how the data was obtained came to ligh
by minimaxir 10y ago
I wrote and posted a statistical analysis of the OKCupid dataset, but took it down immediately once the information about how the data was obtained came to light.
The problem here is that the data included in the release (answers to questions) can only be accessed by logged-in users, and the scraper they used utilizes a login, in which case the researchers blatantly violated the ToS and a DMCA takedown of the dataset was a valid response by OKC.
However, almost everything else in an OKC profile, including username, city, and sexual orientation, is public to logged out users by default. (see this analysis of mass-scraped OKC data: http://yakamo.org/?p=112 http://yakamo.org/?p=112). That makes things a grayer area than usual, and serves as a reminder that nothing online is private.
- us0r 10y ago> http://yakamo.org/?p=112 http://yakamo.org/?p=112 "I setup 7 servers with mpi4py and it took 48 hours to scrape 3.2million profiles. Now i was ready to get the answer and find out how many profiles where public." I think that works out to around 20 requests/sec. At a certain point it becomes borderline DoS?
- meshko 10y agoAt the point where you are negatively affecting their servers availability and doing so with a malicious intent.
- curiousgal 10y ago20 requests per second per 7 servers so it comes out to 3 requests per second. Not even close to a DoS attack.
- anilgulecha 10y agoIt's 20 rps -- not sure why you divided by 7. A DoS attack is measured by total, not per server.
- curiousgal 10y agoWell if you are going to total might as well add in the traffic from all the other users. And in that grand scheme 20 requests per second won't matter. I only divided by 7 because I thought he wasn't running all the 7 servers simultaneously, come to think of it, I was wrong.
- mistermann 10y agoI can't believe a company as large as okcupid has nothing in place to detect and stop this.
- dreamcompiler 10y agoIt's not fast enough to be a DOS attack by any stretch. But it's easy to detect that a given IP address has downloaded e.g. several hundred profiles in a few minutes, and OKC should have been throttling such obvious crawlers to around one per minute. If the behavior repeated, they should have blocked the address altogether for some period of time.
- scotty79 10y agoToo bad that OkCupid just doesn't publish data dump of public profiles or some API. People shouldn't need to resort to scraping to get the stuff that's public.
- sdoering 10y agoJust because someone does not do to your liking (for whatever reasons) you are allowed to do as you please? Is this the philosophy behind your comment? Or at least anyones liking and anyones doing things non the less (to not become too personal here).
- scotty79 10y agoLet me clarify my position. Whatever can be downloaded programmatically will be downloaded programatically. You can either put "Don't" in your "Terms and conditions" (which in most countries in the world and for most internet users translates to "Pretty please") or be honest with yourself and your users that your site, whether you want it or not, is an API and build it like a good API. Side effect of acknowledging that might be that you actually will put in place some throttling mechanisms that will prevent guy with just a handful of machines from downloading the data of 2.8 million users in reasonable time: http://yakamo.org/?p=112 http://yakamo.org/?p=112
- Animats 10y ago"a DMCA takedown of the dataset was a valid response by OKC." No, this is an abuse of the DMCA. The DMCA only applies where there is copyright. Databases of facts are not copyrightable in the US. See Feist vs. Rural Telephone.[1] Violation of the terms of service may be grounds for a civil suit, but it is not a criminal offense. That was settled a few years ago.[2] [1] https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R.... [2] https://www.eff.org/deeplinks/2010/07/court-violating-terms-service-not-crime-bypassing https://www.eff.org/deeplinks/2010/07/court-violating-terms-...
- oneloop 10y agoDoesn't profile data count as _creative works_ as opposed to facts, licensed by the users to OKC?
- Animats 10y agoJustice O'Connor in Feist: "Many compilations consist of nothing but raw data -- i.e. wholly factual information not accompanied by any original expression. On what basis may one claim a copyright upon such work? Common sense tells us that 100 uncopyrightable facts do not magically change their status when gathered together in one place. … The key to resolving the tension lies in understanding why facts are not copyrightable: The sine qua non of copyright is originality." OKCupid is a collector, not an creator.
- scotty79 10y agoI think the point was that users are creators (of at least parts of their profiles that are not factual information) and OKCupid is their publishing platform of choice.
- Animats 10y agoEven if there's originality, OKCupid isn't the creator, though. They at best have a non-exclusive license for the user's text. That doesn't give them DMCA rights.