4 ms·
> You may scrape the website for the following reasons: > Researchers may scrape public, non-personal information from GitHub for academic research purposes, o
by koolba 10y ago
> You may scrape the website for the following reasons:
> Researchers may scrape public, non-personal information from GitHub for academic research purposes, only if any publications resulting from that research are open access.
> Archivists may scrape GitHub for public data for archival purposes.
> It is prohibited to scrape GitHub for spamming purposes, including for the purposes of selling GitHub users' personal information, such as to recruiters, headhunters, and job boards.
How about a level of indirection?
Alice the Archivist scrapes GitHub and compiles a compendium of names / emails. She then publishes this information in bulk for anybody who'd like to use. Alice's cousin Roger the Recruiter downloads the archive and does what he does best. Alice isn't selling anything, she's just providing the information bundled up. Roger isn't getting it from GitHub, he's getting it from the public feed (say via Bitorrent or a public data set posted somewhere).
- bryanrasmussen 10y agowhat makes someone an archivist?
- Algent 10y agoWhen you paid for winrar ? Joke aside: Working in a library I guess ?
- libeclipse 10y agoArchiving github history. Like 4chan has archives of posts, same concept I think.
- jdc0589 10y agoarchive.org, etc...
- noplay 10y agoOr software heritage
- Klathmon 10y agoI just don't think the whole "you can do X but only if you use the data in ways we want" is a good idea. It's completely unenforceable, and there are millions of loopholes like the one you just said that will let them skate around the restrictions both legally and in action. This seems like an addition because of the "GeekedIn leak"[0] of a bunch of scraped GitHub data from Nov of last year, but I just don't get what it's trying to do. Bad actors are just going to ignore it like normal (or find loopholes) and good actors weren't doing this in the first place. Personally, I wish they left out the "what you can do with the data" and focused their ToS only on the "how you are allowed to scrape it". Give solid limits on amounts and speeds and other various aspects of the scraping, make them restrictive enough to prevent a service like that from working, then provide a point of contact so that an "archivist" can be allowed to do more on a case-by-case basis. [0]https://www.troyhunt.com/8-million-github-profiles-were-leaked-from-geekedins-mongodb-heres-how-to-see-yours/ https://www.troyhunt.com/8-million-github-profiles-were-leak...
- koolba 10y agoI would imagine the idea is to have a legal recourse to go after a company built atop breaking the terms of service. An example of that would be Craigslist going after Padmapper. There's tons of scraping of Craiglist and I'm pretty sure it's a violation of the terms, but they'd reserve legal enforcement to specific circumstance.
- Klathmon 10y agoI just think that there are better ways of limiting that (however I'll freely admit that I don't really have any experience in the area, especially the legal aspect, so I could be horribly off on this). I've seen many very cool, creative, and useful tools pop up on top of GitHub and sites like it, and I just fear that changes like this will have the (perhaps unintended) side effect of shutting them down. Things like CommitPrint [https://commitprint.com/ https://commitprint.com/], or Commits.io [https://commits.io/ https://commits.io/] come to mind.
- delinka 10y ago
- johnnyfaehell 10y agoWhat if you're downloading just the git repository and scraping that. In theory you're not scraping github but the files github allows you to download.
- jdormit 10y agoThat sounds like open source to me.
- johnnyfaehell 10y agoYea it seems like they're possibly breaching AGPL on the AGPL licensed repositories.
- trjordan 10y agoIntent matters. If Alice's archive gets no traffic, and Roger's recruiting firm does 40% of their business using Alice's data, and Alice and Roger get dinner together every week, Alice seems to be doing something wrong. Even on his own, Roger is still scraping Github. The defense "It wasn't me, it was the Python program that was in the wrong!" is just as ludicrous as trying to hide behind the fact that Alice's archive is scraping the site itself. They're putting limitations on the information, not limitations on how computers are allowed to talk to each other. http://ansuz.sooke.bc.ca/entry/23 http://ansuz.sooke.bc.ca/entry/23
- mfukar 10y agoBut how can GH establish intent? Legally it's unenforceable (good luck getting an answer from my lawyer), practically it's irrelevant (I will archive this data now, maybe in the future I will be convinced that selling them is ethical), etc. Sure, in hindsight, it'll all be obvious..
- trjordan 10y agoThe only time it'll be tested is in hindsight. If you scrape the data and put it on your laptop, they don't really care. If you then sell the data, they can come after you for breach of service. They're not trying to keep you from scraping, they're trying to keep you from using the data in ways that would be distasteful to their users.
- rmc 10y agoThis is why there are "derivitative" clauses on many licences. "You may do this, so long as what your produce/derive is under the same conditions". That would stop the problem you highlight
- shakna 10y agoThe legal precedent for data scraping and TOS really needs a rethink. There is, unfortunately, precedent. So GitHub can say, "Only scrape if you do x, y, z." But... You can't access the TOS without first requesting the data. Consider it. You cannot request and view data but in a way acceptable to the TOS. You cannot view the TOS without first requesting and viewing the data.