3 ms·
This is talking about The Stack. The poster says: "I found two of my old Github repos in there. Both were deleted last year and both were private." The Stack
by simonw 3y ago
This is talking about The Stack. The poster says:
"I found two of my old Github repos in there. Both were deleted last year and both were private."
The Stack was constructed a while ago, so "deleted last year" wouldn't have an impact if it was constructed before then.
"Both were private" is the thing that needs to be unpacked here. Were these genuinely private repositories that had never been made public on GitHub?
https://huggingface.co/datasets/bigcode/the-stack-v2 https://huggingface.co/datasets/bigcode/the-stack-v2 talks about where the Stack comes from: "This dataset is derived from the Software Heritage archive, the largest public archive of software source code and accompanying development history"
You can search that here: https://archive.softwareheritage.org/browse/search/?q=simonw%2F&with_visit=true&with_content=true https://archive.softwareheritage.org/browse/search/?q=simonw... - it would be interesting to know if the OP's "private" repos are included in that collection.
- errantmind 3y agoFor info, I checked and it looks like none of my AGPL licensed repos are included in The Stack. Neither are my private repos.
- tom_ 3y agoMy GPL ones seems to be excluded. I've got a couple that are intended to be GPL, but you wouldn't know unless you go to the GitHub issues to find the issue I raised about the licence file not being in the repo. They are included.
- latexr 3y ago> Were these genuinely private repositories that had never been made public on GitHub? The poster responded to that: https://hachyderm.io/@emenel@post.lurk.org/112128613137436381 https://hachyderm.io/@emenel@post.lurk.org/11212861313743638... > I’m pretty sure the repos of mine were private. If it were only me i could be misremembering something, but i have heard from a number of people that they also found repos in this dataset that were private.
- simonw 3y agoThis is a really big claim. I think we need specifics on this - I'm inclined to think this is people not understanding the GitHub public/private repo model (or misremembering the history of their repos) over GitHub deliberately leaking private code to third parties.
- latexr 3y agoI agree that might be the case. Since I do not personally know the author, I elected to use “may” in the submission title. Hopefully we get a definitive answer from those involved. Still, the post continues to be useful for those who want to opt-out regardless.
- politelemon 3y agoI would use "claim" rather than may until it's substantiated.
- latexr 3y agoReads a bit weirder, but changed.
- treffer 3y agoI would give it a 50:50 chance. The other option is that the scraper got lucky with a tiny glitch (whatsoever). On the one hand I bet github does everything to keep stuff secure, on the other hand I can't believe there wasn't a single glitch in the last few years. And if there was a glitch then regular automated scrapers are a pretty likely siphon. Heck I should check if any of my private repository show up.....
- simonw 3y agoHere's a reply from a GitHub staff member: https://mastodon.social/@correcthorse/112128192392083842 https://mastodon.social/@correcthorse/112128192392083842 > I work for github. This post was shared in our slack, on a non work related channel. We don't think it's us. > https://huggingface.co/datasets/bigcode/the-stack-v2 https://huggingface.co/datasets/bigcode/the-stack-v2 > They say they get data from SoftwareHeritage, a website that archives repos from github. If your repos were ever open, they might have been archived there even after you deleted from github. > That's my best guess.
- beeboobaa 3y agoDepending on the license, how is this legal?
- simonw 3y agoWhich bit? The archiving on SoftwareHeritage, the gathering of that data into the Stack or the subsequent training of models?
- s4mw1se 3y agothat’s a good question… There seems to be two problems. The definition of open source depends on a license existing in a repo. Without a license it’s not legal to copy and distribute. Public vs Private repo is a platforms issue not the code maintainers. If a public repo does not have a license, it does not mean it free to copy and distribute. If a private repo has an open source license like MIT, then the crawler has a right to copy and distribute that repo. Regardless if it has authorization to access the repo or not.
- calvinmorrison 3y agoThe software heritage sent me an email they were going to steal copies of my software. I think they assumed because it was published as a git repo they had unlimited rights to steal, reproduce, sell, etc my copyrighted work. To quote the email they sent me: "The mission of Software Heritage is to collect, preserve and share all the publicly available source code: https://www.softwareheritage.org https://www.softwareheritage.org" So they're telling me their intent is to reproduce (share) my work, just because it was publicly available. Upon my reply they did offer to cancel the request, but also told me they are facilitating the storage of my code for users private theft " "Add forge now" requests are submitted by Software Heritage users who think that a forge is worth being archived. After a careful examination of your arguments, we acknowledge that your forges may not be archived, so we won't process their ingestion, and close this add forge now request. However, we cannot prevent a publicly available from being archived by anyone using our "Save code now" feature
- remotefonts 3y agoSteal your repos? So you don't have them anymore because they took them? Maybe ask nicely to have them back and with a bit of luck they'll give them back to you.
- calvinmorrison 3y agoreproducing my work without license is theft.
- rpdillon 3y agoI mean, this is a matter of law, and it isn't. It's a curious point to double down on.
- bevekspldnw 3y agoCuriously, the people most vociferous about digital copying of IP not being “theft” tend to be those people who have produced the least amount of original IP. Surprisingly, the people generating real, novel, IP like to buy groceries now and then.
- buffington 3y agoI have about fifty private github repos and hundreds more public github repos (half of them forks of other public repos). I've verified that none of my private repos are in the dataset, and all of my public repos are. I was expecting to see at least one repo that shouldn't be there depending on when the dataset was put together. In 2015 I changed a repo from public to private, which I think might suggest that the dataset was built after 2015 since my now private repo isn't in the dataset?
- godelski 3y agoI did the same and see none of my private repos. Which is quite a lot. I do see plenty of deleted repos but those were public.
- hn72774 3y agoGithub publicly streams the global change log including all public repos. I have a public repo and noticed there were multiple clones daily after every commit I made. That's when I discovered the API: https://docs.github.com/en/rest/activity/events?apiVersion=2022-11-28#list-public-events https://docs.github.com/en/rest/activity/events?apiVersion=2... If the repo was public even for a single commit, that was likely cloned and replicated elsewhere.