6 ms·
Full text from the post: > If you had code on GitHub at any point it looks like it might be included in a large dataset called “The Stack” — If you want your c
by latexr 3y ago
Full text from the post:
> If you had code on GitHub at any point it looks like it might be included in a large dataset called “The Stack” — If you want your code removed from this massive “ai” training data go here:
> https://huggingface.co/spaces/bigcode/in-the-stack https://huggingface.co/spaces/bigcode/in-the-stack
> I found two of my old Github repos in there. Both were deleted last year and both were private. This is a serious breach of trust by Github and @huggingface.
> Remove all your code from Github.
> CONSENT IS NOT OPT-OUT.
- samtheprogram 3y agoThought it was bait, but I can confirm I can find private repos in the search results. What the heck?
- tailspin2019 3y agoWere those repos always private?
- simonw 3y agoWhich private repos? Any that you're willing to share (I get that sharing names of private repos goes against the whole idea of them being private!)
- rfoo 3y agoSince your repo's name is public now anyway, could you please help us and post it here? I'm really curious about what happened. Since public GitHub activities were archived [1], if you post it here we can check if it were ever public or it's truly private at all time. [1] https://www.gharchive.org/ https://www.gharchive.org/
- johncoatesdev 3y agoHow do I check this? I found a repo I'm pretty sure was always private on there that I deleted a while back. https://github.com/johncoates/JCBootstrap https://github.com/johncoates/JCBootstrap There's no archived version on archive.org at least.
- simonw 3y agohttps://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/johncoates/JCBootstrap https://archive.softwareheritage.org/browse/origin/directory... shows a "snapshot date" of "11 August 2015, 07:28:00 UTC" - any chance it was public on that date such that the crawler could have accessed it?
- sdesol 3y agoI checked my GitHub archive (https://www.gharchive.org/ https://www.gharchive.org/) indexed data and the only repo that I saw for johncoates was LanscapeVideos, which has a last event time of 2015-06-09 07:09:52+02 It is important to note that GitHub archive is not 100% accurate and there is over 319 missing hours.
- johncoatesdev 3y agoI can't find any reason why I would have made it public. I made the repo in 2014 for internal use and don't like to share projects like that. I'm pretty careful when releasing any code publicly. It's some code that other private projects depend on. I searched for any references in public code and there are none, so there should have been no reason to make it public. Interestingly my public code with thousands of stars isn't in "The Stack".
- bredren 3y agoThis shouldn't be something where we're relying on recollection. Presumably github repo privacy state has an audit trail. This would allow GH to prove / disprove claims on any given repo easily. I hope a rep steps in to do so.
- johncoatesdev 3y agoYeah I agree. Tried https://news.ycombinator.com/item?id=39771541 https://news.ycombinator.com/item?id=39771541 but there's nothing related to this repo. Does GitHub send an email out when you make something public? I don't have any emails related to this repo.
- yreg 3y agoIt seems the original ones (from the mastodon post) are - https://github.com/emenel/dust https://github.com/emenel/dust - https://github.com/emenel/portfolio https://github.com/emenel/portfolio (based on https://archive.softwareheritage.org/browse/search/?q=emenel&with_visit=true&with_content=true https://archive.softwareheritage.org/browse/search/?q=emenel...) Care to check them on gharchive? I bet they used to be public.
- latexr 3y ago> based on The data for The Stack’s dataset is sourced from the Software Heritage Archive, so checking that is redundant. We need different sources.
- mkishi 3y agoEven still, both repos had READMEs [1][2] clearly meant to be read by the public. The archival was only successful years ago, with a failed snapshot as far back as 2021 [3]. This really seems like they forgot it was ever public. Now, this is only about it being a GitHub breach. Whether unlicensed (emenel/portfolio) or GPL (emenel/dust) code should be allowed in such datasets is a different matter. [1] https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/emenel/portfolio https://archive.softwareheritage.org/browse/origin/directory... [2] https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/emenel/dust https://archive.softwareheritage.org/browse/origin/directory... [3] https://archive.softwareheritage.org/browse/origin/visits/?origin_url=https://github.com/emenel/portfolio https://archive.softwareheritage.org/browse/origin/visits/?o...
- yreg 3y agoI used SW Heritage to identify the repos that were used for training, since op did not post the repo names. The “different source” is supposed to be ghactions.
- sdesol 3y agoI checked my GitHub archive data and emenel/dust is not there, but emenel/portfolio is. Note, GitHub archive is not 100% accurate and it is missing 319 hours.
- tailspin2019 3y ago> "Both were deleted last year and both were private" I'd really like to know if they were ever public at any point because that might explain it. None of my repos that have always been private are included (apparently). That's not to say I'm not concerned by this...
- kevincrane 3y agogod the language in that opt-out link is patronizing as hell. do AI people just assume everyone is happy to be a cog in their monetization scheme?
- CatWChainsaw 3y agoThey assume that they don't need to care and their assumption has so far proven correct.
- codazoda 3y agoThis is interesting... I have 66 repositories picked up and put in The Stack. I spot-checked the first 10. All 10 are Public on GitHub. 8 of the 10 do not have a license of any type, meaning they are covered by copyright, at least in the US, unless GitHub has some terms extending the license of public projects to 3rd parties. One of mine is Private and was an extension I sold for a short time. I can't say if I ever made it public or not. https://github.com/codazoda/like_roller https://github.com/codazoda/like_roller