18 ms·

How to publish Git repos that cannot be republished to GitHub

526 points by edward 6y ago

Aachen 6y ago
I read the post twice but don't quite get what the author wants to share. Is it some git(hub) trick I don't understand, just a way of saying Github's/Microsoft's monopoly is bad, or something else altogether?
- Fellshard 6y ago
  They're using the fact the YouTube-DL (sp?) is being removed by GitHub automatically when rehosted in new repositories, due to DMCA complaint (and likely overreach). If someone rehosts his repository onto GitHub, the repository will be taken down, and the rehosting account likely banned.
  - akerl_ 6y ago
    Is there any evidence that GitHub is automatically removing reheated copies? Last I saw, GitHub had posted a comment asking folks not to re-host, at risk of being suspended, but there was no indication they’d started any kind of automated scanning. Beyond that, it seems like a very community-hostile move to intentionally embed commit hashes that would even potentially cause a user to get banned from another platform if they push a copy of the code there. This doesn’t stick it to GitHub, it sticks it to the users of your repo.
    - dx034 6y ago
      It'll be a question of how many reposts they see. If enough people keep reposting with throwaway accounts, they will have no other choice but to start scanning automatically.
    - viraptor 6y ago
      It depends if you want to be nice to everyone, it burn some bridges to make a point. The point being that there shouldn't be censoring of specific hashes. Honestly if a user did get banned for sharing that poisoned repo, I would still put 95% responsibility for that on GitHub. But it's not happening yet as far as I know.
      - deleted 6y ago
        [deleted]
  - dheera 6y ago
    ROT13 the entire repo? If they figure it out, then ROT14 it. If they figure that out, encrypt it and post the key somewhere that blocks the offices of Github. If they figure that out, we'll still find a way to build a better mouse.
    - nitrogen 6y ago
      If someone wants you to leave, it's usually best to leave. Good riddance to them if you find yourself in that position. Far better would be to spend your efforts finding which RIAA members pressed the RIAA to make this takedown. Follow the money, cui bono, etc.
      - dheera 6y ago
        Why is it best to leave? Protests are a good thing, sometimes, and making sure it is on GitHub in some form or another is a form of such protest. At some point the relevant parties will have to realize that this mouse of a million engineers will always outsmart them. At some point they will have to realize that advocating for FOSS and DRM at the same time is hypocritical. Rosa Parks was asked to leave, but she didn't leave.
        nitrogen 6y ago
        GitHub is not the instigator, as far as we know (MS is apparently a member of the RIAA though). I am as amused by the clever ways to get youtube-dl back onto GitHub as anyone else, but it's protesting the wrong thing in the wrong place. The problem lies with whichever member organizations inside the RIAA (or nonmember orgs friendly with the RIAA) convinced them to file the takedown. Most of us aren't positioned to find that out, but that's where the most can be gained. Maybe the EFF or similar orgs can help. Then we'll know what the true motives were, and how to address them.
        dheera 6y ago
        At the same time, technological measures can show it to them that they can't feasibly take it down and that they're fighting a losing battle. At some point the RIAA will just have to accept defeat because honestly it's not that much work to defeat their DMCA takedowns. Right now they're issuing what, 1 takedown notice to GitHub? When we make that 100,000 takedown notices, some in ROT13, some in ROT24 on an Israeli server, some in RSA encryption on a Chinese server with keys published on Russian servers, maybe they'll just be too overwhelmed with paperwork at that point.
- BoorishBears 6y ago
  It's a trick to (theoretically) get someone banned for pushing a cloned project to Github despite not actually containing banned content.
- renewiltord 6y ago
  Github complies with the DMCA, recently famously through the DMCA takedown of youtube-dl. The writer, Joey Hess (a relatively well-known OSS programmer, moreutils, for instance), is choosing to interpret this as their banning any repo that hosts the commit hash for youtube-dl (which they banned). So he is jokingly showing you a way to create a Github poison pill. The joke is that if you include this 'illegal' hash you subject the repo to being DMCA'd out (not really, but it's sort of an 'imagine if'). Which means his repo is now unpublishable on Github (for long anyway). It's kind of hacker humour in the sort of vein that you'd expect from an old-schooler like him. You sort of have to get the joke to get the joke.
  - syshum 6y ago
    is it a joke though.....
    - o-__-o 6y ago
      ..... yes
  - Wowfunhappy 6y ago
    I'd find the joke funny if it actually worked. Since as far as I can tell, Github isn't actually banning specific hashes (because they care about the content, not the hashes), and there's no sign that they're going to do so in the future, I can't say I understand the humor...
    - geofft 6y ago
      That was my initial reaction too, but thinking about it, it's potentially (probably not, but potentially) a violation of the ToS even if it's not automatically banned. So, the hacker humor point is that you have a "poison" repo which, if you push it to GitHub, might get your account deleted in the future. You have no guarantee that it won't, for sure.
    - DarkWiiPlayer 6y ago
      The way I read it, it seems to claim that github will just "insert" a known object with that hash into the new repository when it's pushed, though I'm somewhat confused as to how this would work, or how the submodule trick even adds the hash to the commit log in the first place. EDIT: I read the article again and couldn't find any of that, so I must have read that in a comment here on HN instead.
- AceJohnny2 6y ago
  It's a way to sneak in the hash of a commit (the latest youtube-DL commit) in a non-obvious way. A git repo is fundamentally composed of 3 object types: * Blob (effectively a file) * Tree (which refers to collection of Blobs and Trees, effectively a dir) * Commit (comment/ author info and various other metadata, a Tree, and a previous Commit) This is what the git tools are used to dealing with. However git submodules changed things. With submodules, a Tree object can also contain a commit object! Together with the .gitmodules metadata file, this allows you to include another git repo inside your repo. Joey leveraged that ability to add the youtube-dl latest master commit into his repo as a submodule, but deleted the .gitmodules file so that git wouldn't be verbose about it. And that's how you sneak a commit into a repo. This additionally leverages GitHub's data model (which is really the git data model), where they just have this huge DB of all the objects of all the repos on GitHub. So effectively by including this commit in a repo on GH, you're making it refer to the the entire (buried) youtube-dl repo, but you need to be sneaky to be able to see it.
  - _ikke_ 6y ago
    Note that it's only a _reference_ to a commit, not the commit object itself. So from a a github point of view, it would not even make sense to scan / trigger on this. As long as there are no commit objects with that hash, any reference is harmless.
- geofft 6y ago
  It's Git performance art/protest art, more or less. Git has a mechanism called "submodules" which allows one repo to reference another repo. It doesn't actually include any of the content from the second repo: all that's included is the URL (in the file .gitmodules) and a commit hash (in Git's view of the directory structure). When you clone a Git repository with submodules and you pass an option to git clone, or if you run a git submodule command later, Git will make a nested clone at whatever subdirectory path contains the submodule, and check out that commit. If you make commits in the submodule (and, hopefully, remember to push them), the outer repo will appear to be modified, and you can git add the changes, which will record the new commit hash, but only the hash. The purpose is to deal with vendoring or sometimes making changes to large repositories owned by someone else without copying the whole source code into your own repository. (It's also occasionally used as a mechanism to split up very large repositories that are owned by the same group/organization, e.g., if you have a test suite or large data files or something, or a shared library used by multiple projects, you might find it helpful to use submodules.) https://github.com/ceph/ceph https://github.com/ceph/ceph is an example repo that uses them - "ceph-object-corpus" and "ceph-erasure-code-corpus" at the top level are submodules, and if you click on the .gitmodules file, you'll find that there are other submodules inside src/, too. This is also an example of how GitHub handles submodules also hosted on GitHub - it will link you to the submodule at that commit. So, GitHub does at least some parsing of submodules. The author is claiming that if you include a banned commit hash as a submodule in the history of your own project, and then promptly delete it, it won't noticeably increase the size of your Git repo, nor will you actually include any of the banned content yourself, but GitHub will potentially prevent that history from being pushed. As a result, you have a Git repo that GitHub would not accept. Since the author both does not use GitHub and wants to encourage others to do the same, this would be a fairly effective way of forcing that. (The article doesn't show GitHub actually refusing to accept the history containing this submodule, though - it only shows the submodule having been created, locally. Although maybe the author's argument is simply that pushing it violates the Terms of Service even if it's not blocked by automated means.)
Fellshard 6y ago
GitHub does not have a monopoly. That said, this is a very clever use of / exposure of the current systems in place. Unintended consequences of bad laws should be highlighted like this more regularly.
- brunoqc 6y ago
  > GitHub does not have a monopoly. Yeah, well, that's just, like, your opinion, man.
- young_unixer 6y ago
  Unintended?
- dx034 6y ago
  It depends which market you look at. Git hosting in general, especially for enterprise, Github has no monopoly. Gitlab (esp self hosted) is very strong there. But for public repositories, OSS development and to a certain degree even job applications, Github does have a monopoly.
dgrin91 6y ago
The author claims the commit hash is bad links to an article... but that claim seems entirely unsubstantiated? Its not the hash that is bad, its the files that are bad. If you manage to hack git to get the commit without the files (clever trick by him), you don't hit any legal issues. Am I missing something?
- Fellshard 6y ago
  The hash is probably an excellent heuristic for spotting commits from the original YouTube-DL Git log, and so GitHub probably scans for that when searching for rehosted copies.
  - dgrin91 6y ago
    Agreed, but no one has actually stated that GH is scanning for these commits, let alone make the claim that the hashes are illegal. If you're going to start making legal claims they have to hold up to scrutiny, which this does not.
- bloaf 6y ago
  All you have to do to substantiate it is follow his procedure and see if you can successfully fork such a project to github. Let me know how it goes!
eminence32 6y ago
Quoting from the linked torrentfreak article: > If you commit or post content to this repository that violates our Terms of Service, we will delete that content and may suspend access to your account as well My reading of this is that github wants to stem the flood of garbage pull requests to their [dmca](https://github.com/github/dmca/pulls?q=is%3Apr+is%3Aclosed https://github.com/github/dmca/pulls?q=is%3Apr+is%3Aclosed) repo, which I guess is probably reasonable. I don't quite see where github is banning certain commit hashes or tree hashes on sight. I certainly respect the authors desire to not publish their repos to github (and indeed one of the nice things about git is how easy it is to self host). And I can understand not wanting others to fork the repos to github (due to objections about the company itself), but I'm not sure how I feel about adding deliberate boobytraps to repos that seems to be purely punitive.
- joeyh 6y ago
  It's currently posssible to make a PR to the github dmca repo that merges youtube-dl, and so violate their TOS. When you make that PR, you do not transfer any of youtube-dl to github. Their repo already has those objects. So all you have done is pushed a repo containing a specific hash, which is now a TOS violation, in that specific case at least. I wrote "Certain commit hashes are rapidly heading toward being illegal" because of that specificity.
  - eminence32 6y ago
    Thanks for clarifying. I do understand that these specific hashes are effectively forbidden in the dmca repo. Are you aware of any specific instances where the presence of youtube-dl hashes in other repos have been found by github and used as the grounds for disciplinary action? That would be noteworthy indeed.
  - nine_k 6y ago
    So, in addition to illegal primes, we are getting a collection of illegal random numbers? (I wonder when illegal transcendental numbers will become a thing.)
    - 6y ago
monksy 6y ago
IF you've already pulled it down your origin will be from where you pulled it. Create an empty repo else where. With your environment do a: git remote rm origin; git remote add origin <new environment> ; git push -u origin master You've filled up the new repo with what you have, history and all.
- judge2020 6y ago
  You should be including --mirror to get all refs pushed.
  - monksy 6y ago
    Nice I had no idea about that arguement.
notsureaboutpg 6y ago
I feel good for Joey because he's really been one of the few who stuck to his principles and kept from hosting on Github even though it probably would have been way easier to do so. The git-annex project (and I'm sure many of his other ones) have been going strong without Github and I appreciate him for that. He's finally being vindicated I believe (and eventually many of the warners against these free-as-in-beer but not free-as-in-software services have been proven right)
hashtagmarkup 6y ago
You just turned a completely defensible innocent act into premeditated destruction of a livelihood.
- joeyh 6y ago
  If you feel that losing access to Github would mean the destruction of your professional life, I'd encourage you to take steps to prevent that corporation from being able to do that to you.
  - hashtagmarkup 6y ago
    I don't feel that... the author feels that. It was the first line of the article.
    - ball_of_lint 6y ago
      (joeyh _is_ the author)
      - hashtagmarkup 6y ago
        hypocrisy has _many_ forms.
    - rnestler 6y ago
      I think that line was intended as a sarcastic joke.
  - deleted 6y ago
    [deleted]
timmit 6y ago
publish to gitlab :)
- mlindner 6y ago
  Gitlab is the most awful mess of javascript you could ever invent though. I use it every day for work and I wish I could avoid it.
  - dx034 6y ago
    Why's that? I've been using it for a while (on prem) and like the interface more than Github by now. I never problems in either Firefox or Chrome.
  - INTPenis 6y ago
    I don't see it, and I'm a long time noscript user. Also been using Gitlab off-prem, on-prem, for more than 6 years. When someone says Gitlab problems all I can think of is resource issues with Sidekiq when I was self-hosting it. All this drama is only bringing to light how unsafe it is for projects to host their code on a closed platform. Better be on the safe side and use something you can easily migrate away, like Gitlab.
- deleted 6y ago
  [deleted]
erling 6y ago
Totally down with the potential conflict of interest or implications of GitHub, but it’s not GitHub specifically that’s zealous about DMCA. Any server and any host of that server is going to be subject to it. The only appeal of a smaller or private server is less visibility, but legally it’s the same. DMCA isn’t going anywhere.
- deleted 6y ago
  [deleted]
- judge2020 6y ago
  Well, GitHub/Microsoft could go on a PR campaign and say "we're not going to honor this RIAA DMCA since we know yt-dl isn't violating the DMCA!" but then GitHub/Microsoft would opening themselves up to a lawsuit against (basically) the entire music industry. The amount of goodwill MSFT loses over this (hopefully isolated) incident has to be worth a few orders of magnitude less than the tens of millions of dollars that would be burned to actually fight the RIAA. Smaller hosts could get away with not honoring DMCAs since the RIAA likely isn't going to waste resources actually filing a lawsuit, but this yt-dl situation seems like the perfect setup for the RIAA to set a precedent outlawing video/music downloaders if someone were to actually fight them on it (and until then, they can continue to take down video/music downloaders until someone does counter it).
  - ponker 6y ago
    Microsoft is much wealthier than the “entire music industry” but this still might not be a hill they want to die on.
    - dx034 6y ago
      They don't want to get rid of copyright, it's just about creating a less punitive system. I don't think there's much downside for them if they really cared.
  - pxeboot 6y ago
    Microsoft is actually a member of the RIAA [1]. [1] https://www.riaa.com/about-riaa/riaa-members/ https://www.riaa.com/about-riaa/riaa-members/
vfclists 6y ago
Why not call Github Microsoft knowing that Microsoft owns Github? Microsoft has refused to challenge a highly dubious DMCA notice when they have more than enough resources to do so. Isn't Microsoft the party at fault given that their lawyers know the DMCA notice doesn't have a leg to stand on.
- kkarakk 6y ago
  probably doing the ol' delayed switcheroo like facebook is doing with whatsapp - maintain product as is for a couple of years to avoid immediate backlash while slowly enmeshing and branding all the tools in the product. then boom one day it's called azure officeserver or some similar microsoft style name and monetised like crazy with github being a deprecated sub-project left to wither and die. it is NOT in their interest to defend github in any way. any negative press attached to the product literally doesn't matter to them since it's not associated with microsoft
  - vfclists 6y ago
    It turns out Microsoft is a member of the RIAA - https://www.riaa.com/about-riaa/riaa-members/ https://www.riaa.com/about-riaa/riaa-members/. I should have known that. It figures.
m0zg 6y ago
I think we need to work towards the future where DVCS's are _distributed_ and federated. Every time you give anyone centralized control, they will abuse it. I'm not aware of any exceptions.
slimsag 6y ago
Others probably have not fully realized this yet, but with GitHub one can: 1) Publish arbitrary commits under your https://github.com/my/project https://github.com/my/project URL, e.g. a fake https://github.com/my/project/blob/<faked_commit>/README.md https://github.com/my/project/blob/<faked_commit>/README.md in your project describing how to install it that actually describes installing malware. 2) Publish those commits under your name, with your email address, and GitHub will prominently display it as if you made the commit (most do not use GPG signatures, and most do not know to look for "Verified" anyway) It seemed only a matter of time before this behavior got abused for something (anti-DMCA action is perhaps the best outcome of this situation I can imagine..)
- eslaught 6y ago
  I'm not quite sure what you're saying here. Are you claiming that you can push a commit to userA/project and then view it under the Github web interface for userB/project (assuming one repo is forked from the other)? If so it seems like that's a relatively easy fix for Github, just check if the commit is actually contained in userB's fork of the repo.
  - slimsag 6y ago
    Quick demo: go to https://github.com/slimsag/linux/tree/5895e21f3c744ed9829e3afe9691e3eb1b1932ae#linux-kernel https://github.com/slimsag/linux/tree/5895e21f3c744ed9829e3a... then try replacing "slimsag" with "torvalds" These are "known issues" I believe GitHub doesn't intend to fix: 1. With youtube-dl[0] (and probably long before?) we know you can push a commit and view it under the web UI under another user, GitHub hasn't indicated this is a vulnerability at all. 2. Many people know you can impersonate another user through Git email addresses[1] (GPG signing is supposed to solve this, but a lot of people don't use it and even when they do others don't really know they should look for "Verified" in GitHub's web UI.) Combine these two known-issues and you get a really convincing phishing attack. I really hope this doesn't become a common-place thing. Hopefully this raises some awareness that, basically, you should not trust any GitHub URL with a commit SHA in it - only trust ones with branch names - because it could be a phishing attack otherwise. [0] https://news.ycombinator.com/item?id=24882921 https://news.ycombinator.com/item?id=24882921 [1] https://bounty.github.com/ineligible.html#impersonating_a_user_through_git_email_address https://bounty.github.com/ineligible.html#impersonating_a_us...
tirz 6y ago
When I first read this article, I was really sad. "If you use one of my open-source repository, you can get in trouble, mouhahaha" It feels like a dick move to me. But then, I realize that the RIAA bans can now be used as an anti RIAA partnership against GitHub and others, which is... pretty cool. Definitively a feature that we need \o/ !
rendall 6y ago
joeyh, if you're reading this, some feedback: on Android Chrome, at least, your top navbar overlays your article. Suggest you update your CSS for mobile views
gkfksjsjdlv 6y ago
Why are you worried about being banned from github? Who cares? Who gives a shit about the host they clone from? Engineer histrionics—just kill me rather than make me deal with these fucks with respect. Absolutely zero sense of priority.
Qwertious 6y ago
This sounds like that social engineering challenge one guy did - where he said "I lose if you convince me to post <token> anywhere in my github repo", and someone won it by saying "you should put that challenge on your website" since the website code was in his github repo.
joshenders 6y ago
Why are we shooting the messenger?
lk23jlkoij 6y ago
So why not just write a tiny script that goes through all youtube-dl commits, recommit them with an added a space character to some file. So you will have a clone with different hashes, but the same content.
- pwdisswordfish0 6y ago
  Since Git's DAG is a hash tree where each commit points to its parent(s) and its own hash is derived from that, then presumably it would suffice to do this only with the first commit.
jbb67 6y ago
first if all GitHub is nobodies "résumé" and secondly you can put a git repo anywhere you like there are any number of hosting sites or easy ways to just host it yourself. I mean really. what a stupid article
- kkarakk 6y ago
  There are a LOT of companies just spamming github repo owners with job interviews nowadays my last job was through an automated email i got about my github resume hashtags. iirc all they look for is certain tags and a regular commit history after which the recruiter takes a brief look at the repo and fires off the email if the criteria is met. seems enough to call github a resume to me.
woofie11 6y ago
Has anyone read github's terms of service? Github is the one violating them, not the users. Yes, github is disabling accounts to people who post youtube-dl. No, this is not in compliance with github's ToS. github says: “Please note that re-posting the exact same content that was the subject of a takedown notice without following the proper process is a violation of GitHub’s DMCA Policy and Terms of Service." Ref: https://torrentfreak.com/github-warns-users-reposting-youtube-dl-they-could-be-banned-201102/ https://torrentfreak.com/github-warns-users-reposting-youtub... github's DMCA policy says: "One of the best features of GitHub is the ability for users to "fork" one another's repositories. What does that mean? In essence, it means that users can make a copy of a project on GitHub into their own repositories. As the license or the law allows, users can then make changes to that fork to either push back to the main project or just keep as their own variation of a project. Each of these copies is a "fork" of the original repository, which in turn may also be called the "parent" of the fork. GitHub will not automatically disable forks when disabling a parent repository. This is because forks belong to different users, may have been altered in significant ways, and may be licensed or used in a different way that is protected by the fair-use doctrine. GitHub does not conduct any independent investigation into forks. We expect copyright owners to conduct that investigation and, if they believe that the forks are also infringing, expressly include forks in their takedown notice." Ref: https://docs.github.com/en/free-pro-team@latest/github/site-policy/dmca-takedown-policy#b-what-about-forks-or-whats-a-fork https://docs.github.com/en/free-pro-team@latest/github/site-... youtube-dl doesn't fall under any of the restrictions in github's ToS either. ref: https://docs.github.com/en/free-pro-team@latest/github/site-policy/github-acceptable-use-policies https://docs.github.com/en/free-pro-team@latest/github/site-... I have very little sympathy for github here. They ought to follow the DMCA process. Per the DMCA process, I can't dispute the takedown of the main Youtube-dl repo, since I'm not a party to the process. If I post youtube-dl to github, it should stay up, pending a DMCA notice from RIAA. If such a notice is sent, I should then have the right to then dispute that.
mlang23 6y ago
Its no longer relevant to me nor him, but I can't help but think the attitude of this article kind of violates the DFSG. If the stance is to insist that free software should be allowed to be redistributed by anyone, flaming those which have in the past wanted to mirror your code on github by indicating you could get them banned by fiddling with the hashes in your repo is pretty aggressive. I totally get the frustration, but when I learnt the DFSG, this was the biggest lesson for me. Freedom means freedom to anyone. Restricting your code to, say, non-military use, isn't compliant with the DFSG. So, now that we have perceived hostile code hosters, do we really want to go there? "You're not going to push my stuff to x.y.z, or I will make you feel the pain."
- lolc 6y ago
  What do you make of an author who only provides tarballs?
booleandilemma 6y ago
illegal number This is why aliens don't visit us.
blitblitblit 6y ago
> Github. Which, as we know, is your resume. There in lies the problem. Encouraging people to put their resume on centralized platforms (Be it Facebook, LinkedIn, or Github) is putting your entire career at risk. It's better to have your own domain and self-host, that way you aren't locked in and at risk of loosing your entire professional history if you get banned or a service goes offline.

526 points by edward 6y ago

I read the post twice but don't quite get what the author wants to share. Is it some git(hub) trick I don't understand, just a way of saying Github's/Microsoft's monopoly is bad, or something else altogether?

Fellshard 6y ago
They're using the fact the YouTube-DL (sp?) is being removed by GitHub automatically when rehosted in new repositories, due to DMCA complaint (and likely overreach). If someone rehosts his repository onto GitHub, the repository will be taken down, and the rehosting account likely banned.
- akerl_ 6y ago
  Is there any evidence that GitHub is automatically removing reheated copies? Last I saw, GitHub had posted a comment asking folks not to re-host, at risk of being suspended, but there was no indication they’d started any kind of automated scanning. Beyond that, it seems like a very community-hostile move to intentionally embed commit hashes that would even potentially cause a user to get banned from another platform if they push a copy of the code there. This doesn’t stick it to GitHub, it sticks it to the users of your repo.
  - dx034 6y ago
    It'll be a question of how many reposts they see. If enough people keep reposting with throwaway accounts, they will have no other choice but to start scanning automatically.
  - viraptor 6y ago
    It depends if you want to be nice to everyone, it burn some bridges to make a point. The point being that there shouldn't be censoring of specific hashes. Honestly if a user did get banned for sharing that poisoned repo, I would still put 95% responsibility for that on GitHub. But it's not happening yet as far as I know.
    - deleted 6y ago
      [deleted]
- dheera 6y ago
  ROT13 the entire repo? If they figure it out, then ROT14 it. If they figure that out, encrypt it and post the key somewhere that blocks the offices of Github. If they figure that out, we'll still find a way to build a better mouse.
  - nitrogen 6y ago
    If someone wants you to leave, it's usually best to leave. Good riddance to them if you find yourself in that position. Far better would be to spend your efforts finding which RIAA members pressed the RIAA to make this takedown. Follow the money, cui bono, etc.
    - dheera 6y ago
      Why is it best to leave? Protests are a good thing, sometimes, and making sure it is on GitHub in some form or another is a form of such protest. At some point the relevant parties will have to realize that this mouse of a million engineers will always outsmart them. At some point they will have to realize that advocating for FOSS and DRM at the same time is hypocritical. Rosa Parks was asked to leave, but she didn't leave.
      - nitrogen 6y ago
        GitHub is not the instigator, as far as we know (MS is apparently a member of the RIAA though). I am as amused by the clever ways to get youtube-dl back onto GitHub as anyone else, but it's protesting the wrong thing in the wrong place. The problem lies with whichever member organizations inside the RIAA (or nonmember orgs friendly with the RIAA) convinced them to file the takedown. Most of us aren't positioned to find that out, but that's where the most can be gained. Maybe the EFF or similar orgs can help. Then we'll know what the true motives were, and how to address them.
        dheera 6y ago
        At the same time, technological measures can show it to them that they can't feasibly take it down and that they're fighting a losing battle. At some point the RIAA will just have to accept defeat because honestly it's not that much work to defeat their DMCA takedowns. Right now they're issuing what, 1 takedown notice to GitHub? When we make that 100,000 takedown notices, some in ROT13, some in ROT24 on an Israeli server, some in RSA encryption on a Chinese server with keys published on Russian servers, maybe they'll just be too overwhelmed with paperwork at that point.
BoorishBears 6y ago
It's a trick to (theoretically) get someone banned for pushing a cloned project to Github despite not actually containing banned content.
renewiltord 6y ago
Github complies with the DMCA, recently famously through the DMCA takedown of youtube-dl. The writer, Joey Hess (a relatively well-known OSS programmer, moreutils, for instance), is choosing to interpret this as their banning any repo that hosts the commit hash for youtube-dl (which they banned). So he is jokingly showing you a way to create a Github poison pill. The joke is that if you include this 'illegal' hash you subject the repo to being DMCA'd out (not really, but it's sort of an 'imagine if'). Which means his repo is now unpublishable on Github (for long anyway). It's kind of hacker humour in the sort of vein that you'd expect from an old-schooler like him. You sort of have to get the joke to get the joke.
- syshum 6y ago
  is it a joke though.....
  - o-__-o 6y ago
    ..... yes
- Wowfunhappy 6y ago
  I'd find the joke funny if it actually worked. Since as far as I can tell, Github isn't actually banning specific hashes (because they care about the content, not the hashes), and there's no sign that they're going to do so in the future, I can't say I understand the humor...
  - geofft 6y ago
    That was my initial reaction too, but thinking about it, it's potentially (probably not, but potentially) a violation of the ToS even if it's not automatically banned. So, the hacker humor point is that you have a "poison" repo which, if you push it to GitHub, might get your account deleted in the future. You have no guarantee that it won't, for sure.
  - DarkWiiPlayer 6y ago
    The way I read it, it seems to claim that github will just "insert" a known object with that hash into the new repository when it's pushed, though I'm somewhat confused as to how this would work, or how the submodule trick even adds the hash to the commit log in the first place. EDIT: I read the article again and couldn't find any of that, so I must have read that in a comment here on HN instead.
AceJohnny2 6y ago
It's a way to sneak in the hash of a commit (the latest youtube-DL commit) in a non-obvious way. A git repo is fundamentally composed of 3 object types: * Blob (effectively a file) * Tree (which refers to collection of Blobs and Trees, effectively a dir) * Commit (comment/ author info and various other metadata, a Tree, and a previous Commit) This is what the git tools are used to dealing with. However git submodules changed things. With submodules, a Tree object can also contain a commit object! Together with the .gitmodules metadata file, this allows you to include another git repo inside your repo. Joey leveraged that ability to add the youtube-dl latest master commit into his repo as a submodule, but deleted the .gitmodules file so that git wouldn't be verbose about it. And that's how you sneak a commit into a repo. This additionally leverages GitHub's data model (which is really the git data model), where they just have this huge DB of all the objects of all the repos on GitHub. So effectively by including this commit in a repo on GH, you're making it refer to the the entire (buried) youtube-dl repo, but you need to be sneaky to be able to see it.
- _ikke_ 6y ago
  Note that it's only a _reference_ to a commit, not the commit object itself. So from a a github point of view, it would not even make sense to scan / trigger on this. As long as there are no commit objects with that hash, any reference is harmless.
geofft 6y ago
It's Git performance art/protest art, more or less. Git has a mechanism called "submodules" which allows one repo to reference another repo. It doesn't actually include any of the content from the second repo: all that's included is the URL (in the file .gitmodules) and a commit hash (in Git's view of the directory structure). When you clone a Git repository with submodules and you pass an option to git clone, or if you run a git submodule command later, Git will make a nested clone at whatever subdirectory path contains the submodule, and check out that commit. If you make commits in the submodule (and, hopefully, remember to push them), the outer repo will appear to be modified, and you can git add the changes, which will record the new commit hash, but only the hash. The purpose is to deal with vendoring or sometimes making changes to large repositories owned by someone else without copying the whole source code into your own repository. (It's also occasionally used as a mechanism to split up very large repositories that are owned by the same group/organization, e.g., if you have a test suite or large data files or something, or a shared library used by multiple projects, you might find it helpful to use submodules.) https://github.com/ceph/ceph https://github.com/ceph/ceph is an example repo that uses them - "ceph-object-corpus" and "ceph-erasure-code-corpus" at the top level are submodules, and if you click on the .gitmodules file, you'll find that there are other submodules inside src/, too. This is also an example of how GitHub handles submodules also hosted on GitHub - it will link you to the submodule at that commit. So, GitHub does at least some parsing of submodules. The author is claiming that if you include a banned commit hash as a submodule in the history of your own project, and then promptly delete it, it won't noticeably increase the size of your Git repo, nor will you actually include any of the banned content yourself, but GitHub will potentially prevent that history from being pushed. As a result, you have a Git repo that GitHub would not accept. Since the author both does not use GitHub and wants to encourage others to do the same, this would be a fairly effective way of forcing that. (The article doesn't show GitHub actually refusing to accept the history containing this submodule, though - it only shows the submodule having been created, locally. Although maybe the author's argument is simply that pushing it violates the Terms of Service even if it's not blocked by automated means.)

Fellshard 6y ago

GitHub does not have a monopoly. That said, this is a very clever use of / exposure of the current systems in place. Unintended consequences of bad laws should be highlighted like this more regularly.

brunoqc 6y ago
> GitHub does not have a monopoly. Yeah, well, that's just, like, your opinion, man.
young_unixer 6y ago
Unintended?
dx034 6y ago
It depends which market you look at. Git hosting in general, especially for enterprise, Github has no monopoly. Gitlab (esp self hosted) is very strong there. But for public repositories, OSS development and to a certain degree even job applications, Github does have a monopoly.

dgrin91 6y ago

The author claims the commit hash is bad links to an article... but that claim seems entirely unsubstantiated? Its not the hash that is bad, its the files that are bad. If you manage to hack git to get the commit without the files (clever trick by him), you don't hit any legal issues. Am I missing something?

Fellshard 6y ago
The hash is probably an excellent heuristic for spotting commits from the original YouTube-DL Git log, and so GitHub probably scans for that when searching for rehosted copies.
- dgrin91 6y ago
  Agreed, but no one has actually stated that GH is scanning for these commits, let alone make the claim that the hashes are illegal. If you're going to start making legal claims they have to hold up to scrutiny, which this does not.
bloaf 6y ago
All you have to do to substantiate it is follow his procedure and see if you can successfully fork such a project to github. Let me know how it goes!

eminence32 6y ago

Quoting from the linked torrentfreak article: > If you commit or post content to this repository that violates our Terms of Service, we will delete that content and may suspend access to your account as well My reading of this is that github wants to stem the flood of garbage pull requests to their [dmca](https://github.com/github/dmca/pulls?q=is%3Apr+is%3Aclosed https://github.com/github/dmca/pulls?q=is%3Apr+is%3Aclosed) repo, which I guess is probably reasonable. I don't quite see where github is banning certain commit hashes or tree hashes on sight. I certainly respect the authors desire to not publish their repos to github (and indeed one of the nice things about git is how easy it is to self host). And I can understand not wanting others to fork the repos to github (due to objections about the company itself), but I'm not sure how I feel about adding deliberate boobytraps to repos that seems to be purely punitive.

joeyh 6y ago
It's currently posssible to make a PR to the github dmca repo that merges youtube-dl, and so violate their TOS. When you make that PR, you do not transfer any of youtube-dl to github. Their repo already has those objects. So all you have done is pushed a repo containing a specific hash, which is now a TOS violation, in that specific case at least. I wrote "Certain commit hashes are rapidly heading toward being illegal" because of that specificity.
- eminence32 6y ago
  Thanks for clarifying. I do understand that these specific hashes are effectively forbidden in the dmca repo. Are you aware of any specific instances where the presence of youtube-dl hashes in other repos have been found by github and used as the grounds for disciplinary action? That would be noteworthy indeed.
- nine_k 6y ago
  So, in addition to illegal primes, we are getting a collection of illegal random numbers? (I wonder when illegal transcendental numbers will become a thing.)
  - 6y ago

monksy 6y ago

IF you've already pulled it down your origin will be from where you pulled it. Create an empty repo else where. With your environment do a: git remote rm origin; git remote add origin <new environment> ; git push -u origin master You've filled up the new repo with what you have, history and all.

judge2020 6y ago
You should be including --mirror to get all refs pushed.
- monksy 6y ago
  Nice I had no idea about that arguement.

notsureaboutpg 6y ago

I feel good for Joey because he's really been one of the few who stuck to his principles and kept from hosting on Github even though it probably would have been way easier to do so. The git-annex project (and I'm sure many of his other ones) have been going strong without Github and I appreciate him for that. He's finally being vindicated I believe (and eventually many of the warners against these free-as-in-beer but not free-as-in-software services have been proven right)

hashtagmarkup 6y ago

You just turned a completely defensible innocent act into premeditated destruction of a livelihood.

joeyh 6y ago
If you feel that losing access to Github would mean the destruction of your professional life, I'd encourage you to take steps to prevent that corporation from being able to do that to you.
- hashtagmarkup 6y ago
  I don't feel that... the author feels that. It was the first line of the article.
  - ball_of_lint 6y ago
    (joeyh _is_ the author)
    - hashtagmarkup 6y ago
      hypocrisy has _many_ forms.
  - rnestler 6y ago
    I think that line was intended as a sarcastic joke.
- deleted 6y ago
  [deleted]

timmit 6y ago

publish to gitlab :)

mlindner 6y ago
Gitlab is the most awful mess of javascript you could ever invent though. I use it every day for work and I wish I could avoid it.
- dx034 6y ago
  Why's that? I've been using it for a while (on prem) and like the interface more than Github by now. I never problems in either Firefox or Chrome.
- INTPenis 6y ago
  I don't see it, and I'm a long time noscript user. Also been using Gitlab off-prem, on-prem, for more than 6 years. When someone says Gitlab problems all I can think of is resource issues with Sidekiq when I was self-hosting it. All this drama is only bringing to light how unsafe it is for projects to host their code on a closed platform. Better be on the safe side and use something you can easily migrate away, like Gitlab.
deleted 6y ago
[deleted]

erling 6y ago

Totally down with the potential conflict of interest or implications of GitHub, but it’s not GitHub specifically that’s zealous about DMCA. Any server and any host of that server is going to be subject to it. The only appeal of a smaller or private server is less visibility, but legally it’s the same. DMCA isn’t going anywhere.

deleted 6y ago
[deleted]
judge2020 6y ago
Well, GitHub/Microsoft could go on a PR campaign and say "we're not going to honor this RIAA DMCA since we know yt-dl isn't violating the DMCA!" but then GitHub/Microsoft would opening themselves up to a lawsuit against (basically) the entire music industry. The amount of goodwill MSFT loses over this (hopefully isolated) incident has to be worth a few orders of magnitude less than the tens of millions of dollars that would be burned to actually fight the RIAA. Smaller hosts could get away with not honoring DMCAs since the RIAA likely isn't going to waste resources actually filing a lawsuit, but this yt-dl situation seems like the perfect setup for the RIAA to set a precedent outlawing video/music downloaders if someone were to actually fight them on it (and until then, they can continue to take down video/music downloaders until someone does counter it).
- ponker 6y ago
  Microsoft is much wealthier than the “entire music industry” but this still might not be a hill they want to die on.
  - dx034 6y ago
    They don't want to get rid of copyright, it's just about creating a less punitive system. I don't think there's much downside for them if they really cared.
- pxeboot 6y ago
  Microsoft is actually a member of the RIAA [1]. [1] https://www.riaa.com/about-riaa/riaa-members/ https://www.riaa.com/about-riaa/riaa-members/

vfclists 6y ago

Why not call Github Microsoft knowing that Microsoft owns Github? Microsoft has refused to challenge a highly dubious DMCA notice when they have more than enough resources to do so. Isn't Microsoft the party at fault given that their lawyers know the DMCA notice doesn't have a leg to stand on.

kkarakk 6y ago
probably doing the ol' delayed switcheroo like facebook is doing with whatsapp - maintain product as is for a couple of years to avoid immediate backlash while slowly enmeshing and branding all the tools in the product. then boom one day it's called azure officeserver or some similar microsoft style name and monetised like crazy with github being a deprecated sub-project left to wither and die. it is NOT in their interest to defend github in any way. any negative press attached to the product literally doesn't matter to them since it's not associated with microsoft
- vfclists 6y ago
  It turns out Microsoft is a member of the RIAA - https://www.riaa.com/about-riaa/riaa-members/ https://www.riaa.com/about-riaa/riaa-members/. I should have known that. It figures.

m0zg 6y ago

I think we need to work towards the future where DVCS's are _distributed_ and federated. Every time you give anyone centralized control, they will abuse it. I'm not aware of any exceptions.

slimsag 6y ago

Others probably have not fully realized this yet, but with GitHub one can: 1) Publish arbitrary commits under your https://github.com/my/project https://github.com/my/project URL, e.g. a fake https://github.com/my/project/blob/<faked_commit>/README.md https://github.com/my/project/blob/<faked_commit>/README.md in your project describing how to install it that actually describes installing malware. 2) Publish those commits under your name, with your email address, and GitHub will prominently display it as if you made the commit (most do not use GPG signatures, and most do not know to look for "Verified" anyway) It seemed only a matter of time before this behavior got abused for something (anti-DMCA action is perhaps the best outcome of this situation I can imagine..)

eslaught 6y ago
I'm not quite sure what you're saying here. Are you claiming that you can push a commit to userA/project and then view it under the Github web interface for userB/project (assuming one repo is forked from the other)? If so it seems like that's a relatively easy fix for Github, just check if the commit is actually contained in userB's fork of the repo.
- slimsag 6y ago
  Quick demo: go to https://github.com/slimsag/linux/tree/5895e21f3c744ed9829e3afe9691e3eb1b1932ae#linux-kernel https://github.com/slimsag/linux/tree/5895e21f3c744ed9829e3a... then try replacing "slimsag" with "torvalds" These are "known issues" I believe GitHub doesn't intend to fix: 1. With youtube-dl[0] (and probably long before?) we know you can push a commit and view it under the web UI under another user, GitHub hasn't indicated this is a vulnerability at all. 2. Many people know you can impersonate another user through Git email addresses[1] (GPG signing is supposed to solve this, but a lot of people don't use it and even when they do others don't really know they should look for "Verified" in GitHub's web UI.) Combine these two known-issues and you get a really convincing phishing attack. I really hope this doesn't become a common-place thing. Hopefully this raises some awareness that, basically, you should not trust any GitHub URL with a commit SHA in it - only trust ones with branch names - because it could be a phishing attack otherwise. [0] https://news.ycombinator.com/item?id=24882921 https://news.ycombinator.com/item?id=24882921 [1] https://bounty.github.com/ineligible.html#impersonating_a_user_through_git_email_address https://bounty.github.com/ineligible.html#impersonating_a_us...

tirz 6y ago

When I first read this article, I was really sad. "If you use one of my open-source repository, you can get in trouble, mouhahaha" It feels like a dick move to me. But then, I realize that the RIAA bans can now be used as an anti RIAA partnership against GitHub and others, which is... pretty cool. Definitively a feature that we need \o/ !

rendall 6y ago

joeyh, if you're reading this, some feedback: on Android Chrome, at least, your top navbar overlays your article. Suggest you update your CSS for mobile views

gkfksjsjdlv 6y ago

Why are you worried about being banned from github? Who cares? Who gives a shit about the host they clone from? Engineer histrionics—just kill me rather than make me deal with these fucks with respect. Absolutely zero sense of priority.

Qwertious 6y ago

This sounds like that social engineering challenge one guy did - where he said "I lose if you convince me to post <token> anywhere in my github repo", and someone won it by saying "you should put that challenge on your website" since the website code was in his github repo.

joshenders 6y ago

Why are we shooting the messenger?

lk23jlkoij 6y ago

So why not just write a tiny script that goes through all youtube-dl commits, recommit them with an added a space character to some file. So you will have a clone with different hashes, but the same content.

pwdisswordfish0 6y ago
Since Git's DAG is a hash tree where each commit points to its parent(s) and its own hash is derived from that, then presumably it would suffice to do this only with the first commit.

jbb67 6y ago

first if all GitHub is nobodies "résumé" and secondly you can put a git repo anywhere you like there are any number of hosting sites or easy ways to just host it yourself. I mean really. what a stupid article

kkarakk 6y ago
There are a LOT of companies just spamming github repo owners with job interviews nowadays my last job was through an automated email i got about my github resume hashtags. iirc all they look for is certain tags and a regular commit history after which the recruiter takes a brief look at the repo and fires off the email if the criteria is met. seems enough to call github a resume to me.

woofie11 6y ago

Has anyone read github's terms of service? Github is the one violating them, not the users. Yes, github is disabling accounts to people who post youtube-dl. No, this is not in compliance with github's ToS. github says: “Please note that re-posting the exact same content that was the subject of a takedown notice without following the proper process is a violation of GitHub’s DMCA Policy and Terms of Service." Ref: https://torrentfreak.com/github-warns-users-reposting-youtube-dl-they-could-be-banned-201102/ https://torrentfreak.com/github-warns-users-reposting-youtub... github's DMCA policy says: "One of the best features of GitHub is the ability for users to "fork" one another's repositories. What does that mean? In essence, it means that users can make a copy of a project on GitHub into their own repositories. As the license or the law allows, users can then make changes to that fork to either push back to the main project or just keep as their own variation of a project. Each of these copies is a "fork" of the original repository, which in turn may also be called the "parent" of the fork. GitHub will not automatically disable forks when disabling a parent repository. This is because forks belong to different users, may have been altered in significant ways, and may be licensed or used in a different way that is protected by the fair-use doctrine. GitHub does not conduct any independent investigation into forks. We expect copyright owners to conduct that investigation and, if they believe that the forks are also infringing, expressly include forks in their takedown notice." Ref: https://docs.github.com/en/free-pro-team@latest/github/site-policy/dmca-takedown-policy#b-what-about-forks-or-whats-a-fork https://docs.github.com/en/free-pro-team@latest/github/site-... youtube-dl doesn't fall under any of the restrictions in github's ToS either. ref: https://docs.github.com/en/free-pro-team@latest/github/site-policy/github-acceptable-use-policies https://docs.github.com/en/free-pro-team@latest/github/site-... I have very little sympathy for github here. They ought to follow the DMCA process. Per the DMCA process, I can't dispute the takedown of the main Youtube-dl repo, since I'm not a party to the process. If I post youtube-dl to github, it should stay up, pending a DMCA notice from RIAA. If such a notice is sent, I should then have the right to then dispute that.

mlang23 6y ago

Its no longer relevant to me nor him, but I can't help but think the attitude of this article kind of violates the DFSG. If the stance is to insist that free software should be allowed to be redistributed by anyone, flaming those which have in the past wanted to mirror your code on github by indicating you could get them banned by fiddling with the hashes in your repo is pretty aggressive. I totally get the frustration, but when I learnt the DFSG, this was the biggest lesson for me. Freedom means freedom to anyone. Restricting your code to, say, non-military use, isn't compliant with the DFSG. So, now that we have perceived hostile code hosters, do we really want to go there? "You're not going to push my stuff to x.y.z, or I will make you feel the pain."

lolc 6y ago
What do you make of an author who only provides tarballs?

booleandilemma 6y ago

illegal number This is why aliens don't visit us.

blitblitblit 6y ago

> Github. Which, as we know, is your resume. There in lies the problem. Encouraging people to put their resume on centralized platforms (Be it Facebook, LinkedIn, or Github) is putting your entire career at risk. It's better to have your own domain and self-host, that way you aren't locked in and at risk of loosing your entire professional history if you get banned or a service goes offline.