10 ms·

GitHub Private Repos Considered Private-Ish

163 points by fagnerbrack 3y ago

justizin 3y ago
> We cram our secrets into git Excuse me?!
- hakre 3y ago
  What for? I've seen this happening and if there would not have been a review, it would have stayed there unnoticed. /edit: Also what someone considers a secret and then not, is often not well defined. If management has no clue what this is about, it is often better to only commit and push on direct and simple work order, because these need to be well understood and you have the paper trail (as that author also suggests blameful retrospectives - IMHO hilarious). Or do we have forgotten about the basic rules sending data over the interwebs to other people computers?
- Waterluvian 3y ago
  It’s slightly annoying when someone blunts doing something dumb by declaring some form of “we all do it.”
- Drakim 3y ago
  Yeah, I reacted to that too. It's like nonchalantly saying that you have all your passwords written on post-it notes at your desk. The topic of discussion shouldn't be how to secure your desk from spying eyes, but about why having post-it notes with passwords is bad practice and just a bad idea overall. If your private github repo accidentally goes public, the response should be "that's annoying but ultimately harmless", anything else is misguided.
  - bluGill 3y ago
    Postits for passwords are better practice than memorizing passwords. If you can memorize it, it is a bad password. Password managers are better yet, but you still need the master password. The problem is not keeping those passwords in a secure location, treat it like a stack of $100 bills.
    - Drakim 3y ago
      For your home desk? I can buy that. But for your work desk? No way is that even remotely more acceptable than having a memorizable password.
      - bluGill 3y ago
        You don't leave them on the desk. Lock them up with a key. Every office gives you file cabinet that locks.
        Drakim 3y ago
        That's basically an analog password manager, we have gone full circle. Or we return the metaphor to github repos, having a separate cabinet is like having a secret vault so that secrets are not directly in plain view in the repo itself, which is exactly what you should be doing.
        bluGill 3y ago
        Not quite full circle as we have now agreed that writing your passwords down on paper is acceptable.
        Drakim 3y ago
        I'll grant you that, it's not the medium that matters. But pen on paper wasn't my objection, it was the post-it note on the bezel of your monitor.
- _joel 3y ago
  This. Every CI platform under the sun has support for secrets and config that should never live in git. It's worth ensuring people know this, of course, but I'm not sure storing secrets in git is all that prevelant. Many platforms also have secrets scanning to ensure you don't accidentally do this too.
  - cratermoon 3y ago
    > Many platforms also have secrets scanning to ensure you don't accidentally do this too. The reason secrets scanning even became a thing is because of how often secrets get committed to git. Some of them even lead to intrusions. Uber (2016) – Attackers gained unrestricted access to Uber’s private Github repositories, found exposed secrets in the source code, and used them to access millions of records in Amazon S3 buckets. Scotiabank (2019) – Login credentials and access keys were left exposed in a public GitHub repo. Amazon (2020) – Credentials including AWS private keys were accidentally posted to a public GitHub repository by an AWS engineer. Symantec – Looking at hardcoded AWS keys in mobile apps, discovered they had a much wider permissions scope and led to a significant data leakage. GitHub – Over 100K public repositories on GitHub were found to contain access tokens. I've worked at companies with developers who didn't know that once committed, the secret remains in the history even if a subsequent commit removes it. It's not trivial, and involves rewriting the history[1]. There's also no way to fix clones of the repo, and there are a handful of other ways secrets can still leak. The most secure way to deal with secrets accidentally committed to git is to rotate the secret. 1 https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository https://docs.github.com/en/authentication/keeping-your-accou...
    - _joel 3y ago
      Yes, they happened in the past. Over 3 years back. Times change. You can use this and crenedials rotation, amazing, I know... https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning https://docs.github.com/en/code-security/secret-scanning/abo... https://docs.gitlab.com/ee/user/application_security/secret_detection/ https://docs.gitlab.com/ee/user/application_security/secret_... https://circleci.com/blog/detect-hardcoded-secrets-with-gitguardian/ https://circleci.com/blog/detect-hardcoded-secrets-with-gitg... https://www.gitkraken.com/media/events/azure-spring-clean-2022 https://www.gitkraken.com/media/events/azure-spring-clean-20... edit: used config/secrets management for as long as I can remember doing this cloud stuff (several years), so the excuses are very, very poor imho.
- deleted 3y ago
  [deleted]
- cobbal 3y ago
  If source code isn't public, it's a secret by the traditional definition of the word.
  - falcor84 3y ago
    A secret is something that you go out of your way to keep private. But there are a lot of other things that are private by default, but aren't really secrets, like "the brand of toilet paper I use".
  - snowwrestler 3y ago
    > by the traditional definition of the word. Sure, but in computer programming “secrets” is also industry jargon for small strings of characters that enable authentication, like passwords or private keys, which have much higher standard of secrecy than the rest of the codebase.
- sobellian 3y ago
  Some folks use tools like https://github.com/mozilla/sops https://github.com/mozilla/sops to store most secrets (besides the sops key, of course) in source control. Of course, you aren't committing the cleartext but if the repo gets published you should probably rotate your keys just to be safe...
  - solatic 3y ago
    Even this I would consider to be bad practice. Old versions of secrets are never relevant. Easy way to break your system: 1. Write code v1 2. Add secret 3. Write code v2 4. Rotate secret 5. Oops, some kind of problem, let's go back to known-good and redeploy (2). Broken because it tries the older secret, not the rotated secret. Just don't store secrets in version control.
    - electroly 3y ago
      That one has an easy fix: store secrets in a separate repo that you never roll back. That's not the reason to avoid storing secrets in git. You might be giving some junior dev here the idea that if they can solve this issue, then storing secrets in git will be ok. Obviously it's not; it's still a bad idea after you've solved this minor annoyance and, indeed, this annoyance had nothing to do with the security reason why you don't store secrets in git.
    - sobellian 3y ago
      This assumes that the secrets are deployed along with everything else in the repository. Even if the same repository contains your app, they needn't be deployed together. And as far as old secrets go, they are at most as sensitive as current secrets.
- electroly 3y ago
  OP's target audience, judging by all the emojis and the subject matter, seems to be a pretty green junior devs who might not know any better. OP is putting themselves in their shoes as a rhetorical device, the author isn't saying that they themselves put secrets into git.
jasoneckert 3y ago
This indirectly outlines one of the primary reasons organizations (like mine) prefer using GitHub Enterprise - we get the collaboration benefits of GitHub, but our data is entirely controlled and hosted by us. It's extra work and more costly overall, but it's a small price to pay for the data security.
- ckdarby 3y ago
  From my perspective, the upkeep of maintaining such versions is far from trivial. It involves dedicated resources ensuring the regularity and reliability of backups. Furthermore, it's rare to find organizations conducting comprehensive disaster recovery (DR) restorations of their self-hosted GitLab or GitHub instances. This is typically due to an inherent expectation that each upgrade will proceed without hitches and yield flawless operation. Moreover, the process of securing approval for NAT inbound rules, particularly for integrations with services like Jira, Slack and so on, often turns into a labyrinthine ordeal. It usually involves navigating the differing viewpoints and interpretations across multiple departments, each with its own unique stance, which further exacerbates the complexity of the task.
  - yetanotherloss 3y ago
    As an aside, it remains funny to me how much effort organizations put into fighting NAT in 2023 instead of effort to just use ipv6. Of course then there is the embarrassment of azure ipv6 so it's perhaps somewhat forgivable:)
    - awesometech 3y ago
      There are many valid reasons to not use IPv6, it would make an interesting HN article.
      - Fogest 3y ago
        My ISP still doesn't even support ipv6. From my limited understanding that itself would already make it a problem to access an ipv6 only site.
cj 3y ago
Recommendations missing from the article: - Enable mandatory 2fa within your Github organization (if you don't use an organization, you probably should) - Disable the ability to fork repos in your organization - Configure and enable mandatory SAML authentication. In combination with mandatory 2fa, this makes phishing and even key leakage less likely (specific keys need to be double authorized for SAML, so that random private key you have on a random CI/CD platform from a few years ago can't access repos in SAML organizations even if it's leaked) - Disable the ability to make repos public at the organization level These are some quick setting changes that should address at least 3 of the 4 main issues in the article. Additional suggestion: Enable branch protection on master. Require at least 1 peer review to merge anything in it. Enforce branch restrictions to include repo admins so restrictions can't be bypassed. This should stop obvious mistakes like accidentally committing .git or random credentials. Edit: if the author sees this, feel free to add these things to the article!
- flerchin 3y ago
  What does "Disable the ability to fork repos in your organization" do? Isn't that entirely defeated by users changing the origin?
  - lelandfe 3y ago
    Yes, it's just lip service. The moment you've cloned a repo to your computer, you have "forked" it. It's simply the nature of decentralized version control.
  - cj 3y ago
    It prevents it from happening in the GitHub UI. Of course it’s still possible to download the code and upload it to a separate repo (but then it’s not a fork).
    - hedora 3y ago
      Of course bypassing the UI means that normal development flows (like force rebasing, or futzing with the CI setup) now are the same set of commands as malicious exfiltration of source code. It also breaks GitHub's normal protection against accidentally creating public forks of private repos (and the feature where it auto-deletes your private forks when you change jobs). There are probably other ways in which disabling private forks undermines your organization's security, but those are pretty obvious.
waynecochran 3y ago
Don’t forget OpenAI and Microsoft using your github data for training GPT. Their privacy statement says your content will not be read by “human eyes.” https://docs.github.com/en/get-started/privacy-on-github/about-githubs-use-of-your-data#privacy-and-data-sharing https://docs.github.com/en/get-started/privacy-on-github/abo...
- dotBen 3y ago
  I never read that before but you are right and based on that wording looks like private repos are open to having LLMs trained upon them. I assumed up to this point they could only use public ones but this wording suggests otherwise.
- paddw 3y ago
  iirc, they said they wouldn't do this? so this could just refer to secret scanning
  - mtlmtlmtlmtl 3y ago
    Clearly they lied, if their policy says otherwise.
    - blackoil 3y ago
      Policy says nothing about ChatGPT or LLM/AI models.
      - deleted 3y ago
        [deleted]
    - mikeryan 3y ago
      Their policy, if you scroll up from this link, is to scan only “aggregate metadata” and only if you opt in. GitHub aggregates metadata and parses content patterns for the purposes of delivering generalized insights within the product. It uses data from public repositories, and also uses metadata and aggregate data from private repositories when a repository's owner has chosen to share the data with GitHub by enabling the dependency graph. If you enable the dependency graph for a private repository, then GitHub will perform read-only analysis of that specific private repository. If you enable data use for a private repository, we will continue to treat your private data, source code, or trade secrets as confidential and private consistent with our Terms of Service. The information we learn only comes from aggregated data. For more information, see "Managing data use settings for your private repository."
majestic5762 3y ago
Silly and HN hand-crafted. Too many noobs nowadays around lol
quickthrower2 3y ago
Avoid having secrets if you can. Use authentication methods that don’t require them. When they are required rotate them often - ideally automated. Keep them out of password managers. They are like condoms: cheap enough to use once, unpleasant if used twice. Hence the common UX where you get to see the key once encourages using once. Give those tokens the minimum permissions possible. Make sure the impact of needing to rotate that secret doesn’t mean other secrets are exposed (looking at you: azure storage account primary access key!) Automated rotations make keeping them up to date in git untenable anyway. Some services like Vercel and Firebase encourage bad practices in this regard. Firebase requires you to download a plaintext JSON of kingdom keys in order to do anything useful! Vercel KV storage encourages you to run an SDK command to download prod secrets into a local env file.
mikeryan 3y ago
I dunno about other people but I’ve always assumed that if we’re giving employees, who we may have to fire someday, access to the code base that repos should be treated as if they could be exposed to the world at any time any way.
- hakre 3y ago
  This is a great baseline, but it is often easier said than done. For example, those repositories contain a lot of privately identifiable information, it is not that easy to get such a baseline ready for that _"should be treated as if they could be exposed to the world at any time any way."_ Depending on jurisdiction this can affect sensitive information that requires much stronger controls in place when you (rightfully!) expect the repository to become public despite it is a private one.
  - mikeryan 3y ago
    For example, those repositories contain a lot of privately identifiable information This seems to miss my point - I have no idea why PII is in a code repo.
    - hakre 3y ago
      Those of the workforce, that is a pretty typical situation. Timestamps of activity etc. .
- bob1029 3y ago
  I've had to explain to leadership that certain employees don't even need access to the actual codebase to potentially cause harm via a competitor. The real solution to this is to have a business model that is about more than just some source code in a repository, or "things one person could undo". Concerns like years-long relationships with customers (aka trust), platform-style lock-in patterns, etc. How many times have various parts of AAA game studio or Microsoft codebases been compromised? I am still waiting for that hacker edition of HL3 and my free copy of windows that doesn't suck. I actually spent a few minutes walking through a hypothetical where 100% of our latest code is lifted and taken to our biggest competitor. I think it would probably cause them more harm than good. Complexity is a hell of a thing. Just having a point-in-time snapshot doesn't really give you an advantage over someone who has been at it continuously for years. Perhaps we can strike a consulting contract with whoever steals our code... I am not against reasonable measures (i.e. MFA, GH Enterprise, VPNs), but I won't go into paranoid-tier (Citrix-style desktops) over stuff like this ever again. Code is a cheap commodity in 2023. Enshrining your repository as if it is the actual vehicle of business value is indicative of poor leadership. Customers, relationships, execution, etc. are way more challenging and important.
tln 3y ago
How about encryption? https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt has been solid for me
- ghusto 3y ago
  I like this a lot and use it myself. Always have a tough time convincing developers though, because they don't like "all that terminal stuff" :/
  - shrimp_emoji 3y ago
    Are you sure they're developers? Nevermind, I guess they can be web developers :p
    - ghusto 3y ago
      Didn't want to go there, but I think you get it. When your developers are using JavaScript to write backend programs, you know where the bar is.
- computerfriend 3y ago
  Old secrets are around forever, tied to long lived credentials (PGP keys) who's access can't be revoked (because it's always in the commit history). Additionally, it can be quite painful in an ever-changing collaborative context.
- remram 3y ago
  Assuming you mean encrypting the secret files specifically and not the whole repo. This could work, but why put the secrets in the repository at all? Either everyone has to replace them with their own secrets before being able to run the code. Or everyone has to have the key to decrypt the actual credentials (developers, interns, deployments, CI, scanners, etc) making it not secret. Why not inject the credentials at runtime, from a system meant for this, with support for auditing and key rotation etc?
blackoil 3y ago
Does the article has anything Github specific? All softwares can have bug so moving from Github to anything else is useless. VPN only self-hosted provides some additional protection but that's true for all cloud services like Jira, CI/CD and not a novel issue.
- awesometech 3y ago
  The article cites several GitHub incidents. If it can be verified GitHub competitors have a roughly equal number of incidents then the lack of novelty would be proven. I'm not supporting either view, just addressing your point.
lvkv 3y ago
GitHub Enterprise had (has?) an interesting loophole to discover the existence of private repositories. Attempting to transfer ownership of a repository to another user was aborted if the user had a repository of the same name—even if it was private. Public GitHub doesn’t seem to have this issue with the transfer request system, though. Maybe it did at some point?
p-e-w 3y ago
This post is mostly FUD. Case in point, the following advice: > So, if you’re worried about it: stop putting sensitive data into private repositories. Most of the issues mentioned in the post (misconfiguration, phishing, mistakes, zero-days) apply to all software, including non-cloud software. So the above advice is equivalent to "stop putting sensitive data into computers". It's run-of-the-mill popular-security nonsense that conveniently ignores that the alternatives come with their own risks, including security risks, and doesn't even attempt to perform a cost-benefit analysis. Less of this stuff, please.
- ghusto 3y ago
  I like to think myself as a pragmatic practitioner of security, and even I would say this isn't "popular-security nonsense". The "cost-benefit analysis" here equals: don't bloody do that, _obviously_. If your system relies on secrets being present in repos, it is a poorly designed system. There is no scenario where this is necessary, other than one not wanting to put in the effort to inject secrets sensibly. Yes, "the alternatives come with their own risks", but those risks are where the "cost-benefit analysis" equals "not worth the effort".
  - p-e-w 3y ago
    > If your system relies on secrets being present in repos, it is a poorly designed system. Repos are databases that track changes in files, nothing more and nothing less. There are millions of repos on GitHub that don't contain application code. Some people put their entire home directory in a Git repository. Home directories almost always contain secrets. That doesn't mean it's a bad idea, it just means the repo isn't meant for the eyes of others. In other words, it's a private repository.
    - ghusto 3y ago
      People putting their entire home directory in a repo, on somebody else's shared computer and network, is going to end in a disaster scenario. This sounds like a bad idea.
- 3y ago
layer8 3y ago
This is true for everything in the cloud that isn’t end-to-end-encrypted. There is the saying that the cloud is just someone else's computer. But in addition, you also have no control over your data on that computer not being leaked elsewhere, or the computer ending up becoming someone different’s computer.
- LinuxBender 3y ago
  Adding to this my own pessimistic take is that once $big_company that acquired $useful_service decides they have obtained a sufficient level of corporate capture with paid businesses then they may decide to add limits to non-paid repository counts, sizes, pulls, pushes and tighten those limits each year until the non-paying members have moved on. Something similar may be slowly happening with Gmail based on ask hn submissions.
st3fan 3y ago
Getting a bit tired of the anti-GitHub threads here ...
ImJasonH 3y ago
You could also leak private repo names via GHCR: https://www.chainguard.dev/unchained/ghcr-private-repos-sometimes-werent https://www.chainguard.dev/unchained/ghcr-private-repos-some...
snowwrestler 3y ago
It’s right there in the name: GitHub private repos are “private,” not “secure” or “secret”. That name was chosen purposefully. When we moved our code from privately hosted SVN to GitHub 10+ years ago, folks at GitHub were quite clear that we should not trust private repos with secrets like private keys or passwords. So why have private repos at all? It is to allow control of collaborators. GitHub originally allowed everyone to see and fork every repo—a true open source approach. Private repos were added basically so corporate codebase managers would not have to spend tons of time rejecting random PRs and responding to random people. For a LOT of private software projects, everything secret can be easily stored in the database, environments, or a dedicated tool for managing secrets. GitHub private repos are great for those. If my entire codebase is a highly sensitive trade secret, I would not use GitHub private repos, personally.
- willis936 3y ago
  When I have an expectation of privacy, say in a fenced backyard, I don't expect to be seen. It doesn't matter if it's a human or a robot commanded by a human that does the looking, it's still a violation of privacy. It's not really "private" if the backyard has a one way glass pane on one wall. Github should not be calling repos private if they're not.
  - snowwrestler 3y ago
    This is one of many instances where developers should not make shallow assumptions without reading the documentation. GitHub is very clear about how private repos should be used. Edit to add: would you store your bank password on a piece of paper you always leave sitting in your backyard? Think carefully about your metaphor.
    - owl57 3y ago
      > would you store your bank password on a piece of paper you always leave sitting in your backyard? Not that I'm advocating for checking passwords in git repos, Github or otherwise, but you metaphor depends on how rich you are and how bad is your neighborhood, I would think. My mom has definitely done exactly that — her medical conditions make it very hard to use bank apps without full daylight and also hard to bring the notebook home after every use — and I don't think that her password management choices, however frowned upon they may be here, have ever been a serious problem in practice.
franciscop 3y ago
I'm not sure why in the "Easy Fixes" it doesn't say the thing that is mentioned in the next point "stop putting sensitive data into private repositories". That should _also_ go in "Easy Fixes" IMHO, use environment variables, vaults, etc to not store your keys into your code.
satya71 3y ago
I wonder, how do Azure DevOps and AWS CodeCommit compare in this regard?
trentnix 3y ago
A company I worked for a few years ago sold a source code license and I was tasked with sharing the source, sans history and only at a specific point in time, to the purchaser’s GitHub organization . I was told it had to be done as quickly as possible, and that we might do it a few more times so figure out a way to make it repeatable. I wrote a Bash script to copy our ~25 repositories to a new organization using the GitHub REST API. Turns out that the default when creating a new repository was to make it public. Within a few minutes I was getting emails by third party services who had downloaded and parsed our code, discovering a long abandoned AWS credential hard-coded in an older codebase. (I think GitHub now does this for you) I was lucky that nothing important was actually compromised (we had done a decent job of keeping secrets out of our repos and the one exception was an account that was long gone), but it was an eye-opening experience. If these services had found and downloaded our code in minutes you can assume any repo made public, even for a few moments, has been downloaded and cataloged by a potential bad actor.
- ec109685 3y ago
  Any idea how successful the company who you sold sold the license to was with code base? Trying to develop on a code base without history and a one time snapshot seems quite hard.
redm 3y ago
I see a lot of deserved distrust of MS, but I thought GitHub was a operated as a separate unit under Microsoft? [1] I expected that Co-Pilot was an initiative of that leadership team and training the LLM is what's likely reading certain repositories? On a side note, I'm trying to imagine what "sensitive" code would be read, incorporated into an LLM such as Co-pilot, and somehow have any meaningful impact to me once incorporated? [1] https://github.com/about/leadership https://github.com/about/leadership
2devnull 3y ago
People like to put api keys in their repos and it’s hard to make them grasp why they shouldn’t. I guess some lessons have to be learned the hard way.
rvz 3y ago
Unsurprising. Private repositories are not 'private' [0] and GitHub, Microsoft and OpenAI are thanking you for allowing them to train their AI on your so-called 'private' code. [1] This is essentially a shameless admission and they not even hiding it any more: Private repository data is scanned by machine and never read by GitHub staff. Human eyes will never see the contents of your private repositories, except as described in our Terms of Service. This is another great reason to self host instead of using services like GitHub openly reading your private code, which I have been saying for years. [0] https://news.ycombinator.com/item?id=34258431 https://news.ycombinator.com/item?id=34258431 [1] https://docs.github.com/en/get-started/privacy-on-github/about-githubs-use-of-your-data#privacy-and-data-sharing https://docs.github.com/en/get-started/privacy-on-github/abo...
sebzim4500 3y ago
>We cram our secrets into git, then shove it off to the most expansive code forge in the history of humanity Do we? Putting secrets in git is against policy almost everywhere.
jononomo 3y ago
Have you guys considered the machines are reading your private repos all the time? Forget about LLMs -- what about your text editor? Also, it is likely that your browser has access to your private repo when you use your browser to browse your private repo.
themoonisachees 3y ago
As the person on call for exactly that sort of problem, it saddens me that people still push secrets to git at all. Every gitops platform on the planet supports not pushing secrets to version control. There is no valid reason to have credentials inside a git repo ever. If the repo itself is what's secret, you should have known the risk when you pushed it to someone else's server. There is no excuse when gitlab is free and a docker container away.

163 points by fagnerbrack 3y ago

justizin 3y ago

> We cram our secrets into git Excuse me?!

hakre 3y ago
What for? I've seen this happening and if there would not have been a review, it would have stayed there unnoticed. /edit: Also what someone considers a secret and then not, is often not well defined. If management has no clue what this is about, it is often better to only commit and push on direct and simple work order, because these need to be well understood and you have the paper trail (as that author also suggests blameful retrospectives - IMHO hilarious). Or do we have forgotten about the basic rules sending data over the interwebs to other people computers?
Waterluvian 3y ago
It’s slightly annoying when someone blunts doing something dumb by declaring some form of “we all do it.”
Drakim 3y ago
Yeah, I reacted to that too. It's like nonchalantly saying that you have all your passwords written on post-it notes at your desk. The topic of discussion shouldn't be how to secure your desk from spying eyes, but about why having post-it notes with passwords is bad practice and just a bad idea overall. If your private github repo accidentally goes public, the response should be "that's annoying but ultimately harmless", anything else is misguided.
- bluGill 3y ago
  Postits for passwords are better practice than memorizing passwords. If you can memorize it, it is a bad password. Password managers are better yet, but you still need the master password. The problem is not keeping those passwords in a secure location, treat it like a stack of $100 bills.
  - Drakim 3y ago
    For your home desk? I can buy that. But for your work desk? No way is that even remotely more acceptable than having a memorizable password.
    - bluGill 3y ago
      You don't leave them on the desk. Lock them up with a key. Every office gives you file cabinet that locks.
      - Drakim 3y ago
        That's basically an analog password manager, we have gone full circle. Or we return the metaphor to github repos, having a separate cabinet is like having a secret vault so that secrets are not directly in plain view in the repo itself, which is exactly what you should be doing.
        bluGill 3y ago
        Not quite full circle as we have now agreed that writing your passwords down on paper is acceptable.
        Drakim 3y ago
        I'll grant you that, it's not the medium that matters. But pen on paper wasn't my objection, it was the post-it note on the bezel of your monitor.
_joel 3y ago
This. Every CI platform under the sun has support for secrets and config that should never live in git. It's worth ensuring people know this, of course, but I'm not sure storing secrets in git is all that prevelant. Many platforms also have secrets scanning to ensure you don't accidentally do this too.
- cratermoon 3y ago
  > Many platforms also have secrets scanning to ensure you don't accidentally do this too. The reason secrets scanning even became a thing is because of how often secrets get committed to git. Some of them even lead to intrusions. Uber (2016) – Attackers gained unrestricted access to Uber’s private Github repositories, found exposed secrets in the source code, and used them to access millions of records in Amazon S3 buckets. Scotiabank (2019) – Login credentials and access keys were left exposed in a public GitHub repo. Amazon (2020) – Credentials including AWS private keys were accidentally posted to a public GitHub repository by an AWS engineer. Symantec – Looking at hardcoded AWS keys in mobile apps, discovered they had a much wider permissions scope and led to a significant data leakage. GitHub – Over 100K public repositories on GitHub were found to contain access tokens. I've worked at companies with developers who didn't know that once committed, the secret remains in the history even if a subsequent commit removes it. It's not trivial, and involves rewriting the history[1]. There's also no way to fix clones of the repo, and there are a handful of other ways secrets can still leak. The most secure way to deal with secrets accidentally committed to git is to rotate the secret. 1 https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository https://docs.github.com/en/authentication/keeping-your-accou...
  - _joel 3y ago
    Yes, they happened in the past. Over 3 years back. Times change. You can use this and crenedials rotation, amazing, I know... https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning https://docs.github.com/en/code-security/secret-scanning/abo... https://docs.gitlab.com/ee/user/application_security/secret_detection/ https://docs.gitlab.com/ee/user/application_security/secret_... https://circleci.com/blog/detect-hardcoded-secrets-with-gitguardian/ https://circleci.com/blog/detect-hardcoded-secrets-with-gitg... https://www.gitkraken.com/media/events/azure-spring-clean-2022 https://www.gitkraken.com/media/events/azure-spring-clean-20... edit: used config/secrets management for as long as I can remember doing this cloud stuff (several years), so the excuses are very, very poor imho.
deleted 3y ago
[deleted]
cobbal 3y ago
If source code isn't public, it's a secret by the traditional definition of the word.
- falcor84 3y ago
  A secret is something that you go out of your way to keep private. But there are a lot of other things that are private by default, but aren't really secrets, like "the brand of toilet paper I use".
- snowwrestler 3y ago
  > by the traditional definition of the word. Sure, but in computer programming “secrets” is also industry jargon for small strings of characters that enable authentication, like passwords or private keys, which have much higher standard of secrecy than the rest of the codebase.
sobellian 3y ago
Some folks use tools like https://github.com/mozilla/sops https://github.com/mozilla/sops to store most secrets (besides the sops key, of course) in source control. Of course, you aren't committing the cleartext but if the repo gets published you should probably rotate your keys just to be safe...
- solatic 3y ago
  Even this I would consider to be bad practice. Old versions of secrets are never relevant. Easy way to break your system: 1. Write code v1 2. Add secret 3. Write code v2 4. Rotate secret 5. Oops, some kind of problem, let's go back to known-good and redeploy (2). Broken because it tries the older secret, not the rotated secret. Just don't store secrets in version control.
  - electroly 3y ago
    That one has an easy fix: store secrets in a separate repo that you never roll back. That's not the reason to avoid storing secrets in git. You might be giving some junior dev here the idea that if they can solve this issue, then storing secrets in git will be ok. Obviously it's not; it's still a bad idea after you've solved this minor annoyance and, indeed, this annoyance had nothing to do with the security reason why you don't store secrets in git.
  - sobellian 3y ago
    This assumes that the secrets are deployed along with everything else in the repository. Even if the same repository contains your app, they needn't be deployed together. And as far as old secrets go, they are at most as sensitive as current secrets.
electroly 3y ago
OP's target audience, judging by all the emojis and the subject matter, seems to be a pretty green junior devs who might not know any better. OP is putting themselves in their shoes as a rhetorical device, the author isn't saying that they themselves put secrets into git.

jasoneckert 3y ago

This indirectly outlines one of the primary reasons organizations (like mine) prefer using GitHub Enterprise - we get the collaboration benefits of GitHub, but our data is entirely controlled and hosted by us. It's extra work and more costly overall, but it's a small price to pay for the data security.

ckdarby 3y ago
From my perspective, the upkeep of maintaining such versions is far from trivial. It involves dedicated resources ensuring the regularity and reliability of backups. Furthermore, it's rare to find organizations conducting comprehensive disaster recovery (DR) restorations of their self-hosted GitLab or GitHub instances. This is typically due to an inherent expectation that each upgrade will proceed without hitches and yield flawless operation. Moreover, the process of securing approval for NAT inbound rules, particularly for integrations with services like Jira, Slack and so on, often turns into a labyrinthine ordeal. It usually involves navigating the differing viewpoints and interpretations across multiple departments, each with its own unique stance, which further exacerbates the complexity of the task.
- yetanotherloss 3y ago
  As an aside, it remains funny to me how much effort organizations put into fighting NAT in 2023 instead of effort to just use ipv6. Of course then there is the embarrassment of azure ipv6 so it's perhaps somewhat forgivable:)
  - awesometech 3y ago
    There are many valid reasons to not use IPv6, it would make an interesting HN article.
    - Fogest 3y ago
      My ISP still doesn't even support ipv6. From my limited understanding that itself would already make it a problem to access an ipv6 only site.

cj 3y ago

Recommendations missing from the article: - Enable mandatory 2fa within your Github organization (if you don't use an organization, you probably should) - Disable the ability to fork repos in your organization - Configure and enable mandatory SAML authentication. In combination with mandatory 2fa, this makes phishing and even key leakage less likely (specific keys need to be double authorized for SAML, so that random private key you have on a random CI/CD platform from a few years ago can't access repos in SAML organizations even if it's leaked) - Disable the ability to make repos public at the organization level These are some quick setting changes that should address at least 3 of the 4 main issues in the article. Additional suggestion: Enable branch protection on master. Require at least 1 peer review to merge anything in it. Enforce branch restrictions to include repo admins so restrictions can't be bypassed. This should stop obvious mistakes like accidentally committing .git or random credentials. Edit: if the author sees this, feel free to add these things to the article!

flerchin 3y ago
What does "Disable the ability to fork repos in your organization" do? Isn't that entirely defeated by users changing the origin?
- lelandfe 3y ago
  Yes, it's just lip service. The moment you've cloned a repo to your computer, you have "forked" it. It's simply the nature of decentralized version control.
- cj 3y ago
  It prevents it from happening in the GitHub UI. Of course it’s still possible to download the code and upload it to a separate repo (but then it’s not a fork).
  - hedora 3y ago
    Of course bypassing the UI means that normal development flows (like force rebasing, or futzing with the CI setup) now are the same set of commands as malicious exfiltration of source code. It also breaks GitHub's normal protection against accidentally creating public forks of private repos (and the feature where it auto-deletes your private forks when you change jobs). There are probably other ways in which disabling private forks undermines your organization's security, but those are pretty obvious.

waynecochran 3y ago

Don’t forget OpenAI and Microsoft using your github data for training GPT. Their privacy statement says your content will not be read by “human eyes.” https://docs.github.com/en/get-started/privacy-on-github/about-githubs-use-of-your-data#privacy-and-data-sharing https://docs.github.com/en/get-started/privacy-on-github/abo...

dotBen 3y ago
I never read that before but you are right and based on that wording looks like private repos are open to having LLMs trained upon them. I assumed up to this point they could only use public ones but this wording suggests otherwise.
paddw 3y ago
iirc, they said they wouldn't do this? so this could just refer to secret scanning
- mtlmtlmtlmtl 3y ago
  Clearly they lied, if their policy says otherwise.
  - blackoil 3y ago
    Policy says nothing about ChatGPT or LLM/AI models.
    - deleted 3y ago
      [deleted]
  - mikeryan 3y ago
    Their policy, if you scroll up from this link, is to scan only “aggregate metadata” and only if you opt in. GitHub aggregates metadata and parses content patterns for the purposes of delivering generalized insights within the product. It uses data from public repositories, and also uses metadata and aggregate data from private repositories when a repository's owner has chosen to share the data with GitHub by enabling the dependency graph. If you enable the dependency graph for a private repository, then GitHub will perform read-only analysis of that specific private repository. If you enable data use for a private repository, we will continue to treat your private data, source code, or trade secrets as confidential and private consistent with our Terms of Service. The information we learn only comes from aggregated data. For more information, see "Managing data use settings for your private repository."

majestic5762 3y ago

Silly and HN hand-crafted. Too many noobs nowadays around lol

quickthrower2 3y ago

Avoid having secrets if you can. Use authentication methods that don’t require them. When they are required rotate them often - ideally automated. Keep them out of password managers. They are like condoms: cheap enough to use once, unpleasant if used twice. Hence the common UX where you get to see the key once encourages using once. Give those tokens the minimum permissions possible. Make sure the impact of needing to rotate that secret doesn’t mean other secrets are exposed (looking at you: azure storage account primary access key!) Automated rotations make keeping them up to date in git untenable anyway. Some services like Vercel and Firebase encourage bad practices in this regard. Firebase requires you to download a plaintext JSON of kingdom keys in order to do anything useful! Vercel KV storage encourages you to run an SDK command to download prod secrets into a local env file.

mikeryan 3y ago

I dunno about other people but I’ve always assumed that if we’re giving employees, who we may have to fire someday, access to the code base that repos should be treated as if they could be exposed to the world at any time any way.

hakre 3y ago
This is a great baseline, but it is often easier said than done. For example, those repositories contain a lot of privately identifiable information, it is not that easy to get such a baseline ready for that _"should be treated as if they could be exposed to the world at any time any way."_ Depending on jurisdiction this can affect sensitive information that requires much stronger controls in place when you (rightfully!) expect the repository to become public despite it is a private one.
- mikeryan 3y ago
  For example, those repositories contain a lot of privately identifiable information This seems to miss my point - I have no idea why PII is in a code repo.
  - hakre 3y ago
    Those of the workforce, that is a pretty typical situation. Timestamps of activity etc. .
bob1029 3y ago
I've had to explain to leadership that certain employees don't even need access to the actual codebase to potentially cause harm via a competitor. The real solution to this is to have a business model that is about more than just some source code in a repository, or "things one person could undo". Concerns like years-long relationships with customers (aka trust), platform-style lock-in patterns, etc. How many times have various parts of AAA game studio or Microsoft codebases been compromised? I am still waiting for that hacker edition of HL3 and my free copy of windows that doesn't suck. I actually spent a few minutes walking through a hypothetical where 100% of our latest code is lifted and taken to our biggest competitor. I think it would probably cause them more harm than good. Complexity is a hell of a thing. Just having a point-in-time snapshot doesn't really give you an advantage over someone who has been at it continuously for years. Perhaps we can strike a consulting contract with whoever steals our code... I am not against reasonable measures (i.e. MFA, GH Enterprise, VPNs), but I won't go into paranoid-tier (Citrix-style desktops) over stuff like this ever again. Code is a cheap commodity in 2023. Enshrining your repository as if it is the actual vehicle of business value is indicative of poor leadership. Customers, relationships, execution, etc. are way more challenging and important.

tln 3y ago

How about encryption? https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt has been solid for me

ghusto 3y ago
I like this a lot and use it myself. Always have a tough time convincing developers though, because they don't like "all that terminal stuff" :/
- shrimp_emoji 3y ago
  Are you sure they're developers? Nevermind, I guess they can be web developers :p
  - ghusto 3y ago
    Didn't want to go there, but I think you get it. When your developers are using JavaScript to write backend programs, you know where the bar is.
computerfriend 3y ago
Old secrets are around forever, tied to long lived credentials (PGP keys) who's access can't be revoked (because it's always in the commit history). Additionally, it can be quite painful in an ever-changing collaborative context.
remram 3y ago
Assuming you mean encrypting the secret files specifically and not the whole repo. This could work, but why put the secrets in the repository at all? Either everyone has to replace them with their own secrets before being able to run the code. Or everyone has to have the key to decrypt the actual credentials (developers, interns, deployments, CI, scanners, etc) making it not secret. Why not inject the credentials at runtime, from a system meant for this, with support for auditing and key rotation etc?

blackoil 3y ago

Does the article has anything Github specific? All softwares can have bug so moving from Github to anything else is useless. VPN only self-hosted provides some additional protection but that's true for all cloud services like Jira, CI/CD and not a novel issue.

awesometech 3y ago
The article cites several GitHub incidents. If it can be verified GitHub competitors have a roughly equal number of incidents then the lack of novelty would be proven. I'm not supporting either view, just addressing your point.

lvkv 3y ago

GitHub Enterprise had (has?) an interesting loophole to discover the existence of private repositories. Attempting to transfer ownership of a repository to another user was aborted if the user had a repository of the same name—even if it was private. Public GitHub doesn’t seem to have this issue with the transfer request system, though. Maybe it did at some point?

p-e-w 3y ago

This post is mostly FUD. Case in point, the following advice: > So, if you’re worried about it: stop putting sensitive data into private repositories. Most of the issues mentioned in the post (misconfiguration, phishing, mistakes, zero-days) apply to all software, including non-cloud software. So the above advice is equivalent to "stop putting sensitive data into computers". It's run-of-the-mill popular-security nonsense that conveniently ignores that the alternatives come with their own risks, including security risks, and doesn't even attempt to perform a cost-benefit analysis. Less of this stuff, please.

ghusto 3y ago
I like to think myself as a pragmatic practitioner of security, and even I would say this isn't "popular-security nonsense". The "cost-benefit analysis" here equals: don't bloody do that, _obviously_. If your system relies on secrets being present in repos, it is a poorly designed system. There is no scenario where this is necessary, other than one not wanting to put in the effort to inject secrets sensibly. Yes, "the alternatives come with their own risks", but those risks are where the "cost-benefit analysis" equals "not worth the effort".
- p-e-w 3y ago
  > If your system relies on secrets being present in repos, it is a poorly designed system. Repos are databases that track changes in files, nothing more and nothing less. There are millions of repos on GitHub that don't contain application code. Some people put their entire home directory in a Git repository. Home directories almost always contain secrets. That doesn't mean it's a bad idea, it just means the repo isn't meant for the eyes of others. In other words, it's a private repository.
  - ghusto 3y ago
    People putting their entire home directory in a repo, on somebody else's shared computer and network, is going to end in a disaster scenario. This sounds like a bad idea.
3y ago

layer8 3y ago

This is true for everything in the cloud that isn’t end-to-end-encrypted. There is the saying that the cloud is just someone else's computer. But in addition, you also have no control over your data on that computer not being leaked elsewhere, or the computer ending up becoming someone different’s computer.

LinuxBender 3y ago
Adding to this my own pessimistic take is that once $big_company that acquired $useful_service decides they have obtained a sufficient level of corporate capture with paid businesses then they may decide to add limits to non-paid repository counts, sizes, pulls, pushes and tighten those limits each year until the non-paying members have moved on. Something similar may be slowly happening with Gmail based on ask hn submissions.

st3fan 3y ago

Getting a bit tired of the anti-GitHub threads here ...

ImJasonH 3y ago

You could also leak private repo names via GHCR: https://www.chainguard.dev/unchained/ghcr-private-repos-sometimes-werent https://www.chainguard.dev/unchained/ghcr-private-repos-some...

snowwrestler 3y ago

It’s right there in the name: GitHub private repos are “private,” not “secure” or “secret”. That name was chosen purposefully. When we moved our code from privately hosted SVN to GitHub 10+ years ago, folks at GitHub were quite clear that we should not trust private repos with secrets like private keys or passwords. So why have private repos at all? It is to allow control of collaborators. GitHub originally allowed everyone to see and fork every repo—a true open source approach. Private repos were added basically so corporate codebase managers would not have to spend tons of time rejecting random PRs and responding to random people. For a LOT of private software projects, everything secret can be easily stored in the database, environments, or a dedicated tool for managing secrets. GitHub private repos are great for those. If my entire codebase is a highly sensitive trade secret, I would not use GitHub private repos, personally.

willis936 3y ago
When I have an expectation of privacy, say in a fenced backyard, I don't expect to be seen. It doesn't matter if it's a human or a robot commanded by a human that does the looking, it's still a violation of privacy. It's not really "private" if the backyard has a one way glass pane on one wall. Github should not be calling repos private if they're not.
- snowwrestler 3y ago
  This is one of many instances where developers should not make shallow assumptions without reading the documentation. GitHub is very clear about how private repos should be used. Edit to add: would you store your bank password on a piece of paper you always leave sitting in your backyard? Think carefully about your metaphor.
  - owl57 3y ago
    > would you store your bank password on a piece of paper you always leave sitting in your backyard? Not that I'm advocating for checking passwords in git repos, Github or otherwise, but you metaphor depends on how rich you are and how bad is your neighborhood, I would think. My mom has definitely done exactly that — her medical conditions make it very hard to use bank apps without full daylight and also hard to bring the notebook home after every use — and I don't think that her password management choices, however frowned upon they may be here, have ever been a serious problem in practice.

franciscop 3y ago

I'm not sure why in the "Easy Fixes" it doesn't say the thing that is mentioned in the next point "stop putting sensitive data into private repositories". That should _also_ go in "Easy Fixes" IMHO, use environment variables, vaults, etc to not store your keys into your code.

satya71 3y ago

I wonder, how do Azure DevOps and AWS CodeCommit compare in this regard?

trentnix 3y ago

A company I worked for a few years ago sold a source code license and I was tasked with sharing the source, sans history and only at a specific point in time, to the purchaser’s GitHub organization . I was told it had to be done as quickly as possible, and that we might do it a few more times so figure out a way to make it repeatable. I wrote a Bash script to copy our ~25 repositories to a new organization using the GitHub REST API. Turns out that the default when creating a new repository was to make it public. Within a few minutes I was getting emails by third party services who had downloaded and parsed our code, discovering a long abandoned AWS credential hard-coded in an older codebase. (I think GitHub now does this for you) I was lucky that nothing important was actually compromised (we had done a decent job of keeping secrets out of our repos and the one exception was an account that was long gone), but it was an eye-opening experience. If these services had found and downloaded our code in minutes you can assume any repo made public, even for a few moments, has been downloaded and cataloged by a potential bad actor.

ec109685 3y ago
Any idea how successful the company who you sold sold the license to was with code base? Trying to develop on a code base without history and a one time snapshot seems quite hard.

redm 3y ago

I see a lot of deserved distrust of MS, but I thought GitHub was a operated as a separate unit under Microsoft? [1] I expected that Co-Pilot was an initiative of that leadership team and training the LLM is what's likely reading certain repositories? On a side note, I'm trying to imagine what "sensitive" code would be read, incorporated into an LLM such as Co-pilot, and somehow have any meaningful impact to me once incorporated? [1] https://github.com/about/leadership https://github.com/about/leadership

2devnull 3y ago

People like to put api keys in their repos and it’s hard to make them grasp why they shouldn’t. I guess some lessons have to be learned the hard way.

rvz 3y ago

Unsurprising. Private repositories are not 'private' [0] and GitHub, Microsoft and OpenAI are thanking you for allowing them to train their AI on your so-called 'private' code. [1] This is essentially a shameless admission and they not even hiding it any more: Private repository data is scanned by machine and never read by GitHub staff. Human eyes will never see the contents of your private repositories, except as described in our Terms of Service. This is another great reason to self host instead of using services like GitHub openly reading your private code, which I have been saying for years. [0] https://news.ycombinator.com/item?id=34258431 https://news.ycombinator.com/item?id=34258431 [1] https://docs.github.com/en/get-started/privacy-on-github/about-githubs-use-of-your-data#privacy-and-data-sharing https://docs.github.com/en/get-started/privacy-on-github/abo...

sebzim4500 3y ago

>We cram our secrets into git, then shove it off to the most expansive code forge in the history of humanity Do we? Putting secrets in git is against policy almost everywhere.

jononomo 3y ago

Have you guys considered the machines are reading your private repos all the time? Forget about LLMs -- what about your text editor? Also, it is likely that your browser has access to your private repo when you use your browser to browse your private repo.

themoonisachees 3y ago

As the person on call for exactly that sort of problem, it saddens me that people still push secrets to git at all. Every gitops platform on the planet supports not pushing secrets to version control. There is no valid reason to have credentials inside a git repo ever. If the repo itself is what's secret, you should have known the risk when you pushed it to someone else's server. There is no excuse when gitlab is free and a docker container away.

GitHub Private Repos Considered Private-­Ish

GitHub Private Repos Considered Private-­Ish

GitHub Private Repos Considered Private-Ish

GitHub Private Repos Considered Private-Ish