10 ms·

GitHub waited 3 months to notify about potential compromise

217 points by zwass 4y ago

It seems GitHub became aware of the issue around March 2 (or before, since the fix was released on March 2), and waited until June 16 to disclose the problem. See full text below: Hi zwass, We're writing to let you know that between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC, due to a bug, GitHub Apps were able to generate new scoped installation tokens with elevated permissions. You are an owner of an organization on GitHub with GitHub Apps installed that generated at least one new token during this time period. While we do not have evidence that this bug was maliciously exploited, with our available data, we are not able to determine if a token was generated with elevated permissions. User privacy and security are essential for maintaining trust, and we want to remain as transparent as possible about events like these. GitHub itself did not experience a compromise or data breach that either created or resulted from this event. Read on for more information. * What happened? * GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour. GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC. GitHub Apps generate scoped installation tokens based on the scopes and permissions granted when the GitHub App is installed into a user account or organization. For example, if a GitHub App requests and is granted read permission to issues during installation, the scoped installation token the App generates to function would have `issues:read` permission. This bug would potentially allow the above installation to generate a token with `issues:write`, an elevation of permission from the granted `issues:read` permission. The bug did not allow a GitHub App to generate a token with additional scopes that were not already granted, such as `discussions:read` in the above example. The bug did not allow a GitHub App to access any repositories that the App did not already have access to. In order to exploit this bug, the GitHub App author would need to modify their app's code to request elevated permissions on generated tokens. * What information was involved? * The following GitHub Apps generated scoped installation tokens during the bug window for your organization(s). We are not able to determine if a token was generated with elevated permissions. Organization: GitHub Apps <redacted> * What GitHub is doing * GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited. We are notifying all organization and user account owners that had GitHub Apps installed and had a scoped installation token generated during the bug window so that they can stay informed and perform their own audit of their installed GitHub Apps. As a followup to this investigation, GitHub is looking at ways to improve our logging to enable more in-depth analysis on scoped token generation and GitHub App permissions in the future. * What was the potential impact? * Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique. * What you can do * While we have updated our systems to resolve this bug and no action is required on your end, we do recommend you review your installed GitHub Apps. You can use the following guidance for assessing GitHub Apps, their permissions, and their access to your private organization repositories: https://docs.github.com/en/organizations/keeping-your-organization-secure

bearjaws 4y ago
I am really curious how SecOps works at GitHub. Why not after remediation inform users of the flaw and potential impact. Then follow up with detailed impact. Instead we get this 3 months later all they can say is "Some of your apps refreshed their tokens during a 5 day period" which is not news... This is also the second time this year there has been significant delay in communication. Granted those involved other third parties so who knows where the delay lived.
- mistrial9 4y ago
  they might assume that the account holder are a likely perpetrator, which might be true but also enables their intermediation of the context and control of the sequence of communication.
- s09dfhks 4y ago
  I would assume the lawyers have to get involved first to write up some document proving that github is not liable for what users do with their app tokens. CYA, then tell the public
lambada 4y ago
Given it existed for 5 days and you’re only now finding out about it, it sounds to me like it was perhaps a bug that was fixed without realising the full impact of it, or perhaps without realising it made it to production; and only an audit that happened later caught it. Not ideal by any means. I’d be curious to know if my theory is correct or not.
- zwass 4y ago
  Their statements indicate they were aware and investigating. My frustration is that they didn't give users the opportunity to do their own timely investigation. > GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour. > GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC. > GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.

217 points by zwass 4y ago

bearjaws 4y ago

I am really curious how SecOps works at GitHub. Why not after remediation inform users of the flaw and potential impact. Then follow up with detailed impact. Instead we get this 3 months later all they can say is "Some of your apps refreshed their tokens during a 5 day period" which is not news... This is also the second time this year there has been significant delay in communication. Granted those involved other third parties so who knows where the delay lived.

mistrial9 4y ago
they might assume that the account holder are a likely perpetrator, which might be true but also enables their intermediation of the context and control of the sequence of communication.
s09dfhks 4y ago
I would assume the lawyers have to get involved first to write up some document proving that github is not liable for what users do with their app tokens. CYA, then tell the public

lambada 4y ago

Given it existed for 5 days and you’re only now finding out about it, it sounds to me like it was perhaps a bug that was fixed without realising the full impact of it, or perhaps without realising it made it to production; and only an audit that happened later caught it. Not ideal by any means. I’d be curious to know if my theory is correct or not.

zwass 4y ago
Their statements indicate they were aware and investigating. My frustration is that they didn't give users the opportunity to do their own timely investigation. > GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour. > GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC. > GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.