3 ms·

We received an email from Github yesterday informing us that one of our repositories had been accessed by a third party due to this issue. While it's not a fun

We received an email from Github yesterday informing us that one of our repositories had been accessed by a third party due to this issue. While it's not a fun notification to receive, it definitely made our general security paranoia feel justified – we're lucky that from the get-go we've held best practices around keeping secrets out of the codebase. Obviously we still dedicated time as a team to prune through our repository history with a fine-toothed comb for anything that could potentially be a vulnerability, as we take this very seriously. One of our engineers came up with a useful script to grab all unique lines from the history of the repository and sort them according to entropy. This helps to lift any access keys or passwords which may have been committed at any point to the top. I think this is a great example to illustrate the tough edges of security to less experienced engineers. Github will most likely never let something like this happen to you, but on the off-chance that they do it's great to be prepared. Additionally, the response from Github was very well received. No excuses, just a thorough explanation of what happened. I also can't help but mention that we're hiring, if you'd like to work at an organization that values security and data privacy very highly. :) usebutton.com/join-us

foota 10y ago
I'm curious, how did they calculate entropy? My first thought was to do something with Huffman encoding.
- jasonmoo 10y ago
  I wrote the script in question and actually used a simple shannon entropy value. (http://codereview.stackexchange.com/questions/868/calculating-entropy-of-a-string/909#909 http://codereview.stackexchange.com/questions/868/calculatin...). It worked well enough help rule out several problem spaces.
  - pavel_lishin 10y ago
    Would you mind posting the script? I'd love to run it against our codebase and see what it comes up with. It might be a fun thing to open source as part of a "I've inherited a project, what now?" toolkit that helps you decide what to fix.
    - jasonmoo 10y ago
      Sure. It's a simple tool but the concept could be augmented toward something like the scenario you described. https://gist.github.com/jasonmoo/06691c8fea09b62aa35235fc93ee31b6 https://gist.github.com/jasonmoo/06691c8fea09b62aa35235fc93e...
    - ubernostrum 10y ago
      IIRC Instagram released a plugin for Bandit (the OpenStack static analyzer for Python) that does this.
tw90210 10y ago
I got the other end of that email today, saying that an account in my organization had inadvertently downloaded private repos from another customer when fetching from one of our own. Fortunately for GitHub/that user, it was almost certainly our automated provisioning system so we never had any idea and whatever it was never made it anywhere interesting. The email was kind of funny though, part of it was effectively "if you have this data pretty please delete it without looking at it". I'm sure that's the best they can do, but it still made me chuckle.
dorianm 10y ago
I thought the idea was interesting so here is a little PoC in Ruby: require 'facets' lines = Dir['**/*.rb', '**/*.py', '**/*.cpp'].map { |f| File.read(f).lines.map(&:chars) }.inject(&:+) puts lines.sort_by(&:entropy).map(&:join).last(10).reverse Using: http://www.rubydoc.info/github/rubyworks/facets/Array%3Aentropy http://www.rubydoc.info/github/rubyworks/facets/Array%3Aentr...
- gcr 10y ago
  Is it easy to modify this script to run over all lines that have ever existed in the repository history? For example, could you pipe the output of `git log -p --all` through this and filter out all the commit hashes somehow?
  - dorianm 10y ago
    Yup, just use: lines = `git log -p --all`.lines.map(&:chars) So I found that `git grep /.+/ $(git rev-list --all)` is a better way to get the content of all the files: https://gist.github.com/Dorian/e1514535c3c5036cf327ce61eb34af89 https://gist.github.com/Dorian/e1514535c3c5036cf327ce61eb34a... But actually an hex number regexp might me far more accurate than the entropy (e.g.: secrets are often long hex numbers). I tried it and it yields interesting results: https://gist.github.com/28110f0b8105db11e8973d1d0be85259 https://gist.github.com/28110f0b8105db11e8973d1d0be85259
tonyarkles 10y ago
Of the very small number of repos affected, there are now two of us reporting that it affected us :). And I had the same approximate response: well, I don't keep secrets in the repository, so not that big of deal. I'd rather the source not get shared with the world, but shit happens and they owned up to it right away. If that source were valuable enough, I'd be hosting it on-prem with encrypted off-site backups.

foota 10y ago

I'm curious, how did they calculate entropy? My first thought was to do something with Huffman encoding.

jasonmoo 10y ago
I wrote the script in question and actually used a simple shannon entropy value. (http://codereview.stackexchange.com/questions/868/calculating-entropy-of-a-string/909#909 http://codereview.stackexchange.com/questions/868/calculatin...). It worked well enough help rule out several problem spaces.
- pavel_lishin 10y ago
  Would you mind posting the script? I'd love to run it against our codebase and see what it comes up with. It might be a fun thing to open source as part of a "I've inherited a project, what now?" toolkit that helps you decide what to fix.
  - jasonmoo 10y ago
    Sure. It's a simple tool but the concept could be augmented toward something like the scenario you described. https://gist.github.com/jasonmoo/06691c8fea09b62aa35235fc93ee31b6 https://gist.github.com/jasonmoo/06691c8fea09b62aa35235fc93e...
  - ubernostrum 10y ago
    IIRC Instagram released a plugin for Bandit (the OpenStack static analyzer for Python) that does this.

tw90210 10y ago

I got the other end of that email today, saying that an account in my organization had inadvertently downloaded private repos from another customer when fetching from one of our own. Fortunately for GitHub/that user, it was almost certainly our automated provisioning system so we never had any idea and whatever it was never made it anywhere interesting. The email was kind of funny though, part of it was effectively "if you have this data pretty please delete it without looking at it". I'm sure that's the best they can do, but it still made me chuckle.

dorianm 10y ago

I thought the idea was interesting so here is a little PoC in Ruby: require 'facets' lines = Dir['**/*.rb', '**/*.py', '**/*.cpp'].map { |f| File.read(f).lines.map(&:chars) }.inject(&:+) puts lines.sort_by(&:entropy).map(&:join).last(10).reverse Using: http://www.rubydoc.info/github/rubyworks/facets/Array%3Aentropy http://www.rubydoc.info/github/rubyworks/facets/Array%3Aentr...

gcr 10y ago
Is it easy to modify this script to run over all lines that have ever existed in the repository history? For example, could you pipe the output of `git log -p --all` through this and filter out all the commit hashes somehow?
- dorianm 10y ago
  Yup, just use: lines = `git log -p --all`.lines.map(&:chars) So I found that `git grep /.+/ $(git rev-list --all)` is a better way to get the content of all the files: https://gist.github.com/Dorian/e1514535c3c5036cf327ce61eb34af89 https://gist.github.com/Dorian/e1514535c3c5036cf327ce61eb34a... But actually an hex number regexp might me far more accurate than the entropy (e.g.: secrets are often long hex numbers). I tried it and it yields interesting results: https://gist.github.com/28110f0b8105db11e8973d1d0be85259 https://gist.github.com/28110f0b8105db11e8973d1d0be85259

tonyarkles 10y ago

Of the very small number of repos affected, there are now two of us reporting that it affected us :). And I had the same approximate response: well, I don't keep secrets in the repository, so not that big of deal. I'd rather the source not get shared with the world, but shit happens and they owned up to it right away. If that source were valuable enough, I'd be hosting it on-prem with encrypted off-site backups.