7 ms·

> We believe that no data has been lost, unless the [...] GitLab copy was the only one. One difference between how GitLab and GitHub run their infrastructure i

by avar 7y ago

> We believe that no data has been lost, unless the [...] GitLab copy was the only one. One difference between how GitLab and GitHub run their infrastructure is that GitLab doesn't keep reflogs, and uses git's default "gc" settings. As a result they won't have the data in question anymore in many cases[1]. Well, I don't 100% know that for sure, but it's the default configuration of their software, and I'm assuming they use like that themselves. Whereas GitHub does keep reflogs, and runs "git repack" with the "--keep-unreachable" option. They don't usually delete git data unless someone bothers to manually do it, and they usually have data to reconstruct repositories as they were at any given point in time. GitHub doesn't expose that to users in any way, although perhaps they'd take pity on some of their users after such an incident. This isn't a critique of GitLab, just trivia about the storage trade-offs different major Git hosting sites have made, which might be informative to some other people. I'm surprised no major Git hosting site has opted to provide such a "we have a snapshot of every version ever" feature. People would probably pay for it, you could even make them opt to pay for access to backups you kept already if they screwed things up :) 1. Well, maybe as disaster backups or something. But those are harder to access...

lostmyoldone 7y ago
Re GitHub keeping unreachable data, if I understand it right, isn't that GitHub painting a giant target on their back? Wouldn't that that imply every secret accidentally committed and then 'deleted' is still accessible, when one would expect it not to be? It's one thing to have your source code in the wild, but pairing it up with thought-to-be-deleted secrets would be an absolute disaster. Certainly one should not ever keep using a secret once it has escaped into a Git repo, but I'm sure it happens quite frequently.
- hnarn 7y ago
  > Wouldn't that that imply every secret accidentally committed and then 'deleted' is still accessible This should be a moot point because anyone (in IT) should realize that an accidentally committed secret is now 100% public for all eternity and needs to be rendered irrelevant to restore secure operations.
  - soulofmischief 7y ago
    Security isn't rendered in absolutes. We have to assume some sheepish new employee somewhere is scared of approaching management about a mistake they made committing a secret, so they reverse the commit and pretend nothing ever happened. We have to try and mitigate damage from lapses in communication and protocol like that.
    - iakh 7y ago
      In either case, the secret is already out whether the user wants to admit to it or not
      - soulofmischief 7y ago
        But in one case, damage is mitigated because the sys admins didn't assume everyone is infallible and strictly adheres to protocol.
        danharaj 7y ago
        The correct way to deal with fallibility in this situation is to make it feasible to change secrets when they leak, not pretend they weren't leaked.
        soulofmischief 7y ago
        That doesn't prevent someone from not following protocol.
        _II__II_ 7y ago
        It's not their job to prevent that.
        soulofmischief 7y ago
        It is a sys admin's job to mitigate damage from security leaks and to introduce hardened, fault-tolerant security paradigms.
- davidmurdoch 7y ago
  You can permanently delete data from github, but doing requires a bit of work and a message to customer service: https://help.github.com/en/articles/removing-sensitive-data-from-a-repository https://help.github.com/en/articles/removing-sensitive-data-...
- avar 7y ago
  If you ask nicely they'll run a one-off "gc expire" for you. It also requires an attacker to know at least the partial SHA-1 anyway. It's infeasible to start brute-forcing that without being banned for dDoSing them, and if you know what the SHA-1 is you probably had access to the data already. But yeah. It definitely creates security caveats peculiar to git, e.g. a hostile actor guessing that a force push in an IRC commit announcement clobbered secret data, and the accessing the old commit in the web UI.
- solatic 7y ago
  This is precisely why secret rotation mechanisms are essential. If you are regularly rotating your secrets, your window of vulnerability for an accidentally leaked secret reduces to the rotation window. With good automation, and in the context of secrets which don't need to be remembered or input by a human, your secrets should be rotating nearly constantly. Additionally, automation greatly reduces the risk of human intervention, which reduces the risk of a human writing secrets to files by hand, which reduces the risk of those secrets being committed to version control in the first place. Of course, automatic secret rotation is hard. Vault is a great help, but it can't be grafted onto everything. Good DevSecOps engineers are worth their weight in gold.
Boulth 7y ago
Just out of curiosity but how do you know about this inner workings of Github and GitLab?
- bigblind 7y ago
  I don't know about GitHub, but GitLab has an open source distribution, so you can assume they use the same configuration internally.
- ishitatsuyuki 7y ago
  You can access unreachable refs by means of Pull Request force pushes; even if you force push, you can still view the state at the original commit given that you have the hash of that. That plus some internal knowing of Git allows the author to guess how GitHub works this way. As on GitLab, it's open source, so you can easily check that.
- avar 7y ago
  I contribute to git, and I read the mailing list, where people a lot smarter than me comment about this sort of thing. I'm also on a team that runs an in-house enterprise GitLab instance for an S&P 100, so I have experience with it in that configuration, which I understand isn't different from what gitlab.com uses in this regard. None of this is secret or some sort of insider knowledge. If you know how "git gc" works you can trivially observe most of the behavior of these hosting sites from the outside. E.g. try pushing a commit and then view it at to git{hub,lab}.com/YOU/PROJECT/commit/SHA-1. Then "push --delete" the branch that references it. You'll find that you can still view it on both sites, even if when you clone the relevant repository you won't get that SHA-1. This is because it's expensive to do a reachability check before serving up the content, and the web frontends access the object store directly. Then if you e.g. keep making pushes sufficient to trigger a "gc --auto" and it's been longer than the relevant git "gc.Expire" time(s) you can deduce that the site uses something close to git's default "gc" semantics, or not. If you do this on GitHub.com you'll find you can access the data for longer than that, possibly "forever". Which is actually a thing relevant to data recovery in this case. If those impacted by this security incident have lost their data, but have some of the SHA-1s involved (e.g. because they were pasted in IRC) they might find they can still view that content on gitlab.com if they were to browse it in the commit/tree/blob view, and painfully recover it that way. They won't be able to clone it since neither site turns on uploadpack.allowAnySHA1InWant=true.
masklinn 7y ago
I'm surprised github runs regular git. I'd always assumed they were emulating it, especially with the lag we've observed between github-api and github-git at $DAYJOB (update repo 1 via api, update repo 2 via api, fetch repo 1 and repo 2 via git, we've had cases where the repo 2 update was visible but not the repo 1).
- pvorb 7y ago
  There was a talk about "Scaling Git at GitHub" [1] a few years ago at Git Merge. While it may no longer be accurate, it still gives some insight into how GitHub is run. [1]: https://youtube.com/watch?v=xK5yaWTt0R0 https://youtube.com/watch?v=xK5yaWTt0R0
- ak217 7y ago
  I think github runs https://github.com/libgit2/libgit2 https://github.com/libgit2/libgit2 which is not regular git.
the_duke 7y ago
They do keep disk snapshots for 2 weeks though (created twice a day). https://about.gitlab.com/handbook/engineering/infrastructure/design/disaster-recovery/#repository-data https://about.gitlab.com/handbook/engineering/infrastructure...
- sytse 7y ago
  Thanks for linking to that. I’m asking inside the company about the comment that the repositories are lost. I think it is a lot of work to restore individual repositories as opposed to restoring a disk, so maybe that is why we said that.
  - the_duke 7y ago
    Also any commits pushed between the last snapshot and the deletion would be lost too.
    - sytse 7y ago
      Correct
      - ndarwincorn 7y ago
        Hey, since I'm having trouble getting a response from support/security and I know you're active on here, can you get someone to respond to this ticket (related to OP)? I don't usually try to publicly escalate something like this but it's urgent and the ticket's gone 24 hours+ without a response. https://support.gitlab.com/hc/en-us/requests/120506 https://support.gitlab.com/hc/en-us/requests/120506
1f60c 7y ago
This tendency of Hacker News users to want to monetize everything is sickening.
- AsyncAwait 7y ago
  Yeah. But it's not surprising considering the site is being run by a VC fund.
  - bdcravens 7y ago
    Host 10m, add a feature that 10k need. Charge each customer enough to cover the 999 who aren’t willing to pay.
- antoineMoPa 7y ago
  I wonder if one could monetize this tendency.
  - soulofmischief 7y ago
    Monetyzer: The world is your oyster, it's about time you start collecting pearls.
    - edbaskerville 7y ago
      Maybe this comment was inspired by WellDeserved, a truly underappreciated but still important app: https://www.youtube.com/watch?v=WoK4_dQbfuU https://www.youtube.com/watch?v=WoK4_dQbfuU
      - soulofmischief 7y ago
        No, but that was incredible. Thanks for introducing me to Cultivated Wit.
- pjc50 7y ago
  It's kind of inherent on a website that's a side project of a venture capital firm. Besides, without monetisation, you're relying on the goodwill of a surprisingly small number of people. I like to call this "Postel decentralisation" - in the early days of the internet before IANA was the bureaucracy it is today, a lot of functions which people might naievely assume were decentralised were in fact done by hand by John Postel.
CSMastermind 7y ago
That's really fascinating. Thanks for sharing. As someone who uses Gitlab (and likes them a lot), I definitely like Github's approach more here.
saagarjha 7y ago
> GitHub doesn't expose that to users in any way Well, links to orphaned commits still work, and GitHub has recently started surfacing UI when you force push a branch.
mikeash 7y ago
That is a crazy-ass quote. “We believe that no data has been lost... well, except for the data we keep. But you weren’t actually relying on us to save any data, right?” I know, back up everything at least twice. But still, when somebody loses one of your copies, they don’t get to say “it’s cool, no data was lost, you have other copies, right?”
- gitlab-security 7y ago
  To provide additional context, on GitLab.com, we maintain two weeks' backups. The last time we restored a single project repo, it was a significant effort that utilized many hours of an SRE's time to complete.
  - insomniacity 7y ago
    "many hours" is too high. Regardless of this incident, if backups take too long to restore, you may as well not have them.
    - GL-pharrison 7y ago
      I agree, the longer it takes to recover the less valuable the backups are. However, in this context where we're restoring individual repo's (and all the metadata baggage involved) it's much different than a DB restore or other disaster recovery/prevention mechanisms. The per-repo restore time today is not where it should be. We're working to speed this up so we can help users recover and get back to a productive state quickly.
  - mikeash 7y ago
    What’s crazy isn’t the loss of data (that happens, and I don’t really expect most cloud services to be save me from explicit deletion requests) but rather framing it as no data being lost because your customers have it elsewhere.
  - zifnab06 7y ago
    Does support have the ability to restore keeparound refs? Internally (at least as of late 10 series), anything show in the UI (ie, merge requests) is also copied to refs/keeparound/sha1, which isn't presented to users.
gcb0 7y ago
you are assuming mistakenly deleted data. circumventing this is very trivial for an attacker.
jtl999 7y ago
> One difference between how GitLab and GitHub run their infrastructure is that GitLab doesn't keep reflogs, and uses git's default "gc" settings. Does this also apply to self hosted GitLab CE/EE? Also how does Gogs/Gitea handle this?
- avar 7y ago
  What I outlined applies to self-hosted GitLab, it just uses git's default settings, and I'm assuming gitlab.com does the same. Of course if you self-host you can simply change the defaults in /etc/gitconfig (which is in /opt/... if you're using the omnibus package).

by avar 7y ago

lostmyoldone 7y ago

Re GitHub keeping unreachable data, if I understand it right, isn't that GitHub painting a giant target on their back? Wouldn't that that imply every secret accidentally committed and then 'deleted' is still accessible, when one would expect it not to be? It's one thing to have your source code in the wild, but pairing it up with thought-to-be-deleted secrets would be an absolute disaster. Certainly one should not ever keep using a secret once it has escaped into a Git repo, but I'm sure it happens quite frequently.

hnarn 7y ago
> Wouldn't that that imply every secret accidentally committed and then 'deleted' is still accessible This should be a moot point because anyone (in IT) should realize that an accidentally committed secret is now 100% public for all eternity and needs to be rendered irrelevant to restore secure operations.
- soulofmischief 7y ago
  Security isn't rendered in absolutes. We have to assume some sheepish new employee somewhere is scared of approaching management about a mistake they made committing a secret, so they reverse the commit and pretend nothing ever happened. We have to try and mitigate damage from lapses in communication and protocol like that.
  - iakh 7y ago
    In either case, the secret is already out whether the user wants to admit to it or not
    - soulofmischief 7y ago
      But in one case, damage is mitigated because the sys admins didn't assume everyone is infallible and strictly adheres to protocol.
      - danharaj 7y ago
        The correct way to deal with fallibility in this situation is to make it feasible to change secrets when they leak, not pretend they weren't leaked.
        soulofmischief 7y ago
        That doesn't prevent someone from not following protocol.
        _II__II_ 7y ago
        It's not their job to prevent that.
        soulofmischief 7y ago
        It is a sys admin's job to mitigate damage from security leaks and to introduce hardened, fault-tolerant security paradigms.
davidmurdoch 7y ago
You can permanently delete data from github, but doing requires a bit of work and a message to customer service: https://help.github.com/en/articles/removing-sensitive-data-from-a-repository https://help.github.com/en/articles/removing-sensitive-data-...
avar 7y ago
If you ask nicely they'll run a one-off "gc expire" for you. It also requires an attacker to know at least the partial SHA-1 anyway. It's infeasible to start brute-forcing that without being banned for dDoSing them, and if you know what the SHA-1 is you probably had access to the data already. But yeah. It definitely creates security caveats peculiar to git, e.g. a hostile actor guessing that a force push in an IRC commit announcement clobbered secret data, and the accessing the old commit in the web UI.
solatic 7y ago
This is precisely why secret rotation mechanisms are essential. If you are regularly rotating your secrets, your window of vulnerability for an accidentally leaked secret reduces to the rotation window. With good automation, and in the context of secrets which don't need to be remembered or input by a human, your secrets should be rotating nearly constantly. Additionally, automation greatly reduces the risk of human intervention, which reduces the risk of a human writing secrets to files by hand, which reduces the risk of those secrets being committed to version control in the first place. Of course, automatic secret rotation is hard. Vault is a great help, but it can't be grafted onto everything. Good DevSecOps engineers are worth their weight in gold.

Boulth 7y ago

Just out of curiosity but how do you know about this inner workings of Github and GitLab?

bigblind 7y ago
I don't know about GitHub, but GitLab has an open source distribution, so you can assume they use the same configuration internally.
ishitatsuyuki 7y ago
You can access unreachable refs by means of Pull Request force pushes; even if you force push, you can still view the state at the original commit given that you have the hash of that. That plus some internal knowing of Git allows the author to guess how GitHub works this way. As on GitLab, it's open source, so you can easily check that.
avar 7y ago
I contribute to git, and I read the mailing list, where people a lot smarter than me comment about this sort of thing. I'm also on a team that runs an in-house enterprise GitLab instance for an S&P 100, so I have experience with it in that configuration, which I understand isn't different from what gitlab.com uses in this regard. None of this is secret or some sort of insider knowledge. If you know how "git gc" works you can trivially observe most of the behavior of these hosting sites from the outside. E.g. try pushing a commit and then view it at to git{hub,lab}.com/YOU/PROJECT/commit/SHA-1. Then "push --delete" the branch that references it. You'll find that you can still view it on both sites, even if when you clone the relevant repository you won't get that SHA-1. This is because it's expensive to do a reachability check before serving up the content, and the web frontends access the object store directly. Then if you e.g. keep making pushes sufficient to trigger a "gc --auto" and it's been longer than the relevant git "gc.Expire" time(s) you can deduce that the site uses something close to git's default "gc" semantics, or not. If you do this on GitHub.com you'll find you can access the data for longer than that, possibly "forever". Which is actually a thing relevant to data recovery in this case. If those impacted by this security incident have lost their data, but have some of the SHA-1s involved (e.g. because they were pasted in IRC) they might find they can still view that content on gitlab.com if they were to browse it in the commit/tree/blob view, and painfully recover it that way. They won't be able to clone it since neither site turns on uploadpack.allowAnySHA1InWant=true.

masklinn 7y ago

I'm surprised github runs regular git. I'd always assumed they were emulating it, especially with the lag we've observed between github-api and github-git at $DAYJOB (update repo 1 via api, update repo 2 via api, fetch repo 1 and repo 2 via git, we've had cases where the repo 2 update was visible but not the repo 1).

pvorb 7y ago
There was a talk about "Scaling Git at GitHub" [1] a few years ago at Git Merge. While it may no longer be accurate, it still gives some insight into how GitHub is run. [1]: https://youtube.com/watch?v=xK5yaWTt0R0 https://youtube.com/watch?v=xK5yaWTt0R0
ak217 7y ago
I think github runs https://github.com/libgit2/libgit2 https://github.com/libgit2/libgit2 which is not regular git.

the_duke 7y ago

They do keep disk snapshots for 2 weeks though (created twice a day). https://about.gitlab.com/handbook/engineering/infrastructure/design/disaster-recovery/#repository-data https://about.gitlab.com/handbook/engineering/infrastructure...

sytse 7y ago
Thanks for linking to that. I’m asking inside the company about the comment that the repositories are lost. I think it is a lot of work to restore individual repositories as opposed to restoring a disk, so maybe that is why we said that.
- the_duke 7y ago
  Also any commits pushed between the last snapshot and the deletion would be lost too.
  - sytse 7y ago
    Correct
    - ndarwincorn 7y ago
      Hey, since I'm having trouble getting a response from support/security and I know you're active on here, can you get someone to respond to this ticket (related to OP)? I don't usually try to publicly escalate something like this but it's urgent and the ticket's gone 24 hours+ without a response. https://support.gitlab.com/hc/en-us/requests/120506 https://support.gitlab.com/hc/en-us/requests/120506

1f60c 7y ago

This tendency of Hacker News users to want to monetize everything is sickening.

AsyncAwait 7y ago
Yeah. But it's not surprising considering the site is being run by a VC fund.
- bdcravens 7y ago
  Host 10m, add a feature that 10k need. Charge each customer enough to cover the 999 who aren’t willing to pay.
antoineMoPa 7y ago
I wonder if one could monetize this tendency.
- soulofmischief 7y ago
  Monetyzer: The world is your oyster, it's about time you start collecting pearls.
  - edbaskerville 7y ago
    Maybe this comment was inspired by WellDeserved, a truly underappreciated but still important app: https://www.youtube.com/watch?v=WoK4_dQbfuU https://www.youtube.com/watch?v=WoK4_dQbfuU
    - soulofmischief 7y ago
      No, but that was incredible. Thanks for introducing me to Cultivated Wit.
pjc50 7y ago
It's kind of inherent on a website that's a side project of a venture capital firm. Besides, without monetisation, you're relying on the goodwill of a surprisingly small number of people. I like to call this "Postel decentralisation" - in the early days of the internet before IANA was the bureaucracy it is today, a lot of functions which people might naievely assume were decentralised were in fact done by hand by John Postel.

CSMastermind 7y ago

That's really fascinating. Thanks for sharing. As someone who uses Gitlab (and likes them a lot), I definitely like Github's approach more here.

saagarjha 7y ago

> GitHub doesn't expose that to users in any way Well, links to orphaned commits still work, and GitHub has recently started surfacing UI when you force push a branch.

mikeash 7y ago

That is a crazy-ass quote. “We believe that no data has been lost... well, except for the data we keep. But you weren’t actually relying on us to save any data, right?” I know, back up everything at least twice. But still, when somebody loses one of your copies, they don’t get to say “it’s cool, no data was lost, you have other copies, right?”

gitlab-security 7y ago
To provide additional context, on GitLab.com, we maintain two weeks' backups. The last time we restored a single project repo, it was a significant effort that utilized many hours of an SRE's time to complete.
- insomniacity 7y ago
  "many hours" is too high. Regardless of this incident, if backups take too long to restore, you may as well not have them.
  - GL-pharrison 7y ago
    I agree, the longer it takes to recover the less valuable the backups are. However, in this context where we're restoring individual repo's (and all the metadata baggage involved) it's much different than a DB restore or other disaster recovery/prevention mechanisms. The per-repo restore time today is not where it should be. We're working to speed this up so we can help users recover and get back to a productive state quickly.
- mikeash 7y ago
  What’s crazy isn’t the loss of data (that happens, and I don’t really expect most cloud services to be save me from explicit deletion requests) but rather framing it as no data being lost because your customers have it elsewhere.
- zifnab06 7y ago
  Does support have the ability to restore keeparound refs? Internally (at least as of late 10 series), anything show in the UI (ie, merge requests) is also copied to refs/keeparound/sha1, which isn't presented to users.

gcb0 7y ago

you are assuming mistakenly deleted data. circumventing this is very trivial for an attacker.

jtl999 7y ago

> One difference between how GitLab and GitHub run their infrastructure is that GitLab doesn't keep reflogs, and uses git's default "gc" settings. Does this also apply to self hosted GitLab CE/EE? Also how does Gogs/Gitea handle this?

avar 7y ago
What I outlined applies to self-hosted GitLab, it just uses git's default settings, and I'm assuming gitlab.com does the same. Of course if you self-host you can simply change the defaults in /etc/gitconfig (which is in /opt/... if you're using the omnibus package).