8 ms·

GitHub major issues - repos have "lost" commits and site is erroring

119 points by andrewljohnson 16y ago

spooneybarger 16y ago
It isnt just private repos. I'm seeing it on various public repos on github.
- cilantro 16y ago
  Someone had shared a public repo with me on Monday. It started 404ing the next day and I had assumed he took it down. I realize now that it must have been this issue. I just checked for it and it is back.
mcmatterson 16y ago
In future, they keep a very up-to-date status page at http://status.github.com/ http://status.github.com/
- andrewljohnson 16y ago
  There is nothing useful on that page. The error is in full effect right now.
  - mcmatterson 16y ago
    Oops, my bad. I mistakenly thought the 'db error' notice was for today. I retract the 'very' in my original parent comment, replacing it with 'usually'.
rabidsnail 16y ago
Is it just me or has guthub's performance and reliability been getting progressively worse over the past month or two? Even putting this issue aside, and the outages, pushing to a private repo takes _minutes_ now, and file lists in the web interface often don't load without a page refresh.
- windsurfer 16y ago
  At the price and size of Linode instances, it's cheaper and I feel better knowing I'm in control when I run Redmine+Gitosis. When I had my github account and something went wrong, I couldn't just ssh in and fix it. All I could do is wait. Now, for some people, I can see how waiting is an easier solution. I personally just don't like it. (of course, this is reffering to private repos. public repos are still king on github)
  - SpiralLab 16y ago
    A major selling point of Git is the distributed part. You are allowed (even encouraged) to have more than one "source of truth". For ~7$/mo (including admin. and cheaper than Linode), GitHub is just another place to have a hosted version of your repo, with a nice UI and social features. Shit happens, servers go down, that's why you also have a remote repo hosted on Linode, and X, and Y too.
    - saurik 16y ago
      This is an interesting theory, but it isn't how git's client actually works. If the repository exported a list of "mirrors" to the client that it then stored and was willing to use, that would be awesome, but otherwise you have a million people out there who are now just getting error messages when they do "git pull" and the only fix is for them to go back to your web page and try to get information on what is happening and where else they can switch their origin. Meanwhile, if you do your own hosting you can just update your DNS to point to another box and no one is the wiser. They key problem, frankly, is that GitHub conflates two entirely unrelated things: a nice UI and social features, and a hosted version of your repo. I love the idea of outsourcing a nice UI and having cool social features, and /maybe/ to make those features work they need to have a mirror of my repository (I'm not convinced), but when people go to pull it the URL listed should be the actual upstream "I own the DNS on this and feel I can make this stable in the long term", not the GitHub mirror.
      - SpiralLab 16y ago
deleted 16y ago
[deleted]
hsuma 16y ago
I'm still on the fence about that whole cloud thing, personally.
- citricsquid 16y ago
  is reliability the price of convenience, or the other way around? edit: that makes no sense
- jameskilton 16y ago
  Not sure how this is relevant. Github isn't hosted on any cloud. http://www.anchor.com.au/blog/2009/09/github-designing-success/ http://www.anchor.com.au/blog/2009/09/github-designing-succe...
  - baddox 16y ago
    Well, every resource is locally on some physical machine somewhere. I think his point is that Github is often treated by its users as a "local" resource, in that it takes the place of what would otherwise be a local server running a Git front end or repository. Now, Github probably wouldn't encourage total reliance on their service. Git is fundamentally distributed, so this doesn't have to be a problem. Still, the Github service is an integral part of many developers' workflow, and I do agree with the above commenter that I'm not sold on the idea of trusting remote servers with integral steps in my workflow.
    - dedward 16y ago
      This echoes my thoughts all along. For open-projects, it's a FANTASTIC platform... no doubt about it. For a corporate closed project, I honestly would not use github - not because of any feature or lack of them, but because it's a hosted service. I want control over where my code sits, period - if only for the same reason as most hosted services - a subpoena or similar court order can be served against the provider with no notification to the owner of the data. This is the #1 reason one should be careful basing one's business on hosted services.
- __david__ 16y ago
jasonervin 16y ago
Not good.
jasonervin 16y ago
Not good.
ericmsimons 16y ago
I hope this has nothing to do with the people who hacked sourceforge..
- jrockway 16y ago
  Git has full cryptographic history (and data) authentication, so any changes to the history would be easily detectable. And you can sign known-good commits, so that even if you've never pulled before, you can still verify the part of the history that's been signed. SVN and CVS are missing this key feature, which is why the sf.net hacking is scary.
  - ericmsimons 16y ago
    I'm a Git n00b, so thanks for pointing that out! No wonder git has killed off SVN
    - oomkiller 16y ago
      It hasn't yet, but it is making great headway, and I for one can't wait!
  - saurik 16y ago
    Git's hashing detects data corruption to an existing user, but to someone who hasn't downloaded the repository yet and is getting it for the first time you really have to remember that SHA1 is currently considered "broken". It is increasingly feasible for someone to generate a collision to an existing file, allowing them to forge a commit. This has been discussed on the git mailing list, for the record, and the response is generally "don't care, we don't claim this to be a security mechanism". This is even a problem if you use signed tags, as you are only signing the result of a broken hash function.
ciaogiorno 16y ago
Why can't we have a web interface for GitHub? I should be push a new release right from my browser.
wizard_2 16y ago
Just posted to the support page. "Sorry guys, there was a Javascript error displaying an error when there was none. We've deployed a fix for this and everything should be good to go."
- andrewljohnson 16y ago
  This seems to be fixed now - for my repo at least. The GitHub guy saying it was a "JavaScript display error" is incorrect though. I was pulling a repo and getting an old version. EDIT: I'm not getting the javaScript display error, but I'm still missing my commit, so I guess these errors are unrelated :/
  - mojombo 16y ago
    Did you file a support ticket for this? It sounds like an unrelated problem. We can investigate if you let us know what repo is causing the problem. Visit http://support.github.com http://support.github.com or email support@github.com.
    - andrewljohnson 16y ago
      I posted the name of the repo to the thread, and someone else on the thread is confirming the same problem.
      - andrewljohnson 16y ago
        It turns out that GitHub didn't lose any of my commits. All future readers should ignore my comments on this.
  - andrewljohnson 16y ago
    This was my fault. GitHub was never messed up.
- mfukar 16y ago
  For what it's worth, I was pulling & pushing just fine while the error was displayed on my repositories, but other people reported commits lost or mangled. Maybe there was something more to the Javascript display error?
deleted 16y ago
[deleted]
btipling 16y ago
Well luckily you have the entire repository where ever it exists on your dev machines. So I don't expect there would be any data loss. In addition all the commits, tags, trees and files are hashed. The content is hashed, if there is a problem you'd know about it.
antirez 16y ago
Suggestion for creating a startup: For private repositories gitolite is really awesome, but I think there should be a company about it. What I mean is that github or some other company should offer hosted gitolite instances, in the following form: You pay something like price_of_small_linode_instace+price_of_service, to get: A virtual machine with gitolite up and running, with a good web interface and online documentation. I think that the web interface part is the piece this company should not opensource. With this web interface you can create repositories and users, and modify the gitolite settings in general. A backup service, in some other different VM and network, so that if something goes wrong with your gitolite VM you can ask a free (and automated) restore, possibly with hour granularity for the latest 24 hours, and less granularity for bigger periods and so forth. Note that this gives the user virtually infinite repositories... if they don't upload movies to their instance, as there is a lot of space even in a small linode instance for git repositories. This way this is a "pure product" as there is no need to repair or install manually virtual machines, and the company can focus into evolving the offering, developing a better version of gitolite, of the web interface, and so forth. p.s. I mentioned linode but can well be ec2 or whatever is better suited for this stuff.
- icefox 16y ago
  What do people really think of this idea? Would you buy it? I have written a web interface for managing git repos called GitHaven (my personal home install is here: http://git.meyerhome.net:8080/ http://git.meyerhome.net:8080/). It has the usual goodies, private/public repos etc. I have some friend's companies that run it in the cloud already. Do you think it would actually make money i.e. Would you be willing to join me in a startup on it (in boston)? Would it pass the y-combinator test?
  - tptacek 16y ago
    Probably not. Github is on a path to become the market leader, has a critical mass of customers, and has a business model that harnesses those customers to benefit from network effects. This idea is akin to building a business offering private white-label Stack Overflow clones. Key point: the actual software is not what makes Stack Overflow and Github valuable. Simpler reaction: Github:FI is already priced at a rounding error for the kinds of companies from which most of the revenue in this space will originate.

119 points by andrewljohnson 16y ago

spooneybarger 16y ago

It isnt just private repos. I'm seeing it on various public repos on github.

cilantro 16y ago
Someone had shared a public repo with me on Monday. It started 404ing the next day and I had assumed he took it down. I realize now that it must have been this issue. I just checked for it and it is back.

mcmatterson 16y ago

In future, they keep a very up-to-date status page at http://status.github.com/ http://status.github.com/

andrewljohnson 16y ago
There is nothing useful on that page. The error is in full effect right now.
- mcmatterson 16y ago
  Oops, my bad. I mistakenly thought the 'db error' notice was for today. I retract the 'very' in my original parent comment, replacing it with 'usually'.

rabidsnail 16y ago

Is it just me or has guthub's performance and reliability been getting progressively worse over the past month or two? Even putting this issue aside, and the outages, pushing to a private repo takes _minutes_ now, and file lists in the web interface often don't load without a page refresh.

windsurfer 16y ago
At the price and size of Linode instances, it's cheaper and I feel better knowing I'm in control when I run Redmine+Gitosis. When I had my github account and something went wrong, I couldn't just ssh in and fix it. All I could do is wait. Now, for some people, I can see how waiting is an easier solution. I personally just don't like it. (of course, this is reffering to private repos. public repos are still king on github)
- SpiralLab 16y ago
  A major selling point of Git is the distributed part. You are allowed (even encouraged) to have more than one "source of truth". For ~7$/mo (including admin. and cheaper than Linode), GitHub is just another place to have a hosted version of your repo, with a nice UI and social features. Shit happens, servers go down, that's why you also have a remote repo hosted on Linode, and X, and Y too.
  - saurik 16y ago
    This is an interesting theory, but it isn't how git's client actually works. If the repository exported a list of "mirrors" to the client that it then stored and was willing to use, that would be awesome, but otherwise you have a million people out there who are now just getting error messages when they do "git pull" and the only fix is for them to go back to your web page and try to get information on what is happening and where else they can switch their origin. Meanwhile, if you do your own hosting you can just update your DNS to point to another box and no one is the wiser. They key problem, frankly, is that GitHub conflates two entirely unrelated things: a nice UI and social features, and a hosted version of your repo. I love the idea of outsourcing a nice UI and having cool social features, and /maybe/ to make those features work they need to have a mirror of my repository (I'm not convinced), but when people go to pull it the URL listed should be the actual upstream "I own the DNS on this and feel I can make this stable in the long term", not the GitHub mirror.
    - SpiralLab 16y ago

deleted 16y ago

[deleted]

hsuma 16y ago

I'm still on the fence about that whole cloud thing, personally.

citricsquid 16y ago
is reliability the price of convenience, or the other way around? edit: that makes no sense
jameskilton 16y ago
Not sure how this is relevant. Github isn't hosted on any cloud. http://www.anchor.com.au/blog/2009/09/github-designing-success/ http://www.anchor.com.au/blog/2009/09/github-designing-succe...
- baddox 16y ago
  Well, every resource is locally on some physical machine somewhere. I think his point is that Github is often treated by its users as a "local" resource, in that it takes the place of what would otherwise be a local server running a Git front end or repository. Now, Github probably wouldn't encourage total reliance on their service. Git is fundamentally distributed, so this doesn't have to be a problem. Still, the Github service is an integral part of many developers' workflow, and I do agree with the above commenter that I'm not sold on the idea of trusting remote servers with integral steps in my workflow.
  - dedward 16y ago
    This echoes my thoughts all along. For open-projects, it's a FANTASTIC platform... no doubt about it. For a corporate closed project, I honestly would not use github - not because of any feature or lack of them, but because it's a hosted service. I want control over where my code sits, period - if only for the same reason as most hosted services - a subpoena or similar court order can be served against the provider with no notification to the owner of the data. This is the #1 reason one should be careful basing one's business on hosted services.
__david__ 16y ago

jasonervin 16y ago

Not good.

jasonervin 16y ago

Not good.

ericmsimons 16y ago

I hope this has nothing to do with the people who hacked sourceforge..

jrockway 16y ago
Git has full cryptographic history (and data) authentication, so any changes to the history would be easily detectable. And you can sign known-good commits, so that even if you've never pulled before, you can still verify the part of the history that's been signed. SVN and CVS are missing this key feature, which is why the sf.net hacking is scary.
- ericmsimons 16y ago
  I'm a Git n00b, so thanks for pointing that out! No wonder git has killed off SVN
  - oomkiller 16y ago
    It hasn't yet, but it is making great headway, and I for one can't wait!
- saurik 16y ago
  Git's hashing detects data corruption to an existing user, but to someone who hasn't downloaded the repository yet and is getting it for the first time you really have to remember that SHA1 is currently considered "broken". It is increasingly feasible for someone to generate a collision to an existing file, allowing them to forge a commit. This has been discussed on the git mailing list, for the record, and the response is generally "don't care, we don't claim this to be a security mechanism". This is even a problem if you use signed tags, as you are only signing the result of a broken hash function.

ciaogiorno 16y ago

Why can't we have a web interface for GitHub? I should be push a new release right from my browser.

wizard_2 16y ago

Just posted to the support page. "Sorry guys, there was a Javascript error displaying an error when there was none. We've deployed a fix for this and everything should be good to go."

andrewljohnson 16y ago
This seems to be fixed now - for my repo at least. The GitHub guy saying it was a "JavaScript display error" is incorrect though. I was pulling a repo and getting an old version. EDIT: I'm not getting the javaScript display error, but I'm still missing my commit, so I guess these errors are unrelated :/
- mojombo 16y ago
  Did you file a support ticket for this? It sounds like an unrelated problem. We can investigate if you let us know what repo is causing the problem. Visit http://support.github.com http://support.github.com or email support@github.com.
  - andrewljohnson 16y ago
    I posted the name of the repo to the thread, and someone else on the thread is confirming the same problem.
    - andrewljohnson 16y ago
      It turns out that GitHub didn't lose any of my commits. All future readers should ignore my comments on this.
- andrewljohnson 16y ago
  This was my fault. GitHub was never messed up.
mfukar 16y ago
For what it's worth, I was pulling & pushing just fine while the error was displayed on my repositories, but other people reported commits lost or mangled. Maybe there was something more to the Javascript display error?

deleted 16y ago

[deleted]

btipling 16y ago

Well luckily you have the entire repository where ever it exists on your dev machines. So I don't expect there would be any data loss. In addition all the commits, tags, trees and files are hashed. The content is hashed, if there is a problem you'd know about it.

antirez 16y ago

Suggestion for creating a startup: For private repositories gitolite is really awesome, but I think there should be a company about it. What I mean is that github or some other company should offer hosted gitolite instances, in the following form: You pay something like price_of_small_linode_instace+price_of_service, to get: A virtual machine with gitolite up and running, with a good web interface and online documentation. I think that the web interface part is the piece this company should not opensource. With this web interface you can create repositories and users, and modify the gitolite settings in general. A backup service, in some other different VM and network, so that if something goes wrong with your gitolite VM you can ask a free (and automated) restore, possibly with hour granularity for the latest 24 hours, and less granularity for bigger periods and so forth. Note that this gives the user virtually infinite repositories... if they don't upload movies to their instance, as there is a lot of space even in a small linode instance for git repositories. This way this is a "pure product" as there is no need to repair or install manually virtual machines, and the company can focus into evolving the offering, developing a better version of gitolite, of the web interface, and so forth. p.s. I mentioned linode but can well be ec2 or whatever is better suited for this stuff.

icefox 16y ago
What do people really think of this idea? Would you buy it? I have written a web interface for managing git repos called GitHaven (my personal home install is here: http://git.meyerhome.net:8080/ http://git.meyerhome.net:8080/). It has the usual goodies, private/public repos etc. I have some friend's companies that run it in the cloud already. Do you think it would actually make money i.e. Would you be willing to join me in a startup on it (in boston)? Would it pass the y-combinator test?
- tptacek 16y ago
  Probably not. Github is on a path to become the market leader, has a critical mass of customers, and has a business model that harnesses those customers to benefit from network effects. This idea is akin to building a business offering private white-label Stack Overflow clones. Key point: the actual software is not what makes Stack Overflow and Github valuable. Simpler reaction: Github:FI is already priced at a rounding error for the kinds of companies from which most of the revenue in this space will originate.