7 ms·
Deploying at GitHub
- alexchamberlain 14y agoI know this is Github, but... Rather than use the Github API, wouldn't it be more efficient to interact with Git directly? Libgit2 maybe?
- badboy 14y agoGithub itself is a Github project, therefore using anything other than the API would be some code duplication. Merging, Pull-Requests, Branches, Issues: all this is already covered when using the normal API. I'm sure it could be "more efficient" when having code explicitely for this purpose, but then again you have to maintain to different code bases which do the same.
- alexchamberlain 14y agoYou either maintain a codebase that calls out to the Github API or a codebase that calls out to Git? What's the difference?
- pbiggar 14y agoTheres a lot of things you can do in the API that you can't do in git. For example, pushes exist in the API, but don't really within git.
- andypants 14y ago> For example, pushes exist in the API, but don't really within git. `git push`?
- pbiggar 14y agoI was unclear, sorry. You can't tell what pushes have occurred in the past from a git repo, but you can with the GitHub API.
- alexchamberlain 14y agoI don't get it... The API is built on top of `git`!
- pbiggar 14y agoGitHub has lots of things that Git doesn't have. As well as recording pushes and making them available over the API, it records fork information, has concepts of users and organizations which dont exist in git, has pull requests, comments, and post-commit code review, an issue tracker, etc, etc, etc.
- technoweenie 14y agoDogfooding aside, the vast majority of the time is spent running tests and actually deploying the code. The time to hit the API to merge the commit is negligible in comparison. Also, Janky and Heaven are both tiny apps that don't necessarily have access to the file servers.
- alexchamberlain 14y agoBut they are hitting the API to get the OID of the master ref as well? Not having access to the file system is a fair excuse...
- technoweenie 14y agoThe merge API takes branches: http://developer.github.com/v3/repos/merging/ http://developer.github.com/v3/repos/merging/
- deleted 14y ago[deleted]
- wamatt 14y agoEncouraging to know this model scales to 100 employees at least. Purely out of intellectual interest, I wonder if a company the size of Google or Facebook could also ship in this way, or if the whole release manager/team is essential.
- xxpor 14y agoAmazon generally doesn't have release teams (except for the retail website). I can't find it right now, but I know someone gave a talk on Apollo, which is how Amazon does deployments.
- shabble 14y agoI think I know the thing you mean, but I can't find it either. I did find 'flippers' from Flickr[1] which sounds like a very similar "keep it all in master and turn stuff on and off" methodology. [1] http://code.flickr.com/blog/2009/12/02/flipping-out/ http://code.flickr.com/blog/2009/12/02/flipping-out/
- xxpor 14y agoHere we go: http://news.ycombinator.com/item?id=2971521 http://news.ycombinator.com/item?id=2971521
- bdonlan 14y agoI think you might be looking for this: http://www.youtube.com/watch?v=dxk8b9rSKOo http://www.youtube.com/watch?v=dxk8b9rSKOo
- jrockway 14y agoSarbanes-Oxley puts a big damper on production deployments at big companies. I don't fully understand it so I won't try to explain it. (I will complain though: the law says developers shouldn't have control over production systems. If that's a requirement, who's going to write the software?)
- 14y ago
- cameronh90 14y agoHow often does code go through security audits? Is every feature audited prior to deploying to live? GitHub is making money by selling private repositories which will often contain very sensitive code, so ensuring nobody can gain unauthorised access to them is presumably one of the top concerns. I'm interested in seeing how tight security requirements fit in with this almost continuous deployment strategy.
- notJim 14y agoWe do continuous deployment at Etsy[1] as well. Here is a talk from our head of application security (probably not his official title, fyi) about some of the stuff we do: http://www.slideshare.net/zanelackey/effective-approaches-to-web-application-security http://www.slideshare.net/zanelackey/effective-approaches-to.... [1]: http://www.etsy.com http://www.etsy.com, the world's marketplace for handmade and vintage goods.
- technoweenie 14y agoEvery commit is reviewed by at least 1 person. Depending on the feature, several people may chime in. I find that reviewing smaller diffs is much easier. We also use Team Mentions (@github/api, for example) liberally to get more eyeballs. We also have regular audits with external security firms.
- smg 14y agoHow do you deal with the github enterprise version of your software? Does it have a separate QA cycle? How often do you ship new releases of that? I am hoping that Github could shed more light on the how they ship an enterprise version along with the SAASy web version that we all know and love.
- technoweenie 14y agoI'm prodding the Enterprise team to blog about this :)
- calavera 14y agoYes, the testing/deployment cycle for Enterprise is totally different. We usually release a major version with new features every two/three months, and 2 or 3 minor versions with bug fixes in between. We always keep the version of github synchronized with master for development/testing, although we only release master directly in major releases. For minor releases we avoid to include major features from github to keep it as much stable as possible.
- tzaman 14y agoWOW. 175 deploys in one day?
- 46Bit 14y agoI'd love to hear the backstory behind this. Company hackday, or a day spent shipping a major set of new features?
- technoweenie 14y agoWe don't do "company hack days". If you feel like hacking on something, hack on it. We do have days where multiple people will be waiting in line waiting for their chance to deploy their tweak. That particular day consisted of staff deploys on multiple in-progress branches, some performance tuning, bug fixes, etc. Nothing crazy. I'm also quite sure the number counts deploys across all of our applications. For instance, deploying a change to github-services counts as two, since I have to deploy changes to GitHub.com also.
- 46Bit 14y agoThanks for this, enjoy hearing about Github as a company. > I'm also quite sure the number counts deploys across all of our applications. For instance, deploying a change to github-services counts as two, since I have to deploy changes to GitHub.com also. That might explain a lot. Still a lot of deploys, but a more sane count :-)
- rsanheim 14y agoWe did have a pretty amazing week right after the summit, where everyone was on fire to ship things and there were a lot of people "in line" to get things deployed. It was pretty awesome, actually, seeing so many things land within a week of the whole team gathering and discussing the future.
- gbin 14y agoI might be wrong but for me this is almost a [Hack] -> [Prod] methodology... Roll back in 30 seconds, cool but how do you manage data / schema migrations ? You have a snapshot also to rollback any data corruption the last hacking session could have introduced ?
- technoweenie 14y agoData migrations are done carefully with the Large Hadron Migrator Ruby gem: https://github.com/soundcloud/large-hadron-migrator https://github.com/soundcloud/large-hadron-migrator. Facebook has a similar tool: https://www.facebook.com/note.php?note_id=430801045932 https://www.facebook.com/note.php?note_id=430801045932
- trustfundbaby 14y agoInteresting ... what's your QA process? Do you have a staging environment where you try stuff out (looking for bugs) before pushing to production?
- psadauskas 14y agoWe have a staging environment, but its really only used for really big changes that might need to be experimented with before being deployed. We can also deploy a branch to a single front end to observe how it behaves with a subset of the traffic, and roll it back quickly if needed. Also, most large user-facing features are released as "staff-only" first, so we as GitHub users are able to play around with it for a few days or weeks before enabling it for everyone.
- vrish88 14y agoHow do you guys release features as "staff-only"? Do you have some internal tool that manages that?
- rtomayko 14y agoNo. There's simple conventions for adding feature flags (user.some_feature_enabled?). Features are enabled and disabled by changing the code and deploying. This works because deploying new code is fast.
- technoweenie 14y agoWe do use Rollout (https://github.com/jamesgolick/rollout https://github.com/jamesgolick/rollout) once in awhile. Most of the time, we like having the history of flipped feature flags in the Git code though.
- amccloud 14y agoIf GitHub uses GitHub to deploy GitHub, what happens when GitHub goes down?
- johns 14y agoThis is a guess, but I'm assuming they use the same software running in an independent environment just for them.
- technoweenie 14y agoHeaven (our deployment tool) does deploy GitHub.com directly from the file servers. But, most of our infrastructure directly relies on GitHub too (such as the Merge API from the blog post, service hooks, and a bunch of OAuth minapps).
- pbiggar 14y agoIf you're looking for a less complex model of this, you should try our Continuous Integration and deployment service: https://circleci.com https://circleci.com. Over time, we'll be providing the sort of complexity that GitHub provides here, now we do about 70% of it.
- nthj 14y agoYou look awesome. I think I'll be setting this up on Friday. Do you have webhooks for pass/fail on the horizon? Or better yet, straight up git push with ssh key support when all the tests pass?
- bostonvaulter2 14y agoLooks interesting. Is it free? It seems like it probably is but I'm not sure. Also when it leaves beta how much will it cost? I don't want to end up depending on something I can't afford.
- pbiggar 14y ago14 day free trial, then $19 for 1 project, $49 for 10 projects, and $149 to run tests twice as fast. Feedback on anything, including pricing, very welcome.
- alexchamberlain 14y agoper... build? day? week? month? year? lifetime?
- sonnenkiste 14y agoInteresting, I saw something like this by looking at the https://github.com/mozilla/pdf.js/ https://github.com/mozilla/pdf.js/ project. Take a look at the closed pull requests. They have some bots listening for commands in comments and do stuff like unit testing and previewing. The result gets posted as comment from the responsible bot. Another one is checking master branch for changes and automatically builds and pushes at gh-pages. Seems to work very well, but don't know how they build/did it.
- arturadib 14y agoHi Artur from PDF.js here. We built Bot.io to handle the workflow you mentioned: http://github.com/arturadib/botio http://github.com/arturadib/botio It's written in Node and is trivial to install/use in any Github project.