15 ms·

Ten Years of Git: An Interview with Linus Torvalds

509 points by lclark 12y ago

worklogin 12y ago
Linus says of the Github platform - That's partly because of how the kernel is developed, but part of it was that the GitHub interfaces were actively encouraging bad behavior. Commits done on GitHub had bad commit messages etc, because the web interfaces at GitHub were actively encouraging bad behavior. They did fix some of that, so it probably works better, but it will never be appropriate for something like the Linux kernel. I haven't ever looked at the kernel workflow, nor have I ever heard systematic criticism of Github's methodology. Does anyone else have input on what Linus may mean by his opinion?
- merqumab 12y ago
  See this 2012 thread https://github.com/torvalds/linux/pull/17#issuecomment-5654674 https://github.com/torvalds/linux/pull/17#issuecomment-56546...
- rentnorove 12y ago
  This pull request: https://github.com/torvalds/linux/pull/17 https://github.com/torvalds/linux/pull/17
- gregkh 12y ago
  The kernel development model is documented here: http://lxr.free-electrons.com/source/Documentation/development-process/ http://lxr.free-electrons.com/source/Documentation/developme... Github doesn't scale at all for large projects. The kernel is averaging over 8 changes an hour, 24 hours a day. That rate of change can never be handled by doing pull requests and web site review. It only can work with email and review and scriptable processes.
  - quantumet 12y ago
    Eh, I mildly disagree on web site review. Perhaps not GitHub's pull request/review model, but big projects like Android and Chromium do all their review on web interfaces.
    - edejong 12y ago
      Using Gerrit, which is an automated mimick of Linus' development model. By the way, we've been succesfully using Gerrit on smaller-scale projects as well (around 8 change-requests / day) and don't even want to think about going back to pull-requests.
      - sunnyps 12y ago
        What you've said is essentially correct, but I'd like to make a minor correction: Chromium uses rietveld whereas Android (and some Chromium related projects) use gerrit. Rietveld and gerrit have similar workflows except that gerrit is more integrated with git (I think). Both are named after Gerrit Rietveld, a famous Dutch designer.
  - wyldfire 12y ago
    But don't "trusted lieutenants" (yourself included?) mitigate a lot of the challenge posed by that high rate of change?
    - luckydude 12y ago
      Greg didn't reply but I think the answer is a resounding yes. I wrote this up a bit back when the devs were using BitKeeper. Lemme go find that. OK, ignore the "I want the 1995 era web pages" look and peek at this: http://www.mcvoy.com/lm/bitmover/lm/talks/socal/ http://www.mcvoy.com/lm/bitmover/lm/talks/socal/ tl;dr: fan in/fan out scales great if you have BitKeeper or Git.
  - Touche 12y ago
    Can you explain what you mean? Why is email more "scalable" than a web interface?
  - steveklabnik 12y ago
    We merged 473 PRs in Rust last month: https://github.com/rust-lang/rust/pulse/monthly https://github.com/rust-lang/rust/pulse/monthly That's roughly one every other hour, so still not quite on the same level. And we do use tooling _in addition_ to GitHub, but reviews are done in PR and not over email. (FWIW I personally wish more projects worked like the kernel's, but such is life)
  - hamstergene 12y ago
    Just because one big project relies on emails does not mean they scale better. They work for the kernel because with that strong organization pretty much anything could work for the kernel. Criticisms in the interview were essentially about low discipline that Github interface allows, and how they were not willing to adapt to the kernel needs, not about fundamental deficiencies of pull request based model. So far nothing has been said about why pull requests via a website might handle less load than pull requests over email.
- teamhappy 12y ago
  > [...] nor have I ever heard systematic criticism of Github's methodology. He's not only criticizing Github: > (a) make a real pull request, not the braindamaged crap that github > does when you ask it to request a pull: real explanation, proper email > addresses, proper shortlog, and proper diffstat. > > (b) since github identities are random, I expect the pull request to > be a signed tag, so that I can verify the identity of the person in > question. Real explanation, proper shortlog, signed commit, etc. (Note that he didn't mention emojis.) All he seems to be asking for is that we take our time and do it properly. Makes all the sense in the world for a large, long-lived open source project like Linux. Imagine 20-year-old commit messages from FreeBSD would look like the ones we write today for, say, a npm module. --- edit --- Yeah ... nevermind. He explains it himself further down: > Btw, Joseph, you're a quality example of why I detest the github > interface. For some reason, github has attracted people who have zero > taste, don't care about commit logs, and can't be bothered. > > The fact that I have higher standards then makes people like you make > snarky comments, thinking that you are cool. > > You're a moron. > > Linus
  - GFK_of_xmaspast 12y ago
    "You're a moron." That's our Linus-kun.
    - rdtsc 12y ago
      Well he was a moron. Linus is Linus-kun because often he is right. Here is more of the exchange for context: --- Umm. Notice how the "Joseph" I replied to had deleted all the comments he wrote? That should tell you something. I smacked down a troll. If I was polite to you all those years ago, and I was polite but firm in the two first responses, please give me credit for when I smack somebody down. There may be a reason for it. The fact that the person deleted his messages (or github deleted them for him - I have no idea what their comment policy is) and you cannot see that context any more online should not make you think that I suddenly went crazy. Btw, since I get the github messages in email too, I have a copy. Joseph replied to those "polite but firm" messages where I explained exactly why I don't want to bother with github pull requests with this gem: "I did not realizes that Linus' shit does not stink. Thanks for clearing that up..." ---
- davexunit 12y ago
  I was so happy to read this. I completely agree with Linus' criticisms of the pull request system that GitHub has popularized. They've deeply embedded really bad habits into programmers. I used to wonder "Why do some projects still do patch review on a mailing list?" and then I realized it was for the reasons that Linus gave here.
  - mushly 12y ago
    So what's the best way to do it instead? And why is the current model so bad?
jordigh 12y ago
Let's not forget the other contender for replacing Bitkeeper, Mercurial: http://lkml.iu.edu/hypermail/linux/kernel/0504.2/0670.html http://lkml.iu.edu/hypermail/linux/kernel/0504.2/0670.html We will also be celebrating Mercurial's 10th anniversary next week during the 3.4 Pycon sprint: http://mercurial.selenic.com/wiki/3.4sprint http://mercurial.selenic.com/wiki/3.4sprint
- tjradcliffe 12y ago
  I used Git for a couple of years then changed to Mercurial and have never looked back, except now and then to wonder at the size of the crowd that has gathered around Git, driven in no small part by the success of github (which is not a bad thing!) Git is an entirely appropriate tool for managing as something as large and complex as the Linux kernel, but I'm doing much simpler stuff, and Mercurial is simpler and has a much shallower learning curve. I'd recommend it unequivocally to teams of modest size working on average applications. The state of git documentation is much better today than it was five or six years ago, so maybe its complexity is less of a big deal, but so far I've not encountered any issues with Mercurial that make me think "If only I was using git this would be easy!" What I found with git was I had to maintain a fairly complex mental model of the current state of affairs, and because I'm not very intelligent that took quite a bit of effort. With Mercurial the model is much simpler, so I can spend more of my very limited attention on writing code. It's quite possible that I simply never took the time to learn git properly, but with Mercurial I didn't have to.
  - Someone1234 12y ago
    I won't comment on Mercurial because I don't know anything about it. I really do feel like Git is appropriate for small projects. The .git store is incredibly small (because it only stores changes) and quite simple when you get used to it. You will only only need to use a tiny subset of Git's functionality (or, heck, just the Github web-UI if you wish). I think the hardest thing about Git for big and small teams is getting used to the mental model of Git (branching, merging, master, etc). Once you get it down you realise that Git is less complex, not more... PS - This comment is mostly in reply to the implication that Git is a better tool for large and complex projects like the Linux Kernel. PPS - Mercurial might be a better tool yet still for small projects, I don't know enough about Mercurial to have a qualified opinion either way.
darkmagnus 12y ago
Here is the first Git commit he talks about in the article: https://github.com/git/git/tree/e83c5163316f89bfbde7d9ab23ca2e25604af290 https://github.com/git/git/tree/e83c5163316f89bfbde7d9ab23ca...
- AceJohnny2 12y ago
  Having read "Git from the Bottom Up" [1], it's interesting and refreshing how concise Torvald's explanation is. Of course, he also didn't have as much to explain at that stage :) [1] https://jwiegley.github.io/git-from-the-bottom-up/ https://jwiegley.github.io/git-from-the-bottom-up/
henrik_w 12y ago
"The trick wasn't really so much the coding but coming up with how it organizes the data." I think this really is the key to understanding git as well. When you understand the git data structures, git makes sense. Otherwise, it can be quite difficult to grasp (http://ftp.newartisans.com/pub/git.from.bottom.up.pdf http://ftp.newartisans.com/pub/git.from.bottom.up.pdf was really useful for me).
- drakenot 12y ago
  That link didn't work for me (Page not found) but I found a mirror: https://github.com/umd-byob/presentations/raw/master/2014/0311-git_with_confidence/references/git.from.bottom.up.pdf https://github.com/umd-byob/presentations/raw/master/2014/03...
- krylon 12y ago
  Not only git. There is a nice quote from Fred Brooks Jr. (I think), that if you show me your flow charts and algorithms, I remain as clueless as before, but show me your data structures, and everything else will follow. (I am paraphrasing this from memory, Brooks put it more eloquently.)
  - henrik_w 12y ago
    Yes! I love that quote - so true. Linus has the same opinion: "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." [1] I actually refered to this when discussing switching from Java to Python. One of the biggest problems for me was not easily being able to see the types of the arguments to a function in Python. [2] [1] http://programmers.stackexchange.com/questions/163185/torvalds-quote-about-good-programmer http://programmers.stackexchange.com/questions/163185/torval... [2] http://henrikwarne.com/2014/06/22/switching-from-java-to-python-first-impressions/ http://henrikwarne.com/2014/06/22/switching-from-java-to-pyt...
    - AceJohnny2 12y ago
      > not easily being able to see the types of the arguments to a function in Python. [2] That's because of Duck Typing [1]. I really like Python for how quick and easy it is to make small pieces of software, but I fully agree that Duck Typing can become more of a hindrance on larger projects. [1] http://en.wikipedia.org/wiki/Duck_typing http://en.wikipedia.org/wiki/Duck_typing
jkot 12y ago
I am sad GIT got widely adopted. GIT-SVN was almost superpower a few years ago. :-)
- kinghajj 12y ago
  It still is, depending upon where you work. "I'm having trouble with merging the latest development changes into my feature branch. Tortoise says there are tree conflicts, what--" "Done, I'm committing the result to the SVN repo now." "What?!"
sytse 12y ago
There was a nice 10 year overview of git linked from the article https://www.atlassian.com/git/articles/10-years-of-git/ https://www.atlassian.com/git/articles/10-years-of-git/ I think Atlassian are great sports for featuring GitLab (that competes with their Bitbucket and Stash products) in there, thanks!
- netinstructions 12y ago
  Kind of a nitpick, but that infographic shows that in 2014 Git was used by 33% of developers. Atlassian didn't say which source that came from, but I'm going to guess it came from a 2014 Eclipse Community Survey of 876 respondents[1][2]. I found the poll a bit misleading. There were separate categories for Git and GitHub, but AFAIK you were only supposed to choose one. I'd wager most of the GitHub users are also using Git, so the graph would look more like this[3]. [1] http://www.slideshare.net/IanSkerrett/eclipse-community-survey-2014 http://www.slideshare.net/IanSkerrett/eclipse-community-surv... [2] http://eclipse.dzone.com/articles/eclipse-community-survey-2014 http://eclipse.dzone.com/articles/eclipse-community-survey-2... [3] http://i.imgur.com/CEkIHSQ.png http://i.imgur.com/CEkIHSQ.png
  - sytse 12y ago
    I agree that using separate categories for git and GitHub doesn't make sense.
ffn 12y ago
Jesus, so Torvalds built the MVP for git in 1 day and pretty much scaled it up for kernel usage in 10 days. Now, as a humble average developer, how do I achieve Torvaldian levels of productivity?
- cms07 12y ago
  Spend your better years inside during the 9 months of Finnish Winter.
- Crito 12y ago
  Work smarter, not harder. The reason he was able to get git up and running so fast is because it is fundamentally a very simple system. If you check out the first commit of git in its own git repo, you can see just how simple it is yourself. In that first commit it is barely anything, yet it was sufficient to write a commit for itself.
- vinceguidry 12y ago
  Well, he'd been collecting ideas and requirements for a long time in his head before he finally made a system out of it. The vast majority of the time, what you're building isn't all that clear to you before you have to start. So there's a lot of time wasted just figuring that out. Cut that wasted time out and you become a powerhouse.
- divs1210 12y ago
  As someone pointed out, locking yourself in could definitely be a catalyst. How to develop his sensibilities has a very different answer, though - Have High Standards. * He aches for the very best tool that will suit all his major needs. * He mentions getting the data structures right first, which is of prime importance. And no, it doesn't mean you start building TreeFactoryAdapter's. * He codes at the lowest level, in C (please don't whine about assembly), to maintain complete control over memory usage. I shudder to think what he would come up with if he ever decides to take Common Lisp on a test ride.
- tikhonj 12y ago
  It was something at the top of his head, enough that he had a coherent mental model of what he wanted to build. The actual coding was less designing git as much as putting an existing design down in executable notation. Couple this with decades of programming experience—enough, crucially, to express his thoughts comfortably, without getting distracted by the incidental minutia of actual programming—and you get feats that seem almost super-human. To borrow an overused cliché, the code was just the tip of the iceberg. The hidden body of the MVP was the design he had been thinking about for a while. But we, as external observers, can't really see or appreciate what was going on in his head as he planned this out. It doesn't feel like "real" work, which makes the small remaining part that does seem all that much more impressive. This is actually a real source of tension between certain programmers and managers: it's hard for managers to tell actual thinking apart from day-dreaming, so programmers just look lazy in their eyes. Torvalds wrote about this himself in the interview: > So I'd like to stress that while it really came together in just about ten days or so (at which point I did my first kernel commit using git), it wasn't like it was some kind of mad dash of coding. The actual amount of that early code is actually fairly small, it all depended on getting the basic ideas right. And that I had been mulling over for a while before the whole project started. I'd seen the problems others had. I'd seen what I wanted to avoid doing. How do you achieve this level of productivity yourself? Personally, I think the main quality you want, as I alluded earlier, is comfort. Get yourself to the point where you trust your tools (and yourself) enough to execute your ideas without apprehension. You want to build yourself up to the point where your tools—everything from your programming language to your editor to your libraries—don't get in the way. They should almost feel like extensions of you. Think about it like riding a bicycle or skiing where you start thinking exclusively about where you're going without being distracted by how. Ideally, you want to think about the problem you're solving and the abstraction you're building rather than the mundane details of how you convey this to the computer.
plongeur 12y ago
99% of the time I use: - git status - git add ... - git reset ... - git commit ... - git log ... - git push/pull ... I guess that's "the basics" - what command should I learn next?
- naggie 12y ago
  > git rebase -i Very useful!
  - qzw 12y ago
    Yes, but watch out if you've already pushed to remote branches or even done merge/rebase with other local branches. You're rewriting history.
    - Zikes 12y ago
      I most often use it because I forgot to include a certain file in my most recent commit. I'll just commit that file then rebase -i to merge the two.
      - regularjack 12y ago
        If you want to alter the most recent commit, you can 'git commit --amend', no need to rebase. http://git-scm.com/book/en/v2/Git-Basics-Undoing-Things http://git-scm.com/book/en/v2/Git-Basics-Undoing-Things
      - lentil 12y ago
        For quick changes to the most recent commit, a handy alias to put in your ~/.gitconfig is: fixup = commit --amend -C HEAD That way you can stage whatever you forgot, and then do `git fixup` to update the latest commit with it. Saves a little bit of typing :) I do love `rebase -i` for anything more complicated though.
  - fennecfoxen 12y ago
    that flows naturally into git reflog -- to get you out of a botched rebase with git reset :P
dangero 12y ago
Somewhat off topic, but I'd love to see something more git-like that works for large repositories. I work on video games and our repos are several TB. It's not practical to use git in those situations and Perforce feels too clunky to me.
- jewel 12y ago
  Have you tried git-annex? It handles large files really well and is as close to git-like as you can get. :)
  - dangero 12y ago
    I haven't tried it, but I've looked at it. I think it adds too much complexity. Who decides which files go on the annex and which don't? Doesn't seem like an automated great solution for a scale-able team. I'm looking for something more like CVS/Perforce where you can check anything in, but then with a more Git feeling interface. What I'd really like to do is to look at Git feature by feature and build the closest thing possible that does not include cloning the entire history locally under normal use. I know that is the fundamental paradigm of Git, but it seems that something closer to a hybrid of Git and Perforce could be created. Not sure exactly what it would look like. It's just jarring to jump from Perforce/CVS to Git and I don't think that's completely due to the local repository model. It's how branching works, it's how merging works, etc.
    - jewel 12y ago
      > I haven't tried it, but I've looked at it. I think it adds too much complexity. Who decides which files go on the annex and which don't? Doesn't seem like an automated great solution for a scale-able team. I would put all binary files into git annex (as determined by `file`). This can be done by a commit hook automatically. With another hook that makes sure that `git annex pull` is run when the user checks out code, you'd have a solution that was close enough to automatic for most use cases. (You'd have to help your users with weird situations from time to time, but that's true of git anyway.)
- munificent 12y ago
thomasfl 12y ago
I wonder if BitKeeper owner Larry McVoy has ever regretted not open sourcing his software? Git tools is a whole industry now. The most promiment one, github, recently become one of the 100 most popular websites on the planet.
- luckydude 12y ago
  Regret it? Sure. I'd do it in a heartbeat if I could figure out how to make it work. Still would and there is plenty in BK that Git doesn't have. Like submodules that actually work exactly like a monolithic tree, just lets you clone what you need. But we've never figured out how to make it work financially. If anyone has any ideas I'm all ears (though pointing at github and saying "do that" isn't an idea that I can execute). BTW, BK used to be pretty darned close to open source, you got the source code under a funky license that said "don't take out the part that lets us make money". We stopped shipping the source when we learned that the very first thing that someone committed to the repo was taking out the part that let us make money.
  - anonbanker 12y ago
    > Sure. I'd do it in a heartbeat if I could figure out how to make it work. just upload it to github. :)
    - luckydude 12y ago
      Have an upvote for making me laugh. It's not as far fetched as you might think though, we've been building million cset trees and git is so darn slow that we made a "bk fast-export" that spits out the stuff that git wants. Because Git was over 20x slower just running the commits.
  - qzw 12y ago
    Very cool of you to share your thoughts. I sympathize with your dilemma. It seems that the people who end up making money out of free/open source software are often not the ones who write the code. And I remember reading back in the day that Linus would talk with you extensively about the nuts and bolts of DVCSs, so I'm sure all git users owe you some gratitude for inspiring him to create git and getting the fundamentals right from the get go. Out of curiosity, and please feel free not to answer, is BK still a viable commercial product bringing in significant revenue? And what obstacle do you see with going the platform/service route like github? I assume that's something you've seriously considered, even without open sourcing BK.
restalis 12y ago
"Why do you think it's been so widely adopted? Torvalds: I think that many others had been frustrated by all the same issues that made me hate SCM's" Actually, the adoption was caused by the network effect. It took off with a core group of Linux-involved developers promoting it religiously, then it went on like a fashion (we are git-wielders, we are cool), and ended up being the default choice for a lot of other projects that either depended on some degree on 3rd party git-managed modules, or wanted to benefit from the existing disciplined git-trained developers, or just... because (for all things being equal). Having this presumption that the adopters actually thought for themselves and Torvalds's brain-child got where it is now thanks to its own technical merits is a pleasant thought though.
netinstructions 12y ago
I wrote an article a few months ago summarizing where Git is in 2015 if anyone is curious[1]. Most developers at my company don't seem to want to learn a "new" SCM so we're stuck with SVN (or CVS for some projects). We rarely have a stable trunk. Branching takes too long so most people don't want to do it. Merging two branches is such a scary thing that people avoid branching in the first place. Or I've seen them copy code from one branch in one Eclipse window and paste into another separate project (on another branch) in another Eclipse window. They repeat this manually for the next 5 to 20 files. I don't think they've realized a merge can often be as simple as a button click or one SCM command. But hey, it works for them. The one time on my team that someone was interested in exploring Git was when we couldn't find an up-to-date Maven SCM connector for SVN that played nicely with Eclipse. The solution? Use an older version of Eclipse. [1] http://www.netinstructions.com/the-case-for-git/ http://www.netinstructions.com/the-case-for-git/
- praneshp 12y ago
  I work for a huge-internet-corp, which I don;t want to name here (pretty easy to find out from my username and a couple of google searches, if you care). I sympathize (actually empathize) with your problem (developers dont want to learn a new SCM). We had a big drive last year where management stuck down a hammer, and told us to move to git, 100%. Several senior developers (many of them actually architects now, not really writing much code) made a lot of noise on internal mailing lists. One person would make terrible mistakes that one reading of any git manual would help you understand, and then complain loudly (with a lot of swearing) how much it sucked compared to SVN. I was annoyed by this, because except for the hammer thing, the company did everything else perfectly. We have a stable corp github, there were several training sessions offered, and the reasons clearly explained. It was the first time I was happy about the slightly dictatorial approach taken towards the whole thing, instead of trying to reason with 50-year old babies.
- serve_yay 12y ago
  Yes, I remember those days. I remember, as Linus says, planning branch merges. It really was terrible. But the solution is to use git (or Mercurial), not shut down the discussion saying you don't want to learn a new tool. That's a very unfortunate attitude for anyone who works in software.
DigitalSea 12y ago
I just wanted to say thank you Linus for giving us Git. Before Git I was using SVN and while the wounds are healing, I will bear these SVN scars for many years to come. As a developer Git has made my life so much easier and not only that, thanks to Github I can help collaborate on open source projects as well as my own with ease.
misiti3780 12y ago
off topic: is the there a third party service that will let me collapse all sub-comments and just read the parent comments?(and then expand them if they seem interesting?)
- kbart 12y ago
  There's API for HN (https://github.com/HackerNews/API https://github.com/HackerNews/API), it's not that hard to make one yourself. This is Hacker News after all.
  - misiti3780 12y ago
    i know - i was hoping someone else did it already! thanks
jammycakes 12y ago
One thing worth saying here is that Git is now the dominant player in the source control market. According to almost every survey I've seen, it's overtaken Subversion (according to some reports by a fairly large margin) not only for open source and hobbyists but in the enterprise as well. It's more or less become a lingua franca for communicating source code between teams these days, as well as being the preferred deployment option for many cloud hosting providers and even some package managers. No other source control system has ever managed that. As such, you simply can't afford not to know it these days. In fact, I can't help thinking that not using Git should now be a major red flag for job candidates and prospective employers alike. Sure, not using your favourite tool may not be that big a deal, but not using what is to all intents and purposes the lingua franca of source code communication is a serious omission.
- minusSeven 12y ago
  Can't agree with this assertion. Your source control of choice should not be in any way a red flag. It just a choice and because of learning curve I choose to learn Mercurial instead. I don't see how source control matters to that extent. If I need to use it I will learn it then. I don't think most people care as such.
- blakeyrat 12y ago
  As long as you don't expect people to like it.
pedrow 12y ago
I wish Linus would now turn his attention to: 1) make (ought to be a simple concept but never seems like it's quite been done right) 2) init (only he has the clout to solve the pid 1 controversy) Will check back in 2025 for progress report.
- stinos 12y ago
  Ok, but only if he first turns his attention to a proper git submodules/multiple reposities kind of thing
INTPenis 12y ago
Linus is not a magician, he is however very good at working with systems and spotting exactly where those systems need improvement. I respect this quality immensely.
xsace 12y ago
Now if you can tackle dependency management and build tools Linus, I would appreciate. With love, me.

509 points by lclark 12y ago

Linus says of the Github platform - That's partly because of how the kernel is developed, but part of it was that the GitHub interfaces were actively encouraging bad behavior. Commits done on GitHub had bad commit messages etc, because the web interfaces at GitHub were actively encouraging bad behavior. They did fix some of that, so it probably works better, but it will never be appropriate for something like the Linux kernel. I haven't ever looked at the kernel workflow, nor have I ever heard systematic criticism of Github's methodology. Does anyone else have input on what Linus may mean by his opinion?

merqumab 12y ago
See this 2012 thread https://github.com/torvalds/linux/pull/17#issuecomment-5654674 https://github.com/torvalds/linux/pull/17#issuecomment-56546...
rentnorove 12y ago
This pull request: https://github.com/torvalds/linux/pull/17 https://github.com/torvalds/linux/pull/17
gregkh 12y ago
The kernel development model is documented here: http://lxr.free-electrons.com/source/Documentation/development-process/ http://lxr.free-electrons.com/source/Documentation/developme... Github doesn't scale at all for large projects. The kernel is averaging over 8 changes an hour, 24 hours a day. That rate of change can never be handled by doing pull requests and web site review. It only can work with email and review and scriptable processes.
- quantumet 12y ago
  Eh, I mildly disagree on web site review. Perhaps not GitHub's pull request/review model, but big projects like Android and Chromium do all their review on web interfaces.
  - edejong 12y ago
    Using Gerrit, which is an automated mimick of Linus' development model. By the way, we've been succesfully using Gerrit on smaller-scale projects as well (around 8 change-requests / day) and don't even want to think about going back to pull-requests.
    - sunnyps 12y ago
      What you've said is essentially correct, but I'd like to make a minor correction: Chromium uses rietveld whereas Android (and some Chromium related projects) use gerrit. Rietveld and gerrit have similar workflows except that gerrit is more integrated with git (I think). Both are named after Gerrit Rietveld, a famous Dutch designer.
- wyldfire 12y ago
  But don't "trusted lieutenants" (yourself included?) mitigate a lot of the challenge posed by that high rate of change?
  - luckydude 12y ago
    Greg didn't reply but I think the answer is a resounding yes. I wrote this up a bit back when the devs were using BitKeeper. Lemme go find that. OK, ignore the "I want the 1995 era web pages" look and peek at this: http://www.mcvoy.com/lm/bitmover/lm/talks/socal/ http://www.mcvoy.com/lm/bitmover/lm/talks/socal/ tl;dr: fan in/fan out scales great if you have BitKeeper or Git.
- Touche 12y ago
  Can you explain what you mean? Why is email more "scalable" than a web interface?
- steveklabnik 12y ago
  We merged 473 PRs in Rust last month: https://github.com/rust-lang/rust/pulse/monthly https://github.com/rust-lang/rust/pulse/monthly That's roughly one every other hour, so still not quite on the same level. And we do use tooling _in addition_ to GitHub, but reviews are done in PR and not over email. (FWIW I personally wish more projects worked like the kernel's, but such is life)
- hamstergene 12y ago
  Just because one big project relies on emails does not mean they scale better. They work for the kernel because with that strong organization pretty much anything could work for the kernel. Criticisms in the interview were essentially about low discipline that Github interface allows, and how they were not willing to adapt to the kernel needs, not about fundamental deficiencies of pull request based model. So far nothing has been said about why pull requests via a website might handle less load than pull requests over email.
teamhappy 12y ago
> [...] nor have I ever heard systematic criticism of Github's methodology. He's not only criticizing Github: > (a) make a real pull request, not the braindamaged crap that github > does when you ask it to request a pull: real explanation, proper email > addresses, proper shortlog, and proper diffstat. > > (b) since github identities are random, I expect the pull request to > be a signed tag, so that I can verify the identity of the person in > question. Real explanation, proper shortlog, signed commit, etc. (Note that he didn't mention emojis.) All he seems to be asking for is that we take our time and do it properly. Makes all the sense in the world for a large, long-lived open source project like Linux. Imagine 20-year-old commit messages from FreeBSD would look like the ones we write today for, say, a npm module. --- edit --- Yeah ... nevermind. He explains it himself further down: > Btw, Joseph, you're a quality example of why I detest the github > interface. For some reason, github has attracted people who have zero > taste, don't care about commit logs, and can't be bothered. > > The fact that I have higher standards then makes people like you make > snarky comments, thinking that you are cool. > > You're a moron. > > Linus
- GFK_of_xmaspast 12y ago
  "You're a moron." That's our Linus-kun.
  - rdtsc 12y ago
    Well he was a moron. Linus is Linus-kun because often he is right. Here is more of the exchange for context: --- Umm. Notice how the "Joseph" I replied to had deleted all the comments he wrote? That should tell you something. I smacked down a troll. If I was polite to you all those years ago, and I was polite but firm in the two first responses, please give me credit for when I smack somebody down. There may be a reason for it. The fact that the person deleted his messages (or github deleted them for him - I have no idea what their comment policy is) and you cannot see that context any more online should not make you think that I suddenly went crazy. Btw, since I get the github messages in email too, I have a copy. Joseph replied to those "polite but firm" messages where I explained exactly why I don't want to bother with github pull requests with this gem: "I did not realizes that Linus' shit does not stink. Thanks for clearing that up..." ---
davexunit 12y ago
I was so happy to read this. I completely agree with Linus' criticisms of the pull request system that GitHub has popularized. They've deeply embedded really bad habits into programmers. I used to wonder "Why do some projects still do patch review on a mailing list?" and then I realized it was for the reasons that Linus gave here.
- mushly 12y ago
  So what's the best way to do it instead? And why is the current model so bad?

jordigh 12y ago

Let's not forget the other contender for replacing Bitkeeper, Mercurial: http://lkml.iu.edu/hypermail/linux/kernel/0504.2/0670.html http://lkml.iu.edu/hypermail/linux/kernel/0504.2/0670.html We will also be celebrating Mercurial's 10th anniversary next week during the 3.4 Pycon sprint: http://mercurial.selenic.com/wiki/3.4sprint http://mercurial.selenic.com/wiki/3.4sprint

tjradcliffe 12y ago
I used Git for a couple of years then changed to Mercurial and have never looked back, except now and then to wonder at the size of the crowd that has gathered around Git, driven in no small part by the success of github (which is not a bad thing!) Git is an entirely appropriate tool for managing as something as large and complex as the Linux kernel, but I'm doing much simpler stuff, and Mercurial is simpler and has a much shallower learning curve. I'd recommend it unequivocally to teams of modest size working on average applications. The state of git documentation is much better today than it was five or six years ago, so maybe its complexity is less of a big deal, but so far I've not encountered any issues with Mercurial that make me think "If only I was using git this would be easy!" What I found with git was I had to maintain a fairly complex mental model of the current state of affairs, and because I'm not very intelligent that took quite a bit of effort. With Mercurial the model is much simpler, so I can spend more of my very limited attention on writing code. It's quite possible that I simply never took the time to learn git properly, but with Mercurial I didn't have to.
- Someone1234 12y ago
  I won't comment on Mercurial because I don't know anything about it. I really do feel like Git is appropriate for small projects. The .git store is incredibly small (because it only stores changes) and quite simple when you get used to it. You will only only need to use a tiny subset of Git's functionality (or, heck, just the Github web-UI if you wish). I think the hardest thing about Git for big and small teams is getting used to the mental model of Git (branching, merging, master, etc). Once you get it down you realise that Git is less complex, not more... PS - This comment is mostly in reply to the implication that Git is a better tool for large and complex projects like the Linux Kernel. PPS - Mercurial might be a better tool yet still for small projects, I don't know enough about Mercurial to have a qualified opinion either way.

darkmagnus 12y ago

Here is the first Git commit he talks about in the article: https://github.com/git/git/tree/e83c5163316f89bfbde7d9ab23ca2e25604af290 https://github.com/git/git/tree/e83c5163316f89bfbde7d9ab23ca...

AceJohnny2 12y ago
Having read "Git from the Bottom Up" [1], it's interesting and refreshing how concise Torvald's explanation is. Of course, he also didn't have as much to explain at that stage :) [1] https://jwiegley.github.io/git-from-the-bottom-up/ https://jwiegley.github.io/git-from-the-bottom-up/

henrik_w 12y ago

"The trick wasn't really so much the coding but coming up with how it organizes the data." I think this really is the key to understanding git as well. When you understand the git data structures, git makes sense. Otherwise, it can be quite difficult to grasp (http://ftp.newartisans.com/pub/git.from.bottom.up.pdf http://ftp.newartisans.com/pub/git.from.bottom.up.pdf was really useful for me).

drakenot 12y ago
That link didn't work for me (Page not found) but I found a mirror: https://github.com/umd-byob/presentations/raw/master/2014/0311-git_with_confidence/references/git.from.bottom.up.pdf https://github.com/umd-byob/presentations/raw/master/2014/03...
krylon 12y ago
Not only git. There is a nice quote from Fred Brooks Jr. (I think), that if you show me your flow charts and algorithms, I remain as clueless as before, but show me your data structures, and everything else will follow. (I am paraphrasing this from memory, Brooks put it more eloquently.)
- henrik_w 12y ago
  Yes! I love that quote - so true. Linus has the same opinion: "Bad programmers worry about the code. Good programmers worry about data structures and their relationships." [1] I actually refered to this when discussing switching from Java to Python. One of the biggest problems for me was not easily being able to see the types of the arguments to a function in Python. [2] [1] http://programmers.stackexchange.com/questions/163185/torvalds-quote-about-good-programmer http://programmers.stackexchange.com/questions/163185/torval... [2] http://henrikwarne.com/2014/06/22/switching-from-java-to-python-first-impressions/ http://henrikwarne.com/2014/06/22/switching-from-java-to-pyt...
  - AceJohnny2 12y ago
    > not easily being able to see the types of the arguments to a function in Python. [2] That's because of Duck Typing [1]. I really like Python for how quick and easy it is to make small pieces of software, but I fully agree that Duck Typing can become more of a hindrance on larger projects. [1] http://en.wikipedia.org/wiki/Duck_typing http://en.wikipedia.org/wiki/Duck_typing

jkot 12y ago

I am sad GIT got widely adopted. GIT-SVN was almost superpower a few years ago. :-)

kinghajj 12y ago
It still is, depending upon where you work. "I'm having trouble with merging the latest development changes into my feature branch. Tortoise says there are tree conflicts, what--" "Done, I'm committing the result to the SVN repo now." "What?!"

sytse 12y ago

There was a nice 10 year overview of git linked from the article https://www.atlassian.com/git/articles/10-years-of-git/ https://www.atlassian.com/git/articles/10-years-of-git/ I think Atlassian are great sports for featuring GitLab (that competes with their Bitbucket and Stash products) in there, thanks!

netinstructions 12y ago
Kind of a nitpick, but that infographic shows that in 2014 Git was used by 33% of developers. Atlassian didn't say which source that came from, but I'm going to guess it came from a 2014 Eclipse Community Survey of 876 respondents[1][2]. I found the poll a bit misleading. There were separate categories for Git and GitHub, but AFAIK you were only supposed to choose one. I'd wager most of the GitHub users are also using Git, so the graph would look more like this[3]. [1] http://www.slideshare.net/IanSkerrett/eclipse-community-survey-2014 http://www.slideshare.net/IanSkerrett/eclipse-community-surv... [2] http://eclipse.dzone.com/articles/eclipse-community-survey-2014 http://eclipse.dzone.com/articles/eclipse-community-survey-2... [3] http://i.imgur.com/CEkIHSQ.png http://i.imgur.com/CEkIHSQ.png
- sytse 12y ago
  I agree that using separate categories for git and GitHub doesn't make sense.

ffn 12y ago

Jesus, so Torvalds built the MVP for git in 1 day and pretty much scaled it up for kernel usage in 10 days. Now, as a humble average developer, how do I achieve Torvaldian levels of productivity?

cms07 12y ago
Spend your better years inside during the 9 months of Finnish Winter.
Crito 12y ago
Work smarter, not harder. The reason he was able to get git up and running so fast is because it is fundamentally a very simple system. If you check out the first commit of git in its own git repo, you can see just how simple it is yourself. In that first commit it is barely anything, yet it was sufficient to write a commit for itself.
vinceguidry 12y ago
Well, he'd been collecting ideas and requirements for a long time in his head before he finally made a system out of it. The vast majority of the time, what you're building isn't all that clear to you before you have to start. So there's a lot of time wasted just figuring that out. Cut that wasted time out and you become a powerhouse.
divs1210 12y ago
As someone pointed out, locking yourself in could definitely be a catalyst. How to develop his sensibilities has a very different answer, though - Have High Standards. * He aches for the very best tool that will suit all his major needs. * He mentions getting the data structures right first, which is of prime importance. And no, it doesn't mean you start building TreeFactoryAdapter's. * He codes at the lowest level, in C (please don't whine about assembly), to maintain complete control over memory usage. I shudder to think what he would come up with if he ever decides to take Common Lisp on a test ride.
tikhonj 12y ago
It was something at the top of his head, enough that he had a coherent mental model of what he wanted to build. The actual coding was less designing git as much as putting an existing design down in executable notation. Couple this with decades of programming experience—enough, crucially, to express his thoughts comfortably, without getting distracted by the incidental minutia of actual programming—and you get feats that seem almost super-human. To borrow an overused cliché, the code was just the tip of the iceberg. The hidden body of the MVP was the design he had been thinking about for a while. But we, as external observers, can't really see or appreciate what was going on in his head as he planned this out. It doesn't feel like "real" work, which makes the small remaining part that does seem all that much more impressive. This is actually a real source of tension between certain programmers and managers: it's hard for managers to tell actual thinking apart from day-dreaming, so programmers just look lazy in their eyes. Torvalds wrote about this himself in the interview: > So I'd like to stress that while it really came together in just about ten days or so (at which point I did my first kernel commit using git), it wasn't like it was some kind of mad dash of coding. The actual amount of that early code is actually fairly small, it all depended on getting the basic ideas right. And that I had been mulling over for a while before the whole project started. I'd seen the problems others had. I'd seen what I wanted to avoid doing. How do you achieve this level of productivity yourself? Personally, I think the main quality you want, as I alluded earlier, is comfort. Get yourself to the point where you trust your tools (and yourself) enough to execute your ideas without apprehension. You want to build yourself up to the point where your tools—everything from your programming language to your editor to your libraries—don't get in the way. They should almost feel like extensions of you. Think about it like riding a bicycle or skiing where you start thinking exclusively about where you're going without being distracted by how. Ideally, you want to think about the problem you're solving and the abstraction you're building rather than the mundane details of how you convey this to the computer.

plongeur 12y ago

99% of the time I use: - git status - git add ... - git reset ... - git commit ... - git log ... - git push/pull ... I guess that's "the basics" - what command should I learn next?

naggie 12y ago
> git rebase -i Very useful!
- qzw 12y ago
  Yes, but watch out if you've already pushed to remote branches or even done merge/rebase with other local branches. You're rewriting history.
  - Zikes 12y ago
    I most often use it because I forgot to include a certain file in my most recent commit. I'll just commit that file then rebase -i to merge the two.
    - regularjack 12y ago
      If you want to alter the most recent commit, you can 'git commit --amend', no need to rebase. http://git-scm.com/book/en/v2/Git-Basics-Undoing-Things http://git-scm.com/book/en/v2/Git-Basics-Undoing-Things
    - lentil 12y ago
      For quick changes to the most recent commit, a handy alias to put in your ~/.gitconfig is: fixup = commit --amend -C HEAD That way you can stage whatever you forgot, and then do `git fixup` to update the latest commit with it. Saves a little bit of typing :) I do love `rebase -i` for anything more complicated though.
- fennecfoxen 12y ago
  that flows naturally into git reflog -- to get you out of a botched rebase with git reset :P

dangero 12y ago

Somewhat off topic, but I'd love to see something more git-like that works for large repositories. I work on video games and our repos are several TB. It's not practical to use git in those situations and Perforce feels too clunky to me.

jewel 12y ago
Have you tried git-annex? It handles large files really well and is as close to git-like as you can get. :)
- dangero 12y ago
  I haven't tried it, but I've looked at it. I think it adds too much complexity. Who decides which files go on the annex and which don't? Doesn't seem like an automated great solution for a scale-able team. I'm looking for something more like CVS/Perforce where you can check anything in, but then with a more Git feeling interface. What I'd really like to do is to look at Git feature by feature and build the closest thing possible that does not include cloning the entire history locally under normal use. I know that is the fundamental paradigm of Git, but it seems that something closer to a hybrid of Git and Perforce could be created. Not sure exactly what it would look like. It's just jarring to jump from Perforce/CVS to Git and I don't think that's completely due to the local repository model. It's how branching works, it's how merging works, etc.
  - jewel 12y ago
    > I haven't tried it, but I've looked at it. I think it adds too much complexity. Who decides which files go on the annex and which don't? Doesn't seem like an automated great solution for a scale-able team. I would put all binary files into git annex (as determined by `file`). This can be done by a commit hook automatically. With another hook that makes sure that `git annex pull` is run when the user checks out code, you'd have a solution that was close enough to automatic for most use cases. (You'd have to help your users with weird situations from time to time, but that's true of git anyway.)
munificent 12y ago

thomasfl 12y ago

I wonder if BitKeeper owner Larry McVoy has ever regretted not open sourcing his software? Git tools is a whole industry now. The most promiment one, github, recently become one of the 100 most popular websites on the planet.

luckydude 12y ago
Regret it? Sure. I'd do it in a heartbeat if I could figure out how to make it work. Still would and there is plenty in BK that Git doesn't have. Like submodules that actually work exactly like a monolithic tree, just lets you clone what you need. But we've never figured out how to make it work financially. If anyone has any ideas I'm all ears (though pointing at github and saying "do that" isn't an idea that I can execute). BTW, BK used to be pretty darned close to open source, you got the source code under a funky license that said "don't take out the part that lets us make money". We stopped shipping the source when we learned that the very first thing that someone committed to the repo was taking out the part that let us make money.
- anonbanker 12y ago
  > Sure. I'd do it in a heartbeat if I could figure out how to make it work. just upload it to github. :)
  - luckydude 12y ago
    Have an upvote for making me laugh. It's not as far fetched as you might think though, we've been building million cset trees and git is so darn slow that we made a "bk fast-export" that spits out the stuff that git wants. Because Git was over 20x slower just running the commits.
- qzw 12y ago
  Very cool of you to share your thoughts. I sympathize with your dilemma. It seems that the people who end up making money out of free/open source software are often not the ones who write the code. And I remember reading back in the day that Linus would talk with you extensively about the nuts and bolts of DVCSs, so I'm sure all git users owe you some gratitude for inspiring him to create git and getting the fundamentals right from the get go. Out of curiosity, and please feel free not to answer, is BK still a viable commercial product bringing in significant revenue? And what obstacle do you see with going the platform/service route like github? I assume that's something you've seriously considered, even without open sourcing BK.

restalis 12y ago

"Why do you think it's been so widely adopted? Torvalds: I think that many others had been frustrated by all the same issues that made me hate SCM's" Actually, the adoption was caused by the network effect. It took off with a core group of Linux-involved developers promoting it religiously, then it went on like a fashion (we are git-wielders, we are cool), and ended up being the default choice for a lot of other projects that either depended on some degree on 3rd party git-managed modules, or wanted to benefit from the existing disciplined git-trained developers, or just... because (for all things being equal). Having this presumption that the adopters actually thought for themselves and Torvalds's brain-child got where it is now thanks to its own technical merits is a pleasant thought though.

netinstructions 12y ago

I wrote an article a few months ago summarizing where Git is in 2015 if anyone is curious[1]. Most developers at my company don't seem to want to learn a "new" SCM so we're stuck with SVN (or CVS for some projects). We rarely have a stable trunk. Branching takes too long so most people don't want to do it. Merging two branches is such a scary thing that people avoid branching in the first place. Or I've seen them copy code from one branch in one Eclipse window and paste into another separate project (on another branch) in another Eclipse window. They repeat this manually for the next 5 to 20 files. I don't think they've realized a merge can often be as simple as a button click or one SCM command. But hey, it works for them. The one time on my team that someone was interested in exploring Git was when we couldn't find an up-to-date Maven SCM connector for SVN that played nicely with Eclipse. The solution? Use an older version of Eclipse. [1] http://www.netinstructions.com/the-case-for-git/ http://www.netinstructions.com/the-case-for-git/

praneshp 12y ago
I work for a huge-internet-corp, which I don;t want to name here (pretty easy to find out from my username and a couple of google searches, if you care). I sympathize (actually empathize) with your problem (developers dont want to learn a new SCM). We had a big drive last year where management stuck down a hammer, and told us to move to git, 100%. Several senior developers (many of them actually architects now, not really writing much code) made a lot of noise on internal mailing lists. One person would make terrible mistakes that one reading of any git manual would help you understand, and then complain loudly (with a lot of swearing) how much it sucked compared to SVN. I was annoyed by this, because except for the hammer thing, the company did everything else perfectly. We have a stable corp github, there were several training sessions offered, and the reasons clearly explained. It was the first time I was happy about the slightly dictatorial approach taken towards the whole thing, instead of trying to reason with 50-year old babies.
serve_yay 12y ago
Yes, I remember those days. I remember, as Linus says, planning branch merges. It really was terrible. But the solution is to use git (or Mercurial), not shut down the discussion saying you don't want to learn a new tool. That's a very unfortunate attitude for anyone who works in software.

DigitalSea 12y ago

I just wanted to say thank you Linus for giving us Git. Before Git I was using SVN and while the wounds are healing, I will bear these SVN scars for many years to come. As a developer Git has made my life so much easier and not only that, thanks to Github I can help collaborate on open source projects as well as my own with ease.

misiti3780 12y ago

off topic: is the there a third party service that will let me collapse all sub-comments and just read the parent comments?(and then expand them if they seem interesting?)

kbart 12y ago
There's API for HN (https://github.com/HackerNews/API https://github.com/HackerNews/API), it's not that hard to make one yourself. This is Hacker News after all.
- misiti3780 12y ago
  i know - i was hoping someone else did it already! thanks

jammycakes 12y ago

One thing worth saying here is that Git is now the dominant player in the source control market. According to almost every survey I've seen, it's overtaken Subversion (according to some reports by a fairly large margin) not only for open source and hobbyists but in the enterprise as well. It's more or less become a lingua franca for communicating source code between teams these days, as well as being the preferred deployment option for many cloud hosting providers and even some package managers. No other source control system has ever managed that. As such, you simply can't afford not to know it these days. In fact, I can't help thinking that not using Git should now be a major red flag for job candidates and prospective employers alike. Sure, not using your favourite tool may not be that big a deal, but not using what is to all intents and purposes the lingua franca of source code communication is a serious omission.

minusSeven 12y ago
Can't agree with this assertion. Your source control of choice should not be in any way a red flag. It just a choice and because of learning curve I choose to learn Mercurial instead. I don't see how source control matters to that extent. If I need to use it I will learn it then. I don't think most people care as such.
blakeyrat 12y ago
As long as you don't expect people to like it.

pedrow 12y ago

I wish Linus would now turn his attention to: 1) make (ought to be a simple concept but never seems like it's quite been done right) 2) init (only he has the clout to solve the pid 1 controversy) Will check back in 2025 for progress report.

stinos 12y ago
Ok, but only if he first turns his attention to a proper git submodules/multiple reposities kind of thing

INTPenis 12y ago

Linus is not a magician, he is however very good at working with systems and spotting exactly where those systems need improvement. I respect this quality immensely.

xsace 12y ago

Now if you can tackle dependency management and build tools Linus, I would appreciate. With love, me.