31 ms·

Scaling Mercurial at Facebook

360 points by jordigh 13y ago

ngoldbaum 13y ago
Mercurial has seriously improved over the past couple of years. If you tried mercurial a few years ago and were scared away due to speed or functionality issues, you might want to give it another shot.
- JoshTriplett 13y ago
  Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches? Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"?
  - ngoldbaum 13y ago
    Mercurial refers to git-like lightweight branches as bookmarks [1]. Branches are still permanent tags in the repository. Commit --amend is now part of base mercurial. Rebase is available as an extension that is shipped with the client, like most advanced functionality. I don't know what add -p does. [1] http://mercurial.selenic.com/wiki/Bookmarks/ http://mercurial.selenic.com/wiki/Bookmarks/
    - durin42 13y ago
      "add -p" is basically the 'hg record' extension.
      - _ikke_ 13y ago
        It's more like `git commit -p`. If you abort `hg record`, the selection you've made is lost.
  - mikegioia 13y ago
    I work in several branches in the same file system directory. Branching in mercurial is as simple as "hg branch branchname", and switching branches is just "hg update branchname". As long as you make sure you commit everything you can easily work in one directory. Merging changes across branches is easy too, as you only need to specify the revision number.
    - ahomescu1 13y ago
      If you push while having open local branches, they get pushed to the remote (or at least that's how it was last time I used them). That's not what you want, in most cases (git branches are local).
      - rbehrends 13y ago
        Mercurial won't push new branches unless you specify -f or --new-branch. You can selectively push only the branches you want with hg push -r branch_or_revision (hg push -r . for the current revision). You can also hide branches using phases, so they won't be pushed even with -f or --new-branch.
        ahomescu1 13y ago
        The guide from [1] confirms it works the way I saw it (it pushes all branches): Mercurial will push/pull all branches by default, while git will push/pull only the current branch. Did this behavior change in recent versions of hg? 1 - http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/ http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me...
        rbehrends 13y ago
        Mercurial will attempt to push all branches. It will then immediately stop with an error message unless you specify -f or --new-branch if you're trying to push a branch that does not already exist in the remote repository (or even to create a new head in an existing branch). In order to get rid of the error message you need to either (1) explicitly tell Mercurial to push all branches, (2) specify which existing branch(es)/revision(s) you want to push (they must already exist remotely), or (3) hide the branch with hg phase secret -f. This means that your local branches will not leak into the remote repository unless you specifically instruct Mercurial to do so. Of these, only the phase approach is new (2.1+). The rest hasn't changed.
  - jordigh 13y ago
    > if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. There is no real standard workflow. There are tools in place to build whatever you want. This is old, but remarkably still relevant: http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/ http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me... If you want git-like branches, Mercurial bookmarks are close, but not quite the same. They have some intentional differences, which make sense and do not impede you, but will not make sense if you absolutely want them to behave like git branches. > "commit --amend", "rebase -i", and "add -p"? hg commit --amend, hg histedit, hg record (but hg crecord is much nicer). They're built-in, but you have to flip them on the last two with a config switch.
    - JoshTriplett 13y ago
      That post helped, and bookmarks might get the job done, but the mercurial culture doesn't seem to encourage them compared to branches. As a result, you tend to find a lot more separate mercurial repositories than single repos with bookmarks, and a lot more named branches with hard-to-eliminate commits as well (since even "closing" a branch doesn't really get rid of it). > hg commit --amend, hg histedit, hg record (but hg crecord is much nicer). > They're built-in, but you have to flip them on the last two with a config switch. And that's the fundamental attitude that I can't stand in mercurial's culture: "careful, you might accidentally throw away that information you specifically wanted to throw away". Those commands aren't available by default because the culture doesn't encourage that workflow, so even if you use them for your own personal workflow you'll tend to find a lot more junk commits and forever-remembered branch names in the average mercurial repository.
      - mixmastamyk 13y ago
        I might have read this wrong, but safety for newbies doesn't sound all that bad to me.
        JoshTriplett 13y ago
        History editing in git is not unsafe: because it's a part of the default set of tools, git offers functionality like the reflog to keep track of old history locally and avoid losing it. It's extremely difficult to permanently lose content in a git repository even using the history editing commands; it's always just a "git reflog" away.
  - durin42 13y ago
    Note that extensions that ship with Mercurial are exactly as supported as the tool itself, and come with the same kinds of backwards compatibility guarantees. It's not worth worrying about rebase, histedit, and record being extensions - if you want them, turn them on and use them.
    - JoshTriplett 13y ago
      They're not exactly as supported, because they're not on by default, and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.
      - durin42 13y ago
        No, they're exactly as supported. What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shooting their foot off unexpectedly, and UI clutter. mq alone (which we actually recommend avoiding now if you don't really need it) adds a bunch of verbs to the command line interface. We've talked about having an 'hg hello' command or something that'd give you a proposed template hgrc that suggest you might want rebase/record/histedit/pager/color/progress, but nobody's worked out exactly what that should be like. Does something like that sound like it might be helpful?
        mushishi 13y ago
        Could you elaborate on why to avoid mq?
        masklinn 13y ago
        It's a very powerful tool, but also a complex one, and one which brings in a bunch of new commands. There are a bunch of extension which provide nice UIs to various subsets of mq's full power, and are much simpler. If you need mq, use mq, but if you just need to e.g. do some history edition, use histedit.
  - k_bx 13y ago
    Why do you think mercurial branches are not lightweight? Which operation (create, close, push, pull) on branches is noticeably slower than git's?
    - wtetzner 13y ago
      I don't think he was talking about performance. I think he meant lightweight in the sense that a branch (in git) is just a pointer to a commit. It's conceptually lightweight.
      - JoshTriplett 13y ago
        Both: it's conceptually lightweight, and because of that "pointer to a commit" structure it's also effectively free to create and move around branches.
        k_bx 13y ago
        Why is mercurial branches ineffective to move around branches? If I'm not mistaken, if you want to move commits from one branch to another you do absolutely the same rebase.
      - k_bx 13y ago
        Well, mercurial branch is just a record in list of branches, so is it then also "conceptually lightweight"? What is "conceptually lightweight" anyway?
    - fhd2 13y ago
      According to #mercurial, branches (even closed ones) begin to noticeably slow down the repository when you reach around 2000. That's not at all impossible to reach with fine-grained feature branches. Bookmarks are really just pointers to commits, no branches in any sense, so they cannot be used for feature branches either. As a result, we simply don't use feature branches in Mercurial. It's literally the only issue I can think of.
      - 13y ago
  - Uchikoma 13y ago
    Let me translate: "if this tool is not exactly like Git, I will not use it".
    - gecko 13y ago
      That sentiment comprises a truly mindblowing number of "bug" reports that hit the Mercurial mailing list every month. I honestly wonder sometimes if I'm simply forgetting how CVS users did the same thing when Subversion shipped and I forgot about it/missed it, or what.
      - durin42 13y ago
        svn was designed to be "CVS done right," which informed a lot of the design and workflow to the point that some CVS misfeatures only got carried through by virtue of that mandate. (Source of this assertion is prolonged discussions with sussman, one of the early svn authors.)
    - JoshTriplett 13y ago
      Not exactly, but close. Switching away from Git would require a tool that is both compellingly better in some key way as well as not worse for any existing workflow. I have yet to see any other version control system that meets both of those criteria, even leaving aside the network effects of using the most popular system.
  - masklinn 13y ago
    > Does the standard branch workflow still expect you to have a separate repository and directory per branch? You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo). > I don't care about plugins That's stupid, mercurial is very much about plugins: there are dozens of official plugins shipped in a standard mercurial install. > the standard workflow there is no such thing. > Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"? All of them are provided in the base install, you just have to enable the corresponding extensions.
    - ahomescu1 13y ago
      Mercurial's branches are server side (learned that the hard way the first time I made a branch), git's are client side.
      - tonfa 13y ago
        I think you are confused, git and hg branches are very similar (I'd try to help but I'm not sure what the confusion is).
        ahomescu1 13y ago
        It's simple: I created a local branch, worked on it, then tried the git workflow: 1. Switch to the default branch 2. Cherry-pick (with the equivalent hg command) my changes from my own branch into default 3. Push the changes to the remote repo What happened was that my local branch got pushed to the server, along with the default one. With git this wouldn't happen, it would push the local master to the remote master.
        tonfa 13y ago
        Yes, by default all branches are pushed, (you would have to use --force if it creates new heads, and --new-branch to push new named branches). But I really don't see how it makes anything "server" or "client" side. (and you could in any case decide to push just default: hg push -r default)
LukeHoersten 13y ago
I love the Mercurial community. We use Mercurial at work and I'm able to get instant support in IRC for any issue we have with an awesome signal/noise ratio. I'm glad Facebook is contributing back so much as well. My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. All anecdotal observations of course ;-)
- VladRussian2 13y ago
  >My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. GitHub is a consequence not the cause (ponder for a moment why there is no MercurialHub...) It is about ability to choose best source control tool for multi-versioned distributed concurrent development. Open source devs have such choice while corporate ones - no. FB choosing Mercurial tells a lot about environment there.
  - Timmmmbob 13y ago
    That makes no sense. Facebook specifically said they might have chosen git! They had the same choices as any developer except that a) they don't care about github, and b) they wanted to modify the VCS to support massive scale. Apparently out of git and mercurial, mercurial is easier to extend.
  - Uchikoma 13y ago
    90% of people choose Git because everyone else uses it. Noone has been fired for buying IBM.
    - Shamanmuni 13y ago
      I suspect many of us use Git because it was made by Linus Torvalds ( I think everyone agrees that he is a great developer) and is used in the Linux kernel. If it can handle that, then you can be pretty sure it will handle whatever you throw at it (unless you are Facebook, it seems). Add Github to that mix and you can see why so many developers sleep like babies at night. Of course it's not rigorous, but the choice is not irrational.
xkarga00 13y ago
"Our engineers were comfortable with Git and we preferred to stay with a familiar tool, so we took a long, hard look at improving it to work at scale. After much deliberation, we concluded that Git's internals would be difficult to work with for an ambitious scaling project." http://www.techradar.com/news/software/how-open-source-changed-google-and-how-google-changed-open-source-1206582 http://www.techradar.com/news/software/how-open-source-chang... "And then Git itself wasn't working for us anymore because it wasn't scaling when we'd have an operating system release. So we ended up hiring most of the Git team - there's like only one or two core committers now for Git who don't work at Google,"
- durin42 13y ago
  Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.
  - daleharvey 13y ago
    I have seen presentations mentioning google development done as a single monolothic repository http://www.infoq.com/presentations/Development-at-Google http://www.infoq.com/presentations/Development-at-Google
    - tonfa 13y ago
      This monolithic repo isn't a git repo.
      - spiffytech 13y ago
        Correct. See https://news.ycombinator.com/item?id=7020859 https://news.ycombinator.com/item?id=7020859
  - Timmmmbob 13y ago
    Android is only split up into smaller repositories because git can't handle having it all in one.
Touche 13y ago
> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?
- kyrra 13y ago
  From the article: >Splitting it up would make large, atomic refactorings more difficult. If you split the code base up into multiple projects, you need some sort of meta project to link them all together if you need to deploy it as a single entity. Doing a meta project isn't exactly easy, so it tends to be better to do a single project if everything is all tightly coupled still. The project could be split up into smaller modules that are linked via shared common interfaces, but takes more time to maintain.
  - Touche 13y ago
    What is a wrong with submodules or subtree for a meta project? Or better yet, package management.
- dudus 13y ago
  On the other hand the Mercurial modularity was key for the migration.
- georgemcbay 13y ago
  I know nothing of the code Facebook is hosting, but one fairly obvious general benefit of having things in a single repository is that you can use relative paths to link dependent projects, like if you have a directory structure such as: libs libs/mylib apps apps/myapp then myapp can reference library mylib via ../../libs/mylib If all of these things are in separate repositories, all bets are off about how their paths relate to each other on the local filesystem, resulting in more complicated build procedures.
  - markcampbell 13y ago
    There's a solution to this. Git submodules.
kisstheblade 13y ago
I wonder what they develop at facebook at that rate, and with such a huge codebase. The main site seems quite/relatively simple in functionality (and doesn't change noticeably very often). Is all that coding going towards "better" advertising :)
mindjiver 13y ago
At a previous job I migrated a quite large code base to git. IIRC it was at least 500k files and a couple of 10MLOCs. We had the same "scaling" issues Facebook mentioned here when trying to place all of this inside one repository. So we ended up going with submodules for this. Another idea was to perhaps enable re-use of repositories and/or disconnect old legacy code this way. We did took a quick look at Mercurial but since lots of the upstream tools we used was using Git (linux, uboot, yocto, etc) it was an obvious choice. I seem to recall there being two hg extension that where of interest at the time (2010-ish), one to add inotify support and another to store large files outside the repository (hg-largefiles?). Seem like Facebook's approach to the lstat(2) issue with watchman [1] is to use inotify on Linux. This has been discussed a couple of times for git as well but nothing has come of it so far [2]. [1] https://github.com/facebook/watchman https://github.com/facebook/watchman [2] http://git.661346.n2.nabble.com/inotify-to-minimize-stat-calls-td7577352.html http://git.661346.n2.nabble.com/inotify-to-minimize-stat-cal...
natrius 13y ago
Branches are my biggest pain point in Mercurial. What branching workflow does Facebook use?
- k_bx 13y ago
  Why not use standard mercurial branches? At least you're able to say in which branch commit was done and draw a clean history for them.
  - natrius 13y ago
    I do use standard Mercurial branches, and I vastly prefer Git's model. I ask because there might be a better way to use them that I've overlooked.
    - masklinn 13y ago
      Mercurial's equivalent to git branches (movable pointer to a commit rather than embedded commit metadata) is bookmarks.
      - fhd2 13y ago
        From what I've seen, bookmarks are not "branches" in any sense, really just pointers to commits. I've tried several times to use bookmarks for feature branches (read: branches developed in parallel to each other and the default branch). I thought I just can't figure it out, but it really seems impossible at this point.
        masklinn 13y ago
        > From what I've seen, bookmarks are not "branches" in any sense, really just pointers to commits. That's exactly what git branches are.
        michal8181 13y ago
        Branches in git are exactly pointers to graph. If you delete such a pointer you loose your branch. Even worse - with next git gc you will loose your data as parts of graph with no pointer is assumed to be dead by git.
Crito 13y ago
I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: <old-comment> Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain point, with a large enough codebase (and, critically, enough throughput) you start to realize that you are about to hit a brick wall. With perforce, this starts to manifest with service brownouts. With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team. Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every object in the DAG), even ignoring throughput. Perforce is probably good enough for Facebook in the present, but if you are a company that large and if you are forward looking, it becomes apparent that with existing version control technology, "one repo per company" is not a long term solution. Even "one repo per department". You can split it even further, but what you realize is that you are developing infrastructure that allows you to use many repos (for instance your build servers and internal code search/browsing tools will now need to understand that concept) but you are losing many of the benefits of Perforce. While you are in the process of adapting your infrastructure to wrap its mind around many repositories, it makes sense to allow dev teams to really take advantage of this splitting. Develop infrastructure that allows Perforce and git repos to coexist in the company, allowing dev teams to spin up new git repos for their every project at will. Done properly, git allows you to create massively scalable systems that you can count on supporting your companies needs for the foreseeable future. Smooth migration, migration that does not disrupt development, takes months (assuming the right initial conditions), so it is best to recognize the problem and start early, before service becomes disrupted. If I understand Facebook's situation currently, they are still in the "try to make Mercurial scale" stage of denial, burning developer time and effort to push back that first brick wall (the same one that git hits, though mercurial hits it after git hits it but before perforce hits it...) Here is a google presentation about extreme scaling with Perforce: http://www.perforce.com/sites/default/files/still-all-one-server-perforce-scale.pdf http://www.perforce.com/sites/default/files/still-all-one-se... An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible. </old-comment> I guess the TL;DR here is that I think it is great that Facebook is making a single Mercurial repository work for their purposes right now, but I think they are kidding themselves if they think that is a long term solution. They are doing a pretty good job of making Mercurial scale like Perforce can scale, but that will only work for them for so long. (In the above comment I talk about building a scalable system with git ("with" git, rather than making git itself scale), but the same can of course be done with mercurial instead of git. I don't mean this to be a comment suggesting that they should use git instead of mercurial.)
moron4hire 13y ago
> "Our code base has grown organically and its internal dependencies are very complex." That's a polite way of saying "we write shitty code without any sort of plan." > "Splitting it up would make large, atomic refactorings more difficult" Actually, it's the other way around. Modularity tends to obviate the need for large, atomic refactorings. And what, exactly, is the meaning of these graphs? This is leading me to believe that being a developer at Facebook is about quantity over quality.
- ZenoArrow 13y ago
  To be honest, whilst we have no way to accurately determine whether the code is a mess without a chance to see it, the most surprising line of this article (in my opinion) was that the code base was larger than the Linux kernel. I'm not seeing anything on the front end that would warrant such complexity, guessing a large chunk of the code base is server code. Would be interested in reading a summary of the components of the Facebook code base.
  - plorkyeran 13y ago
    I suspect that the kernel is one of the only things running on Facebook's servers that they didn't write from scratch. Alexandrescu has mentioned that a 1% speedup to HHVM saves FB about $100k per year, and at that sort of scale it's pretty easy for reinventing every wheel to make sense.
    - moron4hire 13y ago
      right, so why can't projects like that exiat in their own repository? what keeps HHVM and Facebook tightly coupled?
      - pydave 13y ago
        Some of them are. > We already have some of the easily separable projects in separate repositories, like HPHP http://article.gmane.org/gmane.comp.version-control.git/189784 http://article.gmane.org/gmane.comp.version-control.git/1897... Presumably some parts aren't so isolated.
danielrhodes 13y ago
"We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Even at our current scale, we often make large changes throughout our code base, and having a single repository is useful for continuous modernization. Splitting it up would make large, atomic refactorings more difficult. On top of that, the idea that the scaling constraints of our source control system should dictate our code structure just doesn't sit well with us." I remember reading about their Git troubles awhile ago, and I still don't buy this argument that it is better to have one large repository. One reason modularization is important is for the precise reason they are trying to get around it: removing the ability to make large scale changes easy and thus increasing reliability. However, my understanding is that their desire to have one large repo is reflective of their their "move fast and break things" philosophy, which means not being afraid of making large scale changes. So I would be interested in hearing how they mitigate the obvious downsides given how many people they have committing to their codebase. It seems like you would just end up having to create constraints in other ways, so which constraints end up being the lesser of two evils?
- zaphar 13y ago
  We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywhere run all the tests for the entire codebase and then submit the changes thus accomplishing an API change safely in weeks that might take months otherwise with higher risk. Keep in mind that the API change equals real money in networking cost so time to ship is a very real factor. It sounds like facebook also has a very real time to ship need for even core libraries.
  - igravious 13y ago
    What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?
deleted 13y ago
[deleted]
ed_blackburn 13y ago
I wonder what they use at Microsoft. For their sake I hope they don't subject their own engineers to TFS.
- mindjiver 13y ago
  I read somewhere(?) that they where running Perforce with some internal tooling built on top of it.
  - boyter 13y ago
    I saw/heard that as well (hopefully someone can come up with the link). Things like Windows/Office apparently are in a custom perforce. I believe that they do use TFS on a lot of the internal projects though and that Visual Studio is now done in TFS.
    - Maarten88 13y ago
      Here is a link about TFS use inside Microsoft: http://blogs.msdn.com/b/visualstudioalm/archive/2013/08/20/tfs-internal-usage-statistics-1st-half-cy-2013.aspx http://blogs.msdn.com/b/visualstudioalm/archive/2013/08/20/t... Apparently they use several large TFS instances.
    - robaato 13y ago
      The following is a presentation in 2008 by Richard Erwin of Microsoft at the BCS CMSG (Configuration Management Specialist Group): http://bcs-cmsg.org.uk/events/e20081124/2008-11-24-agile-scm-erwin.pdf http://bcs-cmsg.org.uk/events/e20081124/2008-11-24-agile-scm... Lots of good stuff there, but a key one is slide 8. In the question session Richard confirmed that Source Depot (the custom version of Perforce) was at that time still used for the source management although TFS was used for bug tracking etc for Office and Windows. Don't know what has happened in the intervening years...
  - adamnemecek 13y ago
    This is correct. http://programmers.stackexchange.com/a/85891 http://programmers.stackexchange.com/a/85891
agwa 13y ago
For some context of why Facebook choose Hg over Git, here's the mailing list thread where Facebook initially reached out to the Git developers: http://thread.gmane.org/gmane.comp.version-control.git/189776 http://thread.gmane.org/gmane.comp.version-control.git/18977...
- novaleaf 13y ago
  I personally love mercurial (simpler than git), and have been a bit nervious over the last year or so with the mindshare shift to git. So hearing about this (FB all in with hg) ensures that hg won't be falling behind... at least in the nearterm.
  - dgesang 13y ago
    Don't believe the hype. hg will be around for some time.
    - twic 13y ago
      Mercurial is the new FreeBSD.
  - CmonDev 13y ago
    Don't follow the lemmings, choose your own path!
  - tshepang 13y ago
    I created this list to help me stop worrying: http://tshepang.net/major-projects-using-mercurial http://tshepang.net/major-projects-using-mercurial.
- randartie 13y ago
  The discussion appears deleted on that thread
  - novaleaf 13y ago
    fyi, it was working about 8hrs ago. broken now. guess someone didn't like the inuendo ;)
boklm0 13y ago
According to this page, Mercurial project leader is currently working for Facebook: http://mercurial.selenic.com/wiki/mpm http://mercurial.selenic.com/wiki/mpm
- marmoute 13y ago
  Facebook took the Mercurial path about 2 year ago, they hired some core devs one year later to secure and speed up development.
- CmonDev 13y ago
  Google bought Git devs, Facebook bought Mercurial devs - wise choice for companies that even order custom CPUs.
  - jordigh 13y ago
    There are some hg devs at Google too. :-)
    - CmonDev 13y ago
      Hedging to avoid the risk I guess.
rottyguy 13y ago
Any comments on the build system being used for a monolithic hive of this size?
greatsuccess 13y ago
Partitioning is the answer. In a repo that size 99% of the history is useless to anyone. You wouldnt manage a database like this so why force SCM down this path? If they used git with say only the last year of history in it they would be having zero issues.
- ryanobjc 13y ago
  While true, you only need 1% of the history, the problem is WHICH 1% and when? Tracking down bugs in the history is a huge deal, and its often absolutely critical. And yes, regressions go past 1 year.
- exDM69 13y ago
  Partitioning may be the answer but this is a huge problem for corporations like Facebook (and the one I am working for). If things have been done with one giant repo from day one, splitting it is going to be a major engineering/political/social problem when there's thousands of engineers working on the code base and you can't just shut down business for the duration of the migration.
riddlemethat 13y ago
Can anyone explain to me how Mercurial is used at Facebook in conjunction with Murder (https://github.com/lg/murder https://github.com/lg/murder)?
voidr 13y ago
They didn't really scale Mercurial, they basically took it and replaced a lot of it's functionality with remote services. I understand their reasoning that technology shouldn't dictate the way they develop stuff, however I think splitting the project up and using submodules would have been a cleaner approach. If refactoring everything is something you do all the time, you might be doing it wrong.
cool-RR 13y ago
I wonder whether putting all the code on a ramdisk (with backup of course) is feasible. If so it might be a very cheap solution.
- exDM69 13y ago
  No, using a ramdisk these days usually makes things worse, not better. The reason is that the operating system already holds as much of the filesystem in caches (in RAM) as possible. So as long as you have enough RAM in your system, files will be cached and the result is better than using a ramdisk.

360 points by jordigh 13y ago

Mercurial has seriously improved over the past couple of years. If you tried mercurial a few years ago and were scared away due to speed or functionality issues, you might want to give it another shot.

JoshTriplett 13y ago
Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches? Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"?
- ngoldbaum 13y ago
  Mercurial refers to git-like lightweight branches as bookmarks [1]. Branches are still permanent tags in the repository. Commit --amend is now part of base mercurial. Rebase is available as an extension that is shipped with the client, like most advanced functionality. I don't know what add -p does. [1] http://mercurial.selenic.com/wiki/Bookmarks/ http://mercurial.selenic.com/wiki/Bookmarks/
  - durin42 13y ago
    "add -p" is basically the 'hg record' extension.
    - _ikke_ 13y ago
      It's more like `git commit -p`. If you abort `hg record`, the selection you've made is lost.
- mikegioia 13y ago
  I work in several branches in the same file system directory. Branching in mercurial is as simple as "hg branch branchname", and switching branches is just "hg update branchname". As long as you make sure you commit everything you can easily work in one directory. Merging changes across branches is easy too, as you only need to specify the revision number.
  - ahomescu1 13y ago
    If you push while having open local branches, they get pushed to the remote (or at least that's how it was last time I used them). That's not what you want, in most cases (git branches are local).
    - rbehrends 13y ago
      Mercurial won't push new branches unless you specify -f or --new-branch. You can selectively push only the branches you want with hg push -r branch_or_revision (hg push -r . for the current revision). You can also hide branches using phases, so they won't be pushed even with -f or --new-branch.
      - ahomescu1 13y ago
        The guide from [1] confirms it works the way I saw it (it pushes all branches): Mercurial will push/pull all branches by default, while git will push/pull only the current branch. Did this behavior change in recent versions of hg? 1 - http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/ http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me...
        rbehrends 13y ago
        Mercurial will attempt to push all branches. It will then immediately stop with an error message unless you specify -f or --new-branch if you're trying to push a branch that does not already exist in the remote repository (or even to create a new head in an existing branch). In order to get rid of the error message you need to either (1) explicitly tell Mercurial to push all branches, (2) specify which existing branch(es)/revision(s) you want to push (they must already exist remotely), or (3) hide the branch with hg phase secret -f. This means that your local branches will not leak into the remote repository unless you specifically instruct Mercurial to do so. Of these, only the phase approach is new (2.1+). The rest hasn't changed.
- jordigh 13y ago
  > if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. There is no real standard workflow. There are tools in place to build whatever you want. This is old, but remarkably still relevant: http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/ http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me... If you want git-like branches, Mercurial bookmarks are close, but not quite the same. They have some intentional differences, which make sense and do not impede you, but will not make sense if you absolutely want them to behave like git branches. > "commit --amend", "rebase -i", and "add -p"? hg commit --amend, hg histedit, hg record (but hg crecord is much nicer). They're built-in, but you have to flip them on the last two with a config switch.
  - JoshTriplett 13y ago
    That post helped, and bookmarks might get the job done, but the mercurial culture doesn't seem to encourage them compared to branches. As a result, you tend to find a lot more separate mercurial repositories than single repos with bookmarks, and a lot more named branches with hard-to-eliminate commits as well (since even "closing" a branch doesn't really get rid of it). > hg commit --amend, hg histedit, hg record (but hg crecord is much nicer). > They're built-in, but you have to flip them on the last two with a config switch. And that's the fundamental attitude that I can't stand in mercurial's culture: "careful, you might accidentally throw away that information you specifically wanted to throw away". Those commands aren't available by default because the culture doesn't encourage that workflow, so even if you use them for your own personal workflow you'll tend to find a lot more junk commits and forever-remembered branch names in the average mercurial repository.
    - mixmastamyk 13y ago
      I might have read this wrong, but safety for newbies doesn't sound all that bad to me.
      - JoshTriplett 13y ago
        History editing in git is not unsafe: because it's a part of the default set of tools, git offers functionality like the reflog to keep track of old history locally and avoid losing it. It's extremely difficult to permanently lose content in a git repository even using the history editing commands; it's always just a "git reflog" away.
- durin42 13y ago
  Note that extensions that ship with Mercurial are exactly as supported as the tool itself, and come with the same kinds of backwards compatibility guarantees. It's not worth worrying about rebase, histedit, and record being extensions - if you want them, turn them on and use them.
  - JoshTriplett 13y ago
    They're not exactly as supported, because they're not on by default, and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.
    - durin42 13y ago
      No, they're exactly as supported. What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shooting their foot off unexpectedly, and UI clutter. mq alone (which we actually recommend avoiding now if you don't really need it) adds a bunch of verbs to the command line interface. We've talked about having an 'hg hello' command or something that'd give you a proposed template hgrc that suggest you might want rebase/record/histedit/pager/color/progress, but nobody's worked out exactly what that should be like. Does something like that sound like it might be helpful?
      - mushishi 13y ago
        Could you elaborate on why to avoid mq?
        masklinn 13y ago
        It's a very powerful tool, but also a complex one, and one which brings in a bunch of new commands. There are a bunch of extension which provide nice UIs to various subsets of mq's full power, and are much simpler. If you need mq, use mq, but if you just need to e.g. do some history edition, use histedit.
- k_bx 13y ago
  Why do you think mercurial branches are not lightweight? Which operation (create, close, push, pull) on branches is noticeably slower than git's?
  - wtetzner 13y ago
    I don't think he was talking about performance. I think he meant lightweight in the sense that a branch (in git) is just a pointer to a commit. It's conceptually lightweight.
    - JoshTriplett 13y ago
      Both: it's conceptually lightweight, and because of that "pointer to a commit" structure it's also effectively free to create and move around branches.
      - k_bx 13y ago
        Why is mercurial branches ineffective to move around branches? If I'm not mistaken, if you want to move commits from one branch to another you do absolutely the same rebase.
    - k_bx 13y ago
      Well, mercurial branch is just a record in list of branches, so is it then also "conceptually lightweight"? What is "conceptually lightweight" anyway?
  - fhd2 13y ago
    According to #mercurial, branches (even closed ones) begin to noticeably slow down the repository when you reach around 2000. That's not at all impossible to reach with fine-grained feature branches. Bookmarks are really just pointers to commits, no branches in any sense, so they cannot be used for feature branches either. As a result, we simply don't use feature branches in Mercurial. It's literally the only issue I can think of.
    - 13y ago
- Uchikoma 13y ago
  Let me translate: "if this tool is not exactly like Git, I will not use it".
  - gecko 13y ago
    That sentiment comprises a truly mindblowing number of "bug" reports that hit the Mercurial mailing list every month. I honestly wonder sometimes if I'm simply forgetting how CVS users did the same thing when Subversion shipped and I forgot about it/missed it, or what.
    - durin42 13y ago
      svn was designed to be "CVS done right," which informed a lot of the design and workflow to the point that some CVS misfeatures only got carried through by virtue of that mandate. (Source of this assertion is prolonged discussions with sussman, one of the early svn authors.)
  - JoshTriplett 13y ago
    Not exactly, but close. Switching away from Git would require a tool that is both compellingly better in some key way as well as not worse for any existing workflow. I have yet to see any other version control system that meets both of those criteria, even leaving aside the network effects of using the most popular system.
- masklinn 13y ago
  > Does the standard branch workflow still expect you to have a separate repository and directory per branch? You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo). > I don't care about plugins That's stupid, mercurial is very much about plugins: there are dozens of official plugins shipped in a standard mercurial install. > the standard workflow there is no such thing. > Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"? All of them are provided in the base install, you just have to enable the corresponding extensions.
  - ahomescu1 13y ago
    Mercurial's branches are server side (learned that the hard way the first time I made a branch), git's are client side.
    - tonfa 13y ago
      I think you are confused, git and hg branches are very similar (I'd try to help but I'm not sure what the confusion is).
      - ahomescu1 13y ago
        It's simple: I created a local branch, worked on it, then tried the git workflow: 1. Switch to the default branch 2. Cherry-pick (with the equivalent hg command) my changes from my own branch into default 3. Push the changes to the remote repo What happened was that my local branch got pushed to the server, along with the default one. With git this wouldn't happen, it would push the local master to the remote master.
        tonfa 13y ago
        Yes, by default all branches are pushed, (you would have to use --force if it creates new heads, and --new-branch to push new named branches). But I really don't see how it makes anything "server" or "client" side. (and you could in any case decide to push just default: hg push -r default)

LukeHoersten 13y ago

I love the Mercurial community. We use Mercurial at work and I'm able to get instant support in IRC for any issue we have with an awesome signal/noise ratio. I'm glad Facebook is contributing back so much as well. My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. All anecdotal observations of course ;-)

VladRussian2 13y ago
>My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. GitHub is a consequence not the cause (ponder for a moment why there is no MercurialHub...) It is about ability to choose best source control tool for multi-versioned distributed concurrent development. Open source devs have such choice while corporate ones - no. FB choosing Mercurial tells a lot about environment there.
- Timmmmbob 13y ago
  That makes no sense. Facebook specifically said they might have chosen git! They had the same choices as any developer except that a) they don't care about github, and b) they wanted to modify the VCS to support massive scale. Apparently out of git and mercurial, mercurial is easier to extend.
- Uchikoma 13y ago
  90% of people choose Git because everyone else uses it. Noone has been fired for buying IBM.
  - Shamanmuni 13y ago
    I suspect many of us use Git because it was made by Linus Torvalds ( I think everyone agrees that he is a great developer) and is used in the Linux kernel. If it can handle that, then you can be pretty sure it will handle whatever you throw at it (unless you are Facebook, it seems). Add Github to that mix and you can see why so many developers sleep like babies at night. Of course it's not rigorous, but the choice is not irrational.

xkarga00 13y ago

"Our engineers were comfortable with Git and we preferred to stay with a familiar tool, so we took a long, hard look at improving it to work at scale. After much deliberation, we concluded that Git's internals would be difficult to work with for an ambitious scaling project." http://www.techradar.com/news/software/how-open-source-changed-google-and-how-google-changed-open-source-1206582 http://www.techradar.com/news/software/how-open-source-chang... "And then Git itself wasn't working for us anymore because it wasn't scaling when we'd have an operating system release. So we ended up hiring most of the Git team - there's like only one or two core committers now for Git who don't work at Google,"

durin42 13y ago
Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.
- daleharvey 13y ago
  I have seen presentations mentioning google development done as a single monolothic repository http://www.infoq.com/presentations/Development-at-Google http://www.infoq.com/presentations/Development-at-Google
  - tonfa 13y ago
    This monolithic repo isn't a git repo.
    - spiffytech 13y ago
      Correct. See https://news.ycombinator.com/item?id=7020859 https://news.ycombinator.com/item?id=7020859
- Timmmmbob 13y ago
  Android is only split up into smaller repositories because git can't handle having it all in one.

Touche 13y ago

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

kyrra 13y ago
From the article: >Splitting it up would make large, atomic refactorings more difficult. If you split the code base up into multiple projects, you need some sort of meta project to link them all together if you need to deploy it as a single entity. Doing a meta project isn't exactly easy, so it tends to be better to do a single project if everything is all tightly coupled still. The project could be split up into smaller modules that are linked via shared common interfaces, but takes more time to maintain.
- Touche 13y ago
  What is a wrong with submodules or subtree for a meta project? Or better yet, package management.
dudus 13y ago
On the other hand the Mercurial modularity was key for the migration.
georgemcbay 13y ago
I know nothing of the code Facebook is hosting, but one fairly obvious general benefit of having things in a single repository is that you can use relative paths to link dependent projects, like if you have a directory structure such as: libs libs/mylib apps apps/myapp then myapp can reference library mylib via ../../libs/mylib If all of these things are in separate repositories, all bets are off about how their paths relate to each other on the local filesystem, resulting in more complicated build procedures.
- markcampbell 13y ago
  There's a solution to this. Git submodules.

kisstheblade 13y ago

I wonder what they develop at facebook at that rate, and with such a huge codebase. The main site seems quite/relatively simple in functionality (and doesn't change noticeably very often). Is all that coding going towards "better" advertising :)

mindjiver 13y ago

At a previous job I migrated a quite large code base to git. IIRC it was at least 500k files and a couple of 10MLOCs. We had the same "scaling" issues Facebook mentioned here when trying to place all of this inside one repository. So we ended up going with submodules for this. Another idea was to perhaps enable re-use of repositories and/or disconnect old legacy code this way. We did took a quick look at Mercurial but since lots of the upstream tools we used was using Git (linux, uboot, yocto, etc) it was an obvious choice. I seem to recall there being two hg extension that where of interest at the time (2010-ish), one to add inotify support and another to store large files outside the repository (hg-largefiles?). Seem like Facebook's approach to the lstat(2) issue with watchman [1] is to use inotify on Linux. This has been discussed a couple of times for git as well but nothing has come of it so far [2]. [1] https://github.com/facebook/watchman https://github.com/facebook/watchman [2] http://git.661346.n2.nabble.com/inotify-to-minimize-stat-calls-td7577352.html http://git.661346.n2.nabble.com/inotify-to-minimize-stat-cal...

natrius 13y ago

Branches are my biggest pain point in Mercurial. What branching workflow does Facebook use?

k_bx 13y ago
Why not use standard mercurial branches? At least you're able to say in which branch commit was done and draw a clean history for them.
- natrius 13y ago
  I do use standard Mercurial branches, and I vastly prefer Git's model. I ask because there might be a better way to use them that I've overlooked.
  - masklinn 13y ago
    Mercurial's equivalent to git branches (movable pointer to a commit rather than embedded commit metadata) is bookmarks.
    - fhd2 13y ago
      From what I've seen, bookmarks are not "branches" in any sense, really just pointers to commits. I've tried several times to use bookmarks for feature branches (read: branches developed in parallel to each other and the default branch). I thought I just can't figure it out, but it really seems impossible at this point.
      - masklinn 13y ago
        > From what I've seen, bookmarks are not "branches" in any sense, really just pointers to commits. That's exactly what git branches are.
      - michal8181 13y ago
        Branches in git are exactly pointers to graph. If you delete such a pointer you loose your branch. Even worse - with next git gc you will loose your data as parts of graph with no pointer is assumed to be dead by git.

Crito 13y ago

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: <old-comment> Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain point, with a large enough codebase (and, critically, enough throughput) you start to realize that you are about to hit a brick wall. With perforce, this starts to manifest with service brownouts. With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team. Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every object in the DAG), even ignoring throughput. Perforce is probably good enough for Facebook in the present, but if you are a company that large and if you are forward looking, it becomes apparent that with existing version control technology, "one repo per company" is not a long term solution. Even "one repo per department". You can split it even further, but what you realize is that you are developing infrastructure that allows you to use many repos (for instance your build servers and internal code search/browsing tools will now need to understand that concept) but you are losing many of the benefits of Perforce. While you are in the process of adapting your infrastructure to wrap its mind around many repositories, it makes sense to allow dev teams to really take advantage of this splitting. Develop infrastructure that allows Perforce and git repos to coexist in the company, allowing dev teams to spin up new git repos for their every project at will. Done properly, git allows you to create massively scalable systems that you can count on supporting your companies needs for the foreseeable future. Smooth migration, migration that does not disrupt development, takes months (assuming the right initial conditions), so it is best to recognize the problem and start early, before service becomes disrupted. If I understand Facebook's situation currently, they are still in the "try to make Mercurial scale" stage of denial, burning developer time and effort to push back that first brick wall (the same one that git hits, though mercurial hits it after git hits it but before perforce hits it...) Here is a google presentation about extreme scaling with Perforce: http://www.perforce.com/sites/default/files/still-all-one-server-perforce-scale.pdf http://www.perforce.com/sites/default/files/still-all-one-se... An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible. </old-comment> I guess the TL;DR here is that I think it is great that Facebook is making a single Mercurial repository work for their purposes right now, but I think they are kidding themselves if they think that is a long term solution. They are doing a pretty good job of making Mercurial scale like Perforce can scale, but that will only work for them for so long. (In the above comment I talk about building a scalable system with git ("with" git, rather than making git itself scale), but the same can of course be done with mercurial instead of git. I don't mean this to be a comment suggesting that they should use git instead of mercurial.)

moron4hire 13y ago

> "Our code base has grown organically and its internal dependencies are very complex." That's a polite way of saying "we write shitty code without any sort of plan." > "Splitting it up would make large, atomic refactorings more difficult" Actually, it's the other way around. Modularity tends to obviate the need for large, atomic refactorings. And what, exactly, is the meaning of these graphs? This is leading me to believe that being a developer at Facebook is about quantity over quality.

ZenoArrow 13y ago
To be honest, whilst we have no way to accurately determine whether the code is a mess without a chance to see it, the most surprising line of this article (in my opinion) was that the code base was larger than the Linux kernel. I'm not seeing anything on the front end that would warrant such complexity, guessing a large chunk of the code base is server code. Would be interested in reading a summary of the components of the Facebook code base.
- plorkyeran 13y ago
  I suspect that the kernel is one of the only things running on Facebook's servers that they didn't write from scratch. Alexandrescu has mentioned that a 1% speedup to HHVM saves FB about $100k per year, and at that sort of scale it's pretty easy for reinventing every wheel to make sense.
  - moron4hire 13y ago
    right, so why can't projects like that exiat in their own repository? what keeps HHVM and Facebook tightly coupled?
    - pydave 13y ago
      Some of them are. > We already have some of the easily separable projects in separate repositories, like HPHP http://article.gmane.org/gmane.comp.version-control.git/189784 http://article.gmane.org/gmane.comp.version-control.git/1897... Presumably some parts aren't so isolated.

danielrhodes 13y ago

"We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Even at our current scale, we often make large changes throughout our code base, and having a single repository is useful for continuous modernization. Splitting it up would make large, atomic refactorings more difficult. On top of that, the idea that the scaling constraints of our source control system should dictate our code structure just doesn't sit well with us." I remember reading about their Git troubles awhile ago, and I still don't buy this argument that it is better to have one large repository. One reason modularization is important is for the precise reason they are trying to get around it: removing the ability to make large scale changes easy and thus increasing reliability. However, my understanding is that their desire to have one large repo is reflective of their their "move fast and break things" philosophy, which means not being afraid of making large scale changes. So I would be interested in hearing how they mitigate the obvious downsides given how many people they have committing to their codebase. It seems like you would just end up having to create constraints in other ways, so which constraints end up being the lesser of two evils?

zaphar 13y ago
We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywhere run all the tests for the entire codebase and then submit the changes thus accomplishing an API change safely in weeks that might take months otherwise with higher risk. Keep in mind that the API change equals real money in networking cost so time to ship is a very real factor. It sounds like facebook also has a very real time to ship need for even core libraries.
- igravious 13y ago
  What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?

deleted 13y ago

[deleted]

ed_blackburn 13y ago

I wonder what they use at Microsoft. For their sake I hope they don't subject their own engineers to TFS.

mindjiver 13y ago
I read somewhere(?) that they where running Perforce with some internal tooling built on top of it.
- boyter 13y ago
  I saw/heard that as well (hopefully someone can come up with the link). Things like Windows/Office apparently are in a custom perforce. I believe that they do use TFS on a lot of the internal projects though and that Visual Studio is now done in TFS.
  - Maarten88 13y ago
    Here is a link about TFS use inside Microsoft: http://blogs.msdn.com/b/visualstudioalm/archive/2013/08/20/tfs-internal-usage-statistics-1st-half-cy-2013.aspx http://blogs.msdn.com/b/visualstudioalm/archive/2013/08/20/t... Apparently they use several large TFS instances.
  - robaato 13y ago
    The following is a presentation in 2008 by Richard Erwin of Microsoft at the BCS CMSG (Configuration Management Specialist Group): http://bcs-cmsg.org.uk/events/e20081124/2008-11-24-agile-scm-erwin.pdf http://bcs-cmsg.org.uk/events/e20081124/2008-11-24-agile-scm... Lots of good stuff there, but a key one is slide 8. In the question session Richard confirmed that Source Depot (the custom version of Perforce) was at that time still used for the source management although TFS was used for bug tracking etc for Office and Windows. Don't know what has happened in the intervening years...
- adamnemecek 13y ago
  This is correct. http://programmers.stackexchange.com/a/85891 http://programmers.stackexchange.com/a/85891

agwa 13y ago

For some context of why Facebook choose Hg over Git, here's the mailing list thread where Facebook initially reached out to the Git developers: http://thread.gmane.org/gmane.comp.version-control.git/189776 http://thread.gmane.org/gmane.comp.version-control.git/18977...

novaleaf 13y ago
I personally love mercurial (simpler than git), and have been a bit nervious over the last year or so with the mindshare shift to git. So hearing about this (FB all in with hg) ensures that hg won't be falling behind... at least in the nearterm.
- dgesang 13y ago
  Don't believe the hype. hg will be around for some time.
  - twic 13y ago
    Mercurial is the new FreeBSD.
- CmonDev 13y ago
  Don't follow the lemmings, choose your own path!
- tshepang 13y ago
  I created this list to help me stop worrying: http://tshepang.net/major-projects-using-mercurial http://tshepang.net/major-projects-using-mercurial.
randartie 13y ago
The discussion appears deleted on that thread
- novaleaf 13y ago
  fyi, it was working about 8hrs ago. broken now. guess someone didn't like the inuendo ;)

boklm0 13y ago

According to this page, Mercurial project leader is currently working for Facebook: http://mercurial.selenic.com/wiki/mpm http://mercurial.selenic.com/wiki/mpm

marmoute 13y ago
Facebook took the Mercurial path about 2 year ago, they hired some core devs one year later to secure and speed up development.
CmonDev 13y ago
Google bought Git devs, Facebook bought Mercurial devs - wise choice for companies that even order custom CPUs.
- jordigh 13y ago
  There are some hg devs at Google too. :-)
  - CmonDev 13y ago
    Hedging to avoid the risk I guess.

rottyguy 13y ago

Any comments on the build system being used for a monolithic hive of this size?

greatsuccess 13y ago

Partitioning is the answer. In a repo that size 99% of the history is useless to anyone. You wouldnt manage a database like this so why force SCM down this path? If they used git with say only the last year of history in it they would be having zero issues.

ryanobjc 13y ago
While true, you only need 1% of the history, the problem is WHICH 1% and when? Tracking down bugs in the history is a huge deal, and its often absolutely critical. And yes, regressions go past 1 year.
exDM69 13y ago
Partitioning may be the answer but this is a huge problem for corporations like Facebook (and the one I am working for). If things have been done with one giant repo from day one, splitting it is going to be a major engineering/political/social problem when there's thousands of engineers working on the code base and you can't just shut down business for the duration of the migration.

riddlemethat 13y ago

Can anyone explain to me how Mercurial is used at Facebook in conjunction with Murder (https://github.com/lg/murder https://github.com/lg/murder)?

voidr 13y ago

They didn't really scale Mercurial, they basically took it and replaced a lot of it's functionality with remote services. I understand their reasoning that technology shouldn't dictate the way they develop stuff, however I think splitting the project up and using submodules would have been a cleaner approach. If refactoring everything is something you do all the time, you might be doing it wrong.

cool-RR 13y ago

I wonder whether putting all the code on a ramdisk (with backup of course) is feasible. If so it might be a very cheap solution.

exDM69 13y ago
No, using a ramdisk these days usually makes things worse, not better. The reason is that the operating system already holds as much of the filesystem in caches (in RAM) as possible. So as long as you have enough RAM in your system, files will be cached and the result is better than using a ramdisk.