7 ms·

Why Facebook doesn't use Git

53 points by fosterfriends 3y ago

koreadoge 3y ago
[flagged]
weikequ_ 3y ago
Very cool, thanks for sharing. Despite never actually hearing about 'stacked prs' before, I've often encountered the issue and it makes me wonder why a solution like this isn't actually part of Git itself.
gentleman11 3y ago
How much better was performance? Did people prefer or not prefer the new workflows?
- durhamg 3y ago
  The overall performance was massively better. Operations that would take minutes in Git in a large repo would take seconds (and eventually sub-seconds) in Facebook's Mercurial. Latency was a common complaint though. With Git in a small repo, commands can finish in milliseconds. With Facebook's Mercurial most commands took at least a few hundreds milliseconds (Python startup and module initialization alone could take that), which makes the CLI feel more sluggish than most people preferred. Beyond performance, a lot of design effort was put into the workflows as well. The resulting user experience is significantly different from the open source Mercurial experience and in the long run the vast majority of people preferred the new workflows. Even some of the die-hard Git fans who complained about the transition. Part of the motivation of open sourcing Sapling was because ex-employees wanted to use it outside the company. (Source: I was on the Facebook source control team for ten years.)
aleksiy123 3y ago
For some reason I thought part of the reason was ex-googlers at Facebook. But it actually seems that it was somewhat the other way around. Mercurial is used at Google partially because Facebook was already using it. Maybe someone has a more concrete timeline of the cross pollination of VCS between the two.
sneed_chucker 3y ago
Ctrl+F 'Sapling' Ctrl+F 'Eden' Yeah the author of this doesn't really know what's going on with regards to source control at Facebook/Meta. Meta doesn't actually use Mercurial and hasn't for several years now. They basically wrote their own custom centralized version control that uses Mercurial-inspired client commands in order to adapt to the scale of their massive repos. Actual Mercurial scales just as bad as ever if the repo is too large because that breaks a fundamental assumption of distributed version control. Massive repos simply don't work with the traditional model of "no master server" distributed version control. In DVCS, a client checkout is supposed to contain the entire repo metadata. That's why the git people told Facebook that they were doing it wrong. Anyways, stacked diffs are great but they're orthogonal to why Facebook doesn't use git. They use their own thing for the same reason that Google uses piper and why some places still use Perforce. They use it because they want to have massive monorepos and DVCS just doesn't work well for that.
- deleted 3y ago
  [deleted]
- somnium_sn 3y ago
  I prefer the term "derived" from Mercurial. It was Mercurial and slowly but steadily morphed into what's a custom VCS now.
- g1a55er 3y ago
  I'm not sure that this continues to be true, now that Git supports partial clones[1]. It's now a fully supported feature in Git to work with only a partial copy of the code and metadata. In my experience, this plus an fsmonitor[2] makes Git work fine on very, very large repos. This is how Microsoft scaled Git to work with their massive Windows monorepo[3]. (That particular article talks about the earlier work using Git VFS, but it eventually turned into these extensions that got upstreamed into Git, as documented in the Scalar repo[4].) [1] https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-... [2] https://github.blog/2022-06-29-improve-git-monorepo-performance-with-a-file-system-monitor/ https://github.blog/2022-06-29-improve-git-monorepo-performa... [3] https://arstechnica.com/information-technology/2017/02/microsoft-hosts-the-windows-source-in-a-monstrous-300gb-git-repository/ https://arstechnica.com/information-technology/2017/02/micro... [4] https://github.com/microsoft/scalar https://github.com/microsoft/scalar
akasakahakada 3y ago
[flagged]
ChrisArchitect 3y ago
Some recent related discussions on this tip: Ask HN: Can we do better than Git for version control? (December) https://news.ycombinator.com/item?id=38590080 https://news.ycombinator.com/item?id=38590080 I kind of killed Mercurial at Mozilla (November) https://news.ycombinator.com/item?id=38369433 https://news.ycombinator.com/item?id=38369433
neandrake 3y ago
I thought most of the reasons - not being welcomed to modifying git - were known already for someone who went looking, but seeing the quotes from ex facebookers is interesting. I’d be more interested in why facebook ditched mercurial for their own. Once they left more and more notable projects left, though for more than that as reason. The mercurial project is still under development but it feels like a ghost town. It’s a shame as I much prefer mercurial to git but it’s difficult to choose as an option when the integrations and tooling is lacking, and new tool development is less likely to integrate with mercurial when it has so little resources behind it.
al_borland 3y ago
Microsoft had issues git’s ability to scale. According to this[1] they found and submitted solutions to allow git to scale to their needs and worked with the git team. It sounds like Facebook wasn’t partnering with them to solve the problem? The article mentions Microsoft had repos with 6m files and says things now take seconds. I’m not personally dealing with repos of this size, but it sounds like this is a problem of the past, thanks to Microsoft. I have a hard time believing Facebook would be a bigger beast than Windows. [1] https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/ https://devblogs.microsoft.com/bharry/scaling-git-and-some-b...
- NBJack 3y ago
  I suspect you might be surprised. They keep almost everything in the same monorepo: every service, Facebook itself, the backend tools, many of the 'spinoff' experiences, the user data processing pipeline, the list goes on. All in one spot. Now, imagine 60-70k engineers, all wanting to have that sweet, sweet impact needed for good ratings and promotions. Add to that the fact that LOC and reviews are used as a metric for productivity. Windows has to eventually compile into a collection of binaries delivered (at least in part) to their various platforms. Facebook/Meta does not have that kind of limitation; as long as each piece can be extracted for its own purposes and deployed, everyone is 'happy'. Windows is estimated to be about 50M lines of code. Meta, in one post, simply suggests that it is "many times larger" than the Linux Kernel at 17M. [1] Is it all needed? Probably not. But it certainly exists. [1] https://engineering.fb.com/2014/01/07/core-infra/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-infra/scaling-mer...
- BobbyTables2 3y ago
  It’s remarkable they’re the same company that gave the world Sourcesafe…
  - pellenys 3y ago
    Shudder
- 0cf8612b2e1e 3y ago
btreecat 3y ago
Git and Mercurial came out the same year. Both were created due to changing licenses of BitKeeper and how this would affect Linux kernel development.

53 points by fosterfriends 3y ago

[flagged]

Very cool, thanks for sharing. Despite never actually hearing about 'stacked prs' before, I've often encountered the issue and it makes me wonder why a solution like this isn't actually part of Git itself.

gentleman11 3y ago

How much better was performance? Did people prefer or not prefer the new workflows?

durhamg 3y ago
The overall performance was massively better. Operations that would take minutes in Git in a large repo would take seconds (and eventually sub-seconds) in Facebook's Mercurial. Latency was a common complaint though. With Git in a small repo, commands can finish in milliseconds. With Facebook's Mercurial most commands took at least a few hundreds milliseconds (Python startup and module initialization alone could take that), which makes the CLI feel more sluggish than most people preferred. Beyond performance, a lot of design effort was put into the workflows as well. The resulting user experience is significantly different from the open source Mercurial experience and in the long run the vast majority of people preferred the new workflows. Even some of the die-hard Git fans who complained about the transition. Part of the motivation of open sourcing Sapling was because ex-employees wanted to use it outside the company. (Source: I was on the Facebook source control team for ten years.)

aleksiy123 3y ago

For some reason I thought part of the reason was ex-googlers at Facebook. But it actually seems that it was somewhat the other way around. Mercurial is used at Google partially because Facebook was already using it. Maybe someone has a more concrete timeline of the cross pollination of VCS between the two.

sneed_chucker 3y ago

Ctrl+F 'Sapling' Ctrl+F 'Eden' Yeah the author of this doesn't really know what's going on with regards to source control at Facebook/Meta. Meta doesn't actually use Mercurial and hasn't for several years now. They basically wrote their own custom centralized version control that uses Mercurial-inspired client commands in order to adapt to the scale of their massive repos. Actual Mercurial scales just as bad as ever if the repo is too large because that breaks a fundamental assumption of distributed version control. Massive repos simply don't work with the traditional model of "no master server" distributed version control. In DVCS, a client checkout is supposed to contain the entire repo metadata. That's why the git people told Facebook that they were doing it wrong. Anyways, stacked diffs are great but they're orthogonal to why Facebook doesn't use git. They use their own thing for the same reason that Google uses piper and why some places still use Perforce. They use it because they want to have massive monorepos and DVCS just doesn't work well for that.

deleted 3y ago
[deleted]
somnium_sn 3y ago
I prefer the term "derived" from Mercurial. It was Mercurial and slowly but steadily morphed into what's a custom VCS now.
g1a55er 3y ago
I'm not sure that this continues to be true, now that Git supports partial clones[1]. It's now a fully supported feature in Git to work with only a partial copy of the code and metadata. In my experience, this plus an fsmonitor[2] makes Git work fine on very, very large repos. This is how Microsoft scaled Git to work with their massive Windows monorepo[3]. (That particular article talks about the earlier work using Git VFS, but it eventually turned into these extensions that got upstreamed into Git, as documented in the Scalar repo[4].) [1] https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-... [2] https://github.blog/2022-06-29-improve-git-monorepo-performance-with-a-file-system-monitor/ https://github.blog/2022-06-29-improve-git-monorepo-performa... [3] https://arstechnica.com/information-technology/2017/02/microsoft-hosts-the-windows-source-in-a-monstrous-300gb-git-repository/ https://arstechnica.com/information-technology/2017/02/micro... [4] https://github.com/microsoft/scalar https://github.com/microsoft/scalar

akasakahakada 3y ago

[flagged]

ChrisArchitect 3y ago

Some recent related discussions on this tip: Ask HN: Can we do better than Git for version control? (December) https://news.ycombinator.com/item?id=38590080 https://news.ycombinator.com/item?id=38590080 I kind of killed Mercurial at Mozilla (November) https://news.ycombinator.com/item?id=38369433 https://news.ycombinator.com/item?id=38369433

neandrake 3y ago

I thought most of the reasons - not being welcomed to modifying git - were known already for someone who went looking, but seeing the quotes from ex facebookers is interesting. I’d be more interested in why facebook ditched mercurial for their own. Once they left more and more notable projects left, though for more than that as reason. The mercurial project is still under development but it feels like a ghost town. It’s a shame as I much prefer mercurial to git but it’s difficult to choose as an option when the integrations and tooling is lacking, and new tool development is less likely to integrate with mercurial when it has so little resources behind it.

al_borland 3y ago

Microsoft had issues git’s ability to scale. According to this[1] they found and submitted solutions to allow git to scale to their needs and worked with the git team. It sounds like Facebook wasn’t partnering with them to solve the problem? The article mentions Microsoft had repos with 6m files and says things now take seconds. I’m not personally dealing with repos of this size, but it sounds like this is a problem of the past, thanks to Microsoft. I have a hard time believing Facebook would be a bigger beast than Windows. [1] https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/ https://devblogs.microsoft.com/bharry/scaling-git-and-some-b...

NBJack 3y ago
I suspect you might be surprised. They keep almost everything in the same monorepo: every service, Facebook itself, the backend tools, many of the 'spinoff' experiences, the user data processing pipeline, the list goes on. All in one spot. Now, imagine 60-70k engineers, all wanting to have that sweet, sweet impact needed for good ratings and promotions. Add to that the fact that LOC and reviews are used as a metric for productivity. Windows has to eventually compile into a collection of binaries delivered (at least in part) to their various platforms. Facebook/Meta does not have that kind of limitation; as long as each piece can be extracted for its own purposes and deployed, everyone is 'happy'. Windows is estimated to be about 50M lines of code. Meta, in one post, simply suggests that it is "many times larger" than the Linux Kernel at 17M. [1] Is it all needed? Probably not. But it certainly exists. [1] https://engineering.fb.com/2014/01/07/core-infra/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-infra/scaling-mer...
BobbyTables2 3y ago
It’s remarkable they’re the same company that gave the world Sourcesafe…
- pellenys 3y ago
  Shudder
0cf8612b2e1e 3y ago

btreecat 3y ago

Git and Mercurial came out the same year. Both were created due to changing licenses of BitKeeper and how this would affect Linux kernel development.