29 ms·

One of the crazier things a L4 meta colleague of mine told me, that I still don’t believe entirely, is that meta pretty much has their own fork of everything, e

One of the crazier things a L4 meta colleague of mine told me, that I still don’t believe entirely, is that meta pretty much has their own fork of everything, even tools like git. is this true?

sdenton4 2y ago
It wouldn't be terribly surprising. Forking everything provides a liiiitle bit of protection against things like the 'left pad' incident.
- deleted 2y ago
  [deleted]
- 3eb7988a1663 2y ago
  Left pad was from the creator pulling the code from the public source forge, not from a destructive code change. I assume all of the big tech companies host internal mirrors of every single code dependency + tooling. Otherwise they could not guarantee that they can build all of their code.
tqi 2y ago
Facebook actually doesn't use git, they use mercurial (https://graphite.dev/blog/why-facebook-doesnt-use-git https://graphite.dev/blog/why-facebook-doesnt-use-git). That decision is also illustrative of why they end up forking most things - Facebook's usage patterns at the far extreme end for almost any tool, and things thats are non-issues with fewer engineers or a smaller codebase become complete blockers.
- kridsdale3 2y ago
  Yes when I used to talk about this to interviewees, I described that every tool people commonly use is somewhere on the Big-O curves for scaling. Most of the time we don't really care if a tool is O(n) or O(10 n) or whatever. At Meta, N tends to be hundreds of billions to hundreds of trillions. So your algorithm REALLY matters. And git has a Big-O that is worse than Mercurial, so we had to switch.
  - deleted 2y ago
    [deleted]
  - steventhedev 2y ago
    I'm gonna disagree with you there. The difference was with stat patterns, and the person at Facebook who ran the tests had something wrong with the disk setup that was causing it to run slowly. They ignored multiple responses that reproduced very different results. Nail in the coffin on this was a benchmark GitHub ran two years ago that got the results that FB should have: git status within seconds. Facebook didn't use mercurial because of big O, they used it because of hubris and a bad disk config.
    - deadmutex 2y ago
      This sounds plausible, but would love a source
      - steventhedev 2y ago
        I should probably just write it up into a post, but the git mailing list at the time is the source (I remember reading it from the side a few months after convincing our VP R&D to switch from svn to git). We were chuckling around the same time that FB had to reallocate the stack on Galaxy S2 phones because they were somehow unaware of proguard or unable to have it work properly with their codegen. Anyways: 1. Github benchmark: https://github.blog/engineering/infrastructure/improve-git-monorepo-performance-with-a-file-system-monitor/ https://github.blog/engineering/infrastructure/improve-git-m... 2. The original email thread: https://public-inbox.org/git/CB04005C.2C669%25joshua.redstone@fb.com/t/ https://public-inbox.org/git/CB04005C.2C669%25joshua.redston... 3. There's another email thread that gets linked everywhere - but in light of the prior thread, the numbers don't track: https://public-inbox.org/git/CB5074CF.3AD7A%25joshua.redstone@fb.com/t/ https://public-inbox.org/git/CB5074CF.3AD7A%25joshua.redston... I recall there being a message from someone either at AirBnB or Uber who mentioned that they have a similar monorepo but without the slow git status, but can't seem to find it now - it's likely on one of the other mailing list archives but didn't make it to this one. Point being that painting this as "the community was hostile" or "git is too slow for FB" is just disingenuous. The FB engineer barely communicated with the git team (at least publicly) and when there was communication, it was pushing a single benchmark that was deeply flawed, and then ignoring feedback on how to both improve the performance of slow blame, commit by repacking checkpoint packfiles (a one-off effort) and also ignoring feedback that the benchmark numbers didn't make sense in absolute terms.
- LarsDu88 2y ago
  They use sapling. An in-house clone of mercurial that was open sourced 2 years ago
- herval 2y ago
  FB uses mercurial _for most things_, but like any company that size, there's teams that use git and even teams that use perforce
jamra 2y ago
Meta doesn't use git. It uses mercurial. It does fork it because they have a huge monorepo. They created a concept of stacked commits which is a way of not having branches. Each commit is in a stack and then merged into master. Lots of things built for scaling.
ipsum2 2y ago
Yep. Zeus is a fork of Zookeeper, Hack is a fork of PHP, etc. It's usually needed to make it work with the internal environment. The few things that don't have forks are usually the open source projects like React or PyTorch, but even those have some custom features added to make it work with FB internals.
- gcr 2y ago
  This is also how things work at Google. Google also maintains a monorepo with "forks" of all software that they use. History diverges, but is occasionally synchronized for things like security updates etc.
  - zhengyi13 2y ago
    Am I completely off-base/confused thinking that the GFE originally started life (like back under csilver) as a fork of boa[0]? [0]: http://www.boa.org/ http://www.boa.org/
    - lacker 2y ago
      I thought it was GWS that originally started as a fork of boa.
      - zhengyi13 2y ago
        That's it, yes, thank you.
- grantsucceeded 2y ago
  Few companies experienced the explosive growth fb did, though many will claim to have done so. Hack made the existing codebase of php scale to insane levels while reaching escape velocity for the overall company to even attempt to transition away or shrink the php codebase, as i recall (i was an SRE, not a dev) zeus likewise.
  - ipsum2 2y ago

One of the crazier things a L4 meta colleague of mine told me, that I still don’t believe entirely, is that meta pretty much has their own fork of everything, even tools like git. is this true?

sdenton4 2y ago

It wouldn't be terribly surprising. Forking everything provides a liiiitle bit of protection against things like the 'left pad' incident.

deleted 2y ago
[deleted]
3eb7988a1663 2y ago
Left pad was from the creator pulling the code from the public source forge, not from a destructive code change. I assume all of the big tech companies host internal mirrors of every single code dependency + tooling. Otherwise they could not guarantee that they can build all of their code.

tqi 2y ago

Facebook actually doesn't use git, they use mercurial (https://graphite.dev/blog/why-facebook-doesnt-use-git https://graphite.dev/blog/why-facebook-doesnt-use-git). That decision is also illustrative of why they end up forking most things - Facebook's usage patterns at the far extreme end for almost any tool, and things thats are non-issues with fewer engineers or a smaller codebase become complete blockers.

kridsdale3 2y ago
Yes when I used to talk about this to interviewees, I described that every tool people commonly use is somewhere on the Big-O curves for scaling. Most of the time we don't really care if a tool is O(n) or O(10 n) or whatever. At Meta, N tends to be hundreds of billions to hundreds of trillions. So your algorithm REALLY matters. And git has a Big-O that is worse than Mercurial, so we had to switch.
- deleted 2y ago
  [deleted]
- steventhedev 2y ago
  I'm gonna disagree with you there. The difference was with stat patterns, and the person at Facebook who ran the tests had something wrong with the disk setup that was causing it to run slowly. They ignored multiple responses that reproduced very different results. Nail in the coffin on this was a benchmark GitHub ran two years ago that got the results that FB should have: git status within seconds. Facebook didn't use mercurial because of big O, they used it because of hubris and a bad disk config.
  - deadmutex 2y ago
    This sounds plausible, but would love a source
    - steventhedev 2y ago
      I should probably just write it up into a post, but the git mailing list at the time is the source (I remember reading it from the side a few months after convincing our VP R&D to switch from svn to git). We were chuckling around the same time that FB had to reallocate the stack on Galaxy S2 phones because they were somehow unaware of proguard or unable to have it work properly with their codegen. Anyways: 1. Github benchmark: https://github.blog/engineering/infrastructure/improve-git-monorepo-performance-with-a-file-system-monitor/ https://github.blog/engineering/infrastructure/improve-git-m... 2. The original email thread: https://public-inbox.org/git/CB04005C.2C669%25joshua.redstone@fb.com/t/ https://public-inbox.org/git/CB04005C.2C669%25joshua.redston... 3. There's another email thread that gets linked everywhere - but in light of the prior thread, the numbers don't track: https://public-inbox.org/git/CB5074CF.3AD7A%25joshua.redstone@fb.com/t/ https://public-inbox.org/git/CB5074CF.3AD7A%25joshua.redston... I recall there being a message from someone either at AirBnB or Uber who mentioned that they have a similar monorepo but without the slow git status, but can't seem to find it now - it's likely on one of the other mailing list archives but didn't make it to this one. Point being that painting this as "the community was hostile" or "git is too slow for FB" is just disingenuous. The FB engineer barely communicated with the git team (at least publicly) and when there was communication, it was pushing a single benchmark that was deeply flawed, and then ignoring feedback on how to both improve the performance of slow blame, commit by repacking checkpoint packfiles (a one-off effort) and also ignoring feedback that the benchmark numbers didn't make sense in absolute terms.
LarsDu88 2y ago
They use sapling. An in-house clone of mercurial that was open sourced 2 years ago
herval 2y ago
FB uses mercurial _for most things_, but like any company that size, there's teams that use git and even teams that use perforce

jamra 2y ago

Meta doesn't use git. It uses mercurial. It does fork it because they have a huge monorepo. They created a concept of stacked commits which is a way of not having branches. Each commit is in a stack and then merged into master. Lots of things built for scaling.

ipsum2 2y ago

Yep. Zeus is a fork of Zookeeper, Hack is a fork of PHP, etc. It's usually needed to make it work with the internal environment. The few things that don't have forks are usually the open source projects like React or PyTorch, but even those have some custom features added to make it work with FB internals.

gcr 2y ago
This is also how things work at Google. Google also maintains a monorepo with "forks" of all software that they use. History diverges, but is occasionally synchronized for things like security updates etc.
- zhengyi13 2y ago
  Am I completely off-base/confused thinking that the GFE originally started life (like back under csilver) as a fork of boa[0]? [0]: http://www.boa.org/ http://www.boa.org/
  - lacker 2y ago
    I thought it was GWS that originally started as a fork of boa.
    - zhengyi13 2y ago
      That's it, yes, thank you.
grantsucceeded 2y ago
Few companies experienced the explosive growth fb did, though many will claim to have done so. Hack made the existing codebase of php scale to insane levels while reaching escape velocity for the overall company to even attempt to transition away or shrink the php codebase, as i recall (i was an SRE, not a dev) zeus likewise.
- ipsum2 2y ago