5 ms·

As an engineer that worked both at google and Facebook, I vastly prefer googles monorepo on perforce. Combining that with citc was pretty solid way to develop y

As an engineer that worked both at google and Facebook, I vastly prefer googles monorepo on perforce. Combining that with citc was pretty solid way to develop your project. Hg is a bit of a nightmare in the wfh situation. Really slow, hangs for a long time if you haven't synced in a few days. Yes, im sure there are ways to tweak, but not sure if you can tweak them enough!

liveoneggs 6y ago
I can only imagine an attempt at the initial clone.. the hg client is ridiculously slow, memory-hungry, and full of dumb behavior that seems to assume local connections
- est31 6y ago
  hg has a mode to make initial clones faster by offering zipped compressed checkouts for download.
  - liveoneggs 6y ago
    it does range requests to these bundles, attempts to extract the partials while keeping the http connection alive using up GB of memory in the process, and then totally fails pretty regularly when it need to go back to the network. It makes no attempt at recovering those chunks, re-establishing connections, or using any intelligence at all. It also overwhelms http header size norms on the regular and runs up memory when attempting to use those bundles. Chunks/bundles are, in fact, strictly worse unless you get your initial clones with a good piece of software like curl.
Lt_Riza_Hawkeye 6y ago
Somehow FB has made all the common operations work in <6 seconds with millions of files...
- gravypod 6y ago
  It's an impressive feat but something I find even more impressive is the scale they need to be prepared for. The performance challenges of maintaining infrastructure for as many developers as Facebook/Google employ is constant and demanding. At my extremely small startup in the past ~6 months I've made about 1,000 files (@ HEAD, not including deletions). So, maybe, at the absolute worst case each engineer could produce ~150 files/month. Facebook likely has ~1,000 engineers. That's a worst case 1,800,000 files/year + an existing massive code base. Good luck to the engineers who are forever battling the scaling issues underlying these orgs! It's extremely impressive to even hear about overcoming these huge hurdles.
  - tehlike 6y ago
    Facebook has way more than 1000 engineers :)
- tehlike 6y ago
  It all breaks down when it does not because watchman decides to come to a halt and basic arc pull takes more than 10mins after a week of no action on the repo :)
  - millstone 6y ago
    Watchman is gone!
ian-g 6y ago
I'd be very interested to hear what Google's done to their perforce server, if anything. I used to do vendor support for a company that does version control for chip design. Basically Perforce + some metadata and applications that automate away a bunch of the administrative grunt work. Some of our larger customers had quite a few layers of abstraction to try and speed it up (to varying degrees of success). But I don't know how Google's scale would compare to a company that would routinely have check-ins they claimed to be hundreds of GB. Or another that checked in their giant PDKs to send them to each of their sites globally
- GauntletWizard 6y ago
  Google has completely replaced it, in stages, with a whitebox implementation that's by now almost a complete protocol rewrite. You can read about Piper here[1] [1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
  - ian-g 6y ago
    I've heard about that! Was more thinking about what they did while there was still a Perforce server on the backend
    - GauntletWizard 6y ago
      That I can speak (a little) to: They gave it a gigantic RAM-Backed SAN and the best hardware money could buy. Nothing particularly interesting, but there were several perforce servers with 512G of RAM attached to several SANS with terrabytes of RAM storage (In 2012ish when it was replaced). It was only ever explained to me in terms of SWE-Hours, but my understanding is that the hardware budget was enough to fund the several software engineering teams that replaced it, and the licensing was more than the hardware.
millstone 6y ago
When did you work at Facebook? Things used to be bad, but improved greatly after Eden was introduced around 2018. Eden is a FUSE filesystem which uses hg as the backing store. (I also worked at Google but not the google3 monorepo - instead I got emerge, which was misery.)
- tehlike 6y ago
  Joined about 6months ago, after 7.5 yrs at google. Eden still doesn't support all repos :( I do Android occasionally, from laptop, given wfh.
  - millstone 6y ago
    Can you share where Eden doesn't work? I no longer work at Fb but at the time I left Eden was solid for www/android, and mostly there on Mac/iOS.
bolinfest 6y ago
As a fellow Facebook employee, FYI, Hacker News is not a great place to get tech support. For the benefit of everyone else on this thread, note that Facebook uses https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden. From the README: > Despite having originally evolved from Mercurial, EdenSCM is not a distributed source control system. i.e., this is not "stock" Mercurial. For example, the Eden server (Mononoke) is written in Rust and the complementary virtual filesystem is written in C++. This has very different performance characteristics than real Mercurial. EdenSCM works with both EdenFS (the custom filesystem) and a traditional filesystem. If you use EdenFS, pulls will be much cheaper because you only fetch what you use. If you use a traditional filesystem, EdenSCM supports the same "sparse checkouts" feature as stock Mercurial (https://firefox-source-docs.mozilla.org/build/buildsystem/sparse.html https://firefox-source-docs.mozilla.org/build/buildsystem/sp...), which can also be used to reduce the size of the slice of the monorepo you pull down. Last I checked, Perforce (and Google's "implementation" of Perforce, Piper) did not provide nearly the same level of support for stacked diffs as Eden. As both Google and Facebook have cultures of pre-commit code review, working with stacked diffs makes it much easier to make progress while waiting for approvals on earlier diffs. I believe there are relative advantages/disadvantages of Eden vs. Piper+CitC and that both projects aspire to have the best of each in the limit.

liveoneggs 6y ago

I can only imagine an attempt at the initial clone.. the hg client is ridiculously slow, memory-hungry, and full of dumb behavior that seems to assume local connections

est31 6y ago
hg has a mode to make initial clones faster by offering zipped compressed checkouts for download.
- liveoneggs 6y ago
  it does range requests to these bundles, attempts to extract the partials while keeping the http connection alive using up GB of memory in the process, and then totally fails pretty regularly when it need to go back to the network. It makes no attempt at recovering those chunks, re-establishing connections, or using any intelligence at all. It also overwhelms http header size norms on the regular and runs up memory when attempting to use those bundles. Chunks/bundles are, in fact, strictly worse unless you get your initial clones with a good piece of software like curl.

Lt_Riza_Hawkeye 6y ago

Somehow FB has made all the common operations work in <6 seconds with millions of files...

gravypod 6y ago
It's an impressive feat but something I find even more impressive is the scale they need to be prepared for. The performance challenges of maintaining infrastructure for as many developers as Facebook/Google employ is constant and demanding. At my extremely small startup in the past ~6 months I've made about 1,000 files (@ HEAD, not including deletions). So, maybe, at the absolute worst case each engineer could produce ~150 files/month. Facebook likely has ~1,000 engineers. That's a worst case 1,800,000 files/year + an existing massive code base. Good luck to the engineers who are forever battling the scaling issues underlying these orgs! It's extremely impressive to even hear about overcoming these huge hurdles.
- tehlike 6y ago
  Facebook has way more than 1000 engineers :)
tehlike 6y ago
It all breaks down when it does not because watchman decides to come to a halt and basic arc pull takes more than 10mins after a week of no action on the repo :)
- millstone 6y ago
  Watchman is gone!

ian-g 6y ago

I'd be very interested to hear what Google's done to their perforce server, if anything. I used to do vendor support for a company that does version control for chip design. Basically Perforce + some metadata and applications that automate away a bunch of the administrative grunt work. Some of our larger customers had quite a few layers of abstraction to try and speed it up (to varying degrees of success). But I don't know how Google's scale would compare to a company that would routinely have check-ins they claimed to be hundreds of GB. Or another that checked in their giant PDKs to send them to each of their sites globally

GauntletWizard 6y ago
Google has completely replaced it, in stages, with a whitebox implementation that's by now almost a complete protocol rewrite. You can read about Piper here[1] [1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
- ian-g 6y ago
  I've heard about that! Was more thinking about what they did while there was still a Perforce server on the backend
  - GauntletWizard 6y ago
    That I can speak (a little) to: They gave it a gigantic RAM-Backed SAN and the best hardware money could buy. Nothing particularly interesting, but there were several perforce servers with 512G of RAM attached to several SANS with terrabytes of RAM storage (In 2012ish when it was replaced). It was only ever explained to me in terms of SWE-Hours, but my understanding is that the hardware budget was enough to fund the several software engineering teams that replaced it, and the licensing was more than the hardware.

millstone 6y ago

When did you work at Facebook? Things used to be bad, but improved greatly after Eden was introduced around 2018. Eden is a FUSE filesystem which uses hg as the backing store. (I also worked at Google but not the google3 monorepo - instead I got emerge, which was misery.)

tehlike 6y ago
Joined about 6months ago, after 7.5 yrs at google. Eden still doesn't support all repos :( I do Android occasionally, from laptop, given wfh.
- millstone 6y ago
  Can you share where Eden doesn't work? I no longer work at Fb but at the time I left Eden was solid for www/android, and mostly there on Mac/iOS.

bolinfest 6y ago

As a fellow Facebook employee, FYI, Hacker News is not a great place to get tech support. For the benefit of everyone else on this thread, note that Facebook uses https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden. From the README: > Despite having originally evolved from Mercurial, EdenSCM is not a distributed source control system. i.e., this is not "stock" Mercurial. For example, the Eden server (Mononoke) is written in Rust and the complementary virtual filesystem is written in C++. This has very different performance characteristics than real Mercurial. EdenSCM works with both EdenFS (the custom filesystem) and a traditional filesystem. If you use EdenFS, pulls will be much cheaper because you only fetch what you use. If you use a traditional filesystem, EdenSCM supports the same "sparse checkouts" feature as stock Mercurial (https://firefox-source-docs.mozilla.org/build/buildsystem/sparse.html https://firefox-source-docs.mozilla.org/build/buildsystem/sp...), which can also be used to reduce the size of the slice of the monorepo you pull down. Last I checked, Perforce (and Google's "implementation" of Perforce, Piper) did not provide nearly the same level of support for stacked diffs as Eden. As both Google and Facebook have cultures of pre-commit code review, working with stacked diffs makes it much easier to make progress while waiting for approvals on earlier diffs. I believe there are relative advantages/disadvantages of Eden vs. Piper+CitC and that both projects aspire to have the best of each in the limit.