18 ms·

Meta developer tools: Working at scale

245 points by ot 3y ago

arnon 3y ago
Surprised to see them use Phabricator (I know it came out of there, but I basically already forgot it existed) I used it briefly but couldn't get most people to adopt it widely enough.
- pdpi 3y ago
  Their internal Phabricator is quite different from the open source version. When I worked there (around 2017), it was definitely miles ahead of anything I'd used before. A lot of how it works is probably... controversial, I guess, but it suited my mental model quite well.
- goodoldneon 3y ago
  I used Phabricator at a previous (non-Meta) company. It was better than GitHub in some ways but worse in others. Overall I didn’t mind it
- seagullriffic 3y ago
  I wonder whether they'll continue using their in-house Phrabricator or choose to support Phorge (https://we.phorge.it/ https://we.phorge.it/) now that open-source Phabricator is no longer supported.
  - Kilenaitor 3y ago
    Internal Phabricator isn't even really Phabricator anymore. In fact, the name changed to just Diffs. I'm pretty sure its entire codebase has been rewritten, at the very least into Hack from PHP. They have a custom API; not Conduit. Etc. So, no. They wont support Phorge. They really never supported open-source Phabricator after Evan left and made it his own with Phacility.
- epage 3y ago
  I used Phab at a job and it was a complete mess. Stacked Diffs only sort-of worked, CI integration was bad, notifications were so noisy everyone tuned them out, etc. Talking with a Meta person, it sounds like Phab really needs Mercurial to work well, at least for Stacked Diffs because you need to be able to identify commits independent of their location in history to properly maintain the Stacked Diff associations.
  - deleted 3y ago
    [deleted]
- alexhornby 3y ago
  reviewstack is the thing now, adds some parts of phab UX ontop of github apis. Been enjoying using it for oss stuff oursite meta and fixed a limitation of it recently, https://reviewstack.dev/facebook/sapling/pull/656 https://reviewstack.dev/facebook/sapling/pull/656 shows what it looks like with the versioning available
blitz_skull 3y ago
Sapling looks quite cool! I've used git extensively in my career and consider myself as having a slightly-more-advanced-than-typical understanding of how to use it just based on conversations with colleagues. However, one thing that's always been very limiting with git has been stack-based PR reviews, and as they mentioned amending deep commits. It's not impossible, but it makes it awkward enough that I usually avoid it if possible. Curious if anyone has used Sapling after lots of time using git. Is it the future?
- ink_13 3y ago
  Probably not, if only for reasons of inertia. Git plus third-party review tools (like GitHub) is more than "good enough" for most purposes. I used to work at FB, and while sapling is quite nice to use in practice, without the internal version of Phabricator to do code review (and, in all likelihood, mononoke), I don't think I'd pick it up again.
  - c_crank 3y ago
    GitHub's gotten so much worse since Microsoft bought it. The drop in quality compared to when I used it in school is remarkable.
    - HumanOstrich 3y ago
      Can you provide some specific examples?
      - c_crank 3y ago
        The token system used instead of passwords now was horribly explained. I had to look at several online resources to figure out how to actually use it. I've noticed that the system seems to repeatedly throttle me where it never did before. Logging in often results in being given an error until I try it enough times. Sometimes attempting to pull recent changes from a repo will fail, and tell me that I pulled the recent changes even though I am stuck on an old commit. I tried several things to fix this and the only thing that seems to work is doing a hard reset on the latest commit id. I never noticed github doing this before.
mrweasel 3y ago
Someone is going to read this and start to retooling their five person developer organisation because: "Facebook uses it". It's funny that the Sapling command is "sl", that's going to conflict with installations of "stream locomotive".
- counters 3y ago
  lol I've suffered under that before... except it was an ex-Googler forcing bazel on us, all of a 10 person dev team working on a codebase that was probably less than 15,000 LOC across three or four packages that just _had_ to be packaged into a monorepo.
  - 000ooo000 3y ago
    I worked with a guy who loved to use "Microsoft does it" as justification. Likewise, my suggestions such as "maybe a consistent naming convention for these components would be sensible" were met with (literally) "hmm I haven't seen MS suggest this". That was my shortest developer gig.
  - throwaway1777 3y ago
    Once it was setup was it really that painful? I’ve worked with buck and for day to day usage it didn’t really affect much.
say_it_as_it_is 3y ago
How many engineers did Meta lay off in the last 12 months? Is there a developer tool for laying off developers?
- throwaway1777 3y ago
  Fewer than some other companies. More than some others. But yeah, there are internal tools for layoffs. How else would you turn off access to hundreds of laptops at once?
yawnxyz 3y ago
This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"
- skizm 3y ago
  Most (all?) of these are open source and can be used outside of Meta infra. https://github.com/facebook/sapling https://github.com/facebook/sapling
  - patrec 3y ago
    > Mononoke is the server-side component of Sapling SCM. > While it is used in production within Meta, it currently does not build in an > open source context and is not yet supported for external usage.
    - tekknolagi 3y ago
      You can still use sapling with existing Git repos
- voidfunc 3y ago
  No see, what happens is you leave then go to some other company and complain about how shitty their tools are then build a crappy half-baked version of whatever you had at $BigTechCorp and when shit hits the fan you boomerang back to $BigTechCorp for a sweet promo and raise.
  - stcroixx 3y ago
    I avoid hiring people coming from places like this for reasons similar. Extends past tools into libraries and frameworks, databases and other middleware, developer and business workflows. Only knowing proprietary stuff is a handicap. Assuming that proprietary stuff is superior because the big ad companies prefer it is even worse.
zug_zug 3y ago
I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only tool they had that nobody disliked was their log-searching thing (can't remember what it's called) but it was still lightyears behind something likes splunk. It was my conclusion that "Wow you have 15 people working on a dev-tool compared to a public company with 100 building the industry-standard version over 10 years with actual product managers and UI experts, no wonder ours looks like crap... wouldn't it be cheaper just to take .01% of your salary and buy standard dev tools" I guess "Clunky" is the word I'm looking for. "Blow it away and make a new one" was a phrase that happened with some regularity for dev-envs, repo-checkouts, etc. And iirc restarting your dev box took like >30min. --- Side note - the other strangest thing was some of these tools people agreed were terrible (restart takes 30min). So you'd expect thousands of engineers to be swarming any system with any UI bug, edge-case, or whatever. But it just didn't work that way.
- typon 3y ago
  It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?
  - kevinventullo 3y ago
    FWIW I am a Meta-to-Google transplant and I feel the opposite way. There’s a lot of internal tooling and functionality I miss dearly. Most of all, Workplace.
    - mattnewton 3y ago
      FWIW I did the opposite leap and felt the opposite way- I really disliked workplace’s stream of seemingly randomly sorted information compared to e-mail lists I could filter, control, and search better. I really dislike workplace as a store of institutional knowledge. The chat was definitely way better than hangouts chat though. I also felt like a lot of the developer tools looked nicer than their google counterparts at the surface but had major reliability problems under the hood where you would need to do a lot of turn it off and on again style operations.
kernal 3y ago
Sapling looks interesting. Git has a horrible user experience.
jonathankoren 3y ago
I stopped reading when I reached that they made their own CVS. This is a solved problem, and even if it isn’t, there’s entire communities dedicated to it. There’s literally no reason why Facebook needs to be in the business of reinventing the wheel from scratch beyond some dude trying to show “impact”. It’s a distraction from core business problems.
- rigelbm 3y ago
  > This is a solved problem. Hosting a gigantic monorepo for 25K concurrent users is so far from a solved problem.
  - jonathankoren 3y ago
    [flagged]
    - sangnoir 3y ago
      For a moment, let's assume they do abandon the monorepo. What solution would you recommend for managing code dependencies and coordinating releases between thousands of teams (at a modest 5 repos per team) - git tags?
      - jonathankoren 3y ago
        This coordinating releases across teams is not a unique a problem. In fact, every large software organization solves this problem. They don’t usually do it in a assbackwards way due to institutional blindness.
        sangnoir 3y ago
        You're deflecting. What solution would you recommend, since you disapprove of monorepo as a solution to this problem we both agree exists. If you're not simultaneously updating the code and all it's references (i.e. a monorepo), you will need a version dependency graph system (with integrated with your build system). I'm yet to encounter one such tool that isn't awful to use[1]: monorepos are an improvement when you grow beyond a couple dozen repos. Git submodules aren't a good solution either. If you familiar with a decent tool/workflow that is not "institutionally blind", I'd love to learn more about it. 1. Gradle, Android's "repo", home-grown git-submodule-based build systems.
kayson 3y ago
I know git is complex, and the UX is sometimes messy, but after really, really learning it (shoutout to the Github training folks), I've never had any problems that couldn't be solved. I understand the desire to simplify some things, and their log looks way better than gits, but I wish they'd contribute back instead of rolling their own entire VCS. The one thing that is super exciting to me is the stacked pull request support. Using Github for this kind of workflow is enormously painful. Conversations constantly get outdated, and its nearly impossible to track whether comments have been addressed. I know they're working on an improved UX/experience there, but it seems like it'll be a good long while, especially for enterprise server customers.
- dmoy 3y ago
  I think the biggest issue is that back around 2011/2012 ish, when Facebook devs went to git core devs and asked how they could get git to scale to the size of their predicted monorepo, the response was roughly "no, shard it". git falls over and dies really, really badly when the repo gets stupidly large. There's an article alluding to the discussion here: https://engineering.fb.com/2014/01/07/core-data/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-data/scaling-merc..., but I can't find the original thread on the git mailing list.
  - 4ggr0 3y ago
    Maybe this is the mailing list you were searching for?[0] [0] https://web.archive.org/web/20210119051414/http://git.661346.n2.nabble.com/Git-performance-results-on-a-large-repository-td7250867.html https://web.archive.org/web/20210119051414/http://git.661346...
    - dmoy 3y ago
      Ah thanks! Yes I think that was it.
- jeffbee 3y ago
  "Contribute back to our piece of crap that is 99% antagonistic to your use case" is not realistic. No amount of third party contributions to git will relieve git of its opinions about how development workflow should be done, and those opinions are not shared with every organization.
cletus 3y ago
I've worked for both Facebook and Google so can make informed comments on this with two exceptions: Buck2 came after I left and I'm honestly not sure what sapling is. Is it some Mercurial-like re-implementation a bit like how Google's Piper is a re-implementation of Perforce? The tl;dr is that Google's developer tooling and ifnrastructure is superior in almost every way. Examples: - When I started at FB we used Nuclide, an internal fork of the Atom editor. While I was there it was replaced by VS Code. It's better but honestly they should've built their tooling off of Jetbrains products. Jetbrains make IDEs. VS Code is a text editor like vim or emacs. There's a massive difference; - Buck should've been killed and replaced by Bazel. I can't speak to Buck2 but this seems like a pointless investment; - Thrift should be killed and replaced with gRPC/Protobuf. Same deal; - FB's code search is just grep. It's literally called BigGrep. Grep can get you pretty far but it's just not the same as something with semantic understanding. Google has codesearch, which does understand code, and it's miles ahead. This has all sorts of weird side effects too, like Hack code at FB can't use namespaces or type aliasing because then grep wouldn't be able to find it. When there were name conflicts you'd sometimes be forced to rename something to get something to compile; - Tupperware (FB's container system) is a pale shadow of Borg; - Pushing www code at FB is a very good experience overall. You commit something and it'll get pushed to production possibly within an hour or two or, at busier times, it might take to the next day. This requires no release process or manual build. It's basically automatic; Google's build and release process tends to be way more onerous; - The big achillees heel in FB's www code is that it is one giant binary. There's no dependency declaration at all. This means there's an automatic system to detect if your change affects other things and that process often fails. This leads to trunk getting broken. A lot. - Because of the above problem there is a system to determine what tests to run for a given commit. This is partially about what the affected components are but also longer-running tests aren't run-on-commit and often those tests would've found the problem. There is no way to say "if this file is modified, run this test". That's a huge problem; - FB has a consistent system for running experiements and having features behind flags (ie gatekeeper). This wasn't the case when I was at Google. It may well have changed; - Creating a UI for an internal tool or a new page is incredibly easy at FB. There are standard components with the correct styling for everything. If you want to write an internal tool, you can start at 9am and have it in production by noon if it's not terribly complicated; - The build system for C++ at FB is, well, trash. For Buck (and Bazel), the build system creates a DAG of the build artifacts to decide what to build. FB C++ might take 2 minutes just to load the DAG before it builds anything. This is essentially instant at Google because a lot of infrastructure has been built to solve this problem. This is a combination of SrcFS and ObjFS. Incremental builds at FB to run tests doesn't really work as a workflow; - All non-www builds at FB are local builds. Nothing at Google (on Google3 at least) is built locally, including mobile apps. This is way faster because of build artifact cachcing and you have beefier build machines. - There tends to be less choices as to what to use for FB code (eg storage systems). I consider this largely a good thing. You will typically find 5 different way of doing anything at Google and then need to consider why. You will often find different teams solving the same problem in slightly different ways or even the exact same way. - There are people at FB who work on system-wide refactors (eg Web security, storage). These people can often only commit their diffs that might touch thousnads of files on weekends. - A lot of generated code is committed at FB that isn't at Google. This exacerbates the previous problem. FB has a ton of partially and completely generated files that mean a change to the generating code has a massive effect. At Google, for example, the protobuf generated code is genearted at build time and isn't in the repo. There's probably more but that's what comes to mind.
ghnws 3y ago
The linked article has a section about IDE that this one does not touch. I was surprised they are locking them selves in to one tool with their workflow. I'm a bit worried about the dominance of VS Code. I can't stand the editor and it's popularity just grows and grows.
- lostmsu 3y ago
  What are you using and why?
  - ghnws 3y ago
    IntelliJ. To name a few features I've not been able to get in VS Code or have had a worse experience: - debugger - refactoring (extracting functions and variables, renaming across entire project etc.) - context aware selection (expand selection from cursor logically) - stack trace / error parsing - comparing anything (diff selection with clipboard for example) - DB schema support even in SQL formatted as strings (e.g. when using psycopg) directly from DB by just connecting to a DB - find in files (I've never seen a VS code user have an easy time finding stuff) Etc.
- charcircuit 3y ago
  They aren't locking themselves in. It just makes sense to support less IDEs than more IDEs. You can move faster if you don't have to duplicate your work between vscode, intellij, android studio, emacs, vim, etc. Nothing is stop developers from using ed, the standard editor, if they wanted to.
ffpip 3y ago
Archive link for people who block facebook - https://web.archive.org/web/20230628131034/https://engineering.fb.com/2023/06/27/developer-tools/meta-developer-tools-open-source/ https://web.archive.org/web/20230628131034/https://engineeri...
GingerBoats 3y ago
[dead]
qwertywert_ 3y ago
If Meta acquires a new company codebase, do they just move it into their monorepo immediately?
- escapecharacter 3y ago
  Context: I was part of a company acquired by FB in 2019, worked there until Dec 2022. It really depends, based on how independent the acquired company needs to be, and how useful it would be to have overlap with code components & engineering resources with the main monorepo. In our case, up until Dec 2022, parts our pre-acquisition monorepo were still separate, but gradually components and workflows, such as tests and reviews, were moved into the main repo. From my awareness, we didn't even start merging into the main monorepo until more than a year after we were acquired, though of course there were exploratory efforts before. In general, I'm pro-monorepo, it makes sense to be able to update multiple interconnected components in lockstep. For the startup, we were still in research mode, so it was less urgent to spend eng/sci/TPM to incorporate with anything on the FB monorepo side...until it was.
TechBro8615 3y ago
Off topic, but why haven't they migrated these kind of sites to the meta.com domain? I mean, these are Meta tools, right? Not FB tools? Unless... the rename was more of an exercise in liability obfuscation than it was an indication of any kind of reorganization...
baggiponte 3y ago
I am genuinely curious about wasabi, the python LSP they announced on Meta open source but is not available anywhere. Would love to try that out, there is not enough competition in the LSP space in Python and it would foster new development https://developers.facebook.com/blog/post/2022/07/18/enabling-faster-python-authoring-with-wasabi/ https://developers.facebook.com/blog/post/2022/07/18/enablin...
- ngai_aku 3y ago
  In the “Offline + Online Processing” section, are they talking about an external service that needs to run alongside the processing that occurs on your machine? Or am I misunderstanding that?
Maxff 3y ago
Efficiency! - Where are you going? - I'm going on vacation. - Have you finished your project? - Not yet. Just submitted the diff. [^_^]
zdgeier 3y ago
If anyone is interested in an open source VCS that's trying to solve similar problems to these internal tools check out https://jamhub.dev https://jamhub.dev. (I am the author)
rochak 3y ago
I just love talking about build systems, editors and developer tooling. Anyone has resources on where I can read about how different companies do it?
- phyrex 3y ago
  Google describes theirs in their software engineering at Google book. It’s available for free online
vkaku 3y ago
Might as well release the Tupperware/Twine part and complete the picture :)
roland35 3y ago
Best tool: macros. You can easily tie in gifs and memes into just about anywhere

245 points by ot 3y ago

Surprised to see them use Phabricator (I know it came out of there, but I basically already forgot it existed) I used it briefly but couldn't get most people to adopt it widely enough.

pdpi 3y ago
Their internal Phabricator is quite different from the open source version. When I worked there (around 2017), it was definitely miles ahead of anything I'd used before. A lot of how it works is probably... controversial, I guess, but it suited my mental model quite well.
goodoldneon 3y ago
I used Phabricator at a previous (non-Meta) company. It was better than GitHub in some ways but worse in others. Overall I didn’t mind it
seagullriffic 3y ago
I wonder whether they'll continue using their in-house Phrabricator or choose to support Phorge (https://we.phorge.it/ https://we.phorge.it/) now that open-source Phabricator is no longer supported.
- Kilenaitor 3y ago
  Internal Phabricator isn't even really Phabricator anymore. In fact, the name changed to just Diffs. I'm pretty sure its entire codebase has been rewritten, at the very least into Hack from PHP. They have a custom API; not Conduit. Etc. So, no. They wont support Phorge. They really never supported open-source Phabricator after Evan left and made it his own with Phacility.
epage 3y ago
I used Phab at a job and it was a complete mess. Stacked Diffs only sort-of worked, CI integration was bad, notifications were so noisy everyone tuned them out, etc. Talking with a Meta person, it sounds like Phab really needs Mercurial to work well, at least for Stacked Diffs because you need to be able to identify commits independent of their location in history to properly maintain the Stacked Diff associations.
- deleted 3y ago
  [deleted]
alexhornby 3y ago
reviewstack is the thing now, adds some parts of phab UX ontop of github apis. Been enjoying using it for oss stuff oursite meta and fixed a limitation of it recently, https://reviewstack.dev/facebook/sapling/pull/656 https://reviewstack.dev/facebook/sapling/pull/656 shows what it looks like with the versioning available

blitz_skull 3y ago

Sapling looks quite cool! I've used git extensively in my career and consider myself as having a slightly-more-advanced-than-typical understanding of how to use it just based on conversations with colleagues. However, one thing that's always been very limiting with git has been stack-based PR reviews, and as they mentioned amending deep commits. It's not impossible, but it makes it awkward enough that I usually avoid it if possible. Curious if anyone has used Sapling after lots of time using git. Is it the future?

ink_13 3y ago
Probably not, if only for reasons of inertia. Git plus third-party review tools (like GitHub) is more than "good enough" for most purposes. I used to work at FB, and while sapling is quite nice to use in practice, without the internal version of Phabricator to do code review (and, in all likelihood, mononoke), I don't think I'd pick it up again.
- c_crank 3y ago
  GitHub's gotten so much worse since Microsoft bought it. The drop in quality compared to when I used it in school is remarkable.
  - HumanOstrich 3y ago
    Can you provide some specific examples?
    - c_crank 3y ago
      The token system used instead of passwords now was horribly explained. I had to look at several online resources to figure out how to actually use it. I've noticed that the system seems to repeatedly throttle me where it never did before. Logging in often results in being given an error until I try it enough times. Sometimes attempting to pull recent changes from a repo will fail, and tell me that I pulled the recent changes even though I am stuck on an old commit. I tried several things to fix this and the only thing that seems to work is doing a hard reset on the latest commit id. I never noticed github doing this before.

mrweasel 3y ago

Someone is going to read this and start to retooling their five person developer organisation because: "Facebook uses it". It's funny that the Sapling command is "sl", that's going to conflict with installations of "stream locomotive".

counters 3y ago
lol I've suffered under that before... except it was an ex-Googler forcing bazel on us, all of a 10 person dev team working on a codebase that was probably less than 15,000 LOC across three or four packages that just _had_ to be packaged into a monorepo.
- 000ooo000 3y ago
  I worked with a guy who loved to use "Microsoft does it" as justification. Likewise, my suggestions such as "maybe a consistent naming convention for these components would be sensible" were met with (literally) "hmm I haven't seen MS suggest this". That was my shortest developer gig.
- throwaway1777 3y ago
  Once it was setup was it really that painful? I’ve worked with buck and for day to day usage it didn’t really affect much.

say_it_as_it_is 3y ago

How many engineers did Meta lay off in the last 12 months? Is there a developer tool for laying off developers?

throwaway1777 3y ago
Fewer than some other companies. More than some others. But yeah, there are internal tools for layoffs. How else would you turn off access to hundreds of laptops at once?

yawnxyz 3y ago

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

skizm 3y ago
Most (all?) of these are open source and can be used outside of Meta infra. https://github.com/facebook/sapling https://github.com/facebook/sapling
- patrec 3y ago
  > Mononoke is the server-side component of Sapling SCM. > While it is used in production within Meta, it currently does not build in an > open source context and is not yet supported for external usage.
  - tekknolagi 3y ago
    You can still use sapling with existing Git repos
voidfunc 3y ago
No see, what happens is you leave then go to some other company and complain about how shitty their tools are then build a crappy half-baked version of whatever you had at $BigTechCorp and when shit hits the fan you boomerang back to $BigTechCorp for a sweet promo and raise.
- stcroixx 3y ago
  I avoid hiring people coming from places like this for reasons similar. Extends past tools into libraries and frameworks, databases and other middleware, developer and business workflows. Only knowing proprietary stuff is a handicap. Assuming that proprietary stuff is superior because the big ad companies prefer it is even worse.

zug_zug 3y ago

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only tool they had that nobody disliked was their log-searching thing (can't remember what it's called) but it was still lightyears behind something likes splunk. It was my conclusion that "Wow you have 15 people working on a dev-tool compared to a public company with 100 building the industry-standard version over 10 years with actual product managers and UI experts, no wonder ours looks like crap... wouldn't it be cheaper just to take .01% of your salary and buy standard dev tools" I guess "Clunky" is the word I'm looking for. "Blow it away and make a new one" was a phrase that happened with some regularity for dev-envs, repo-checkouts, etc. And iirc restarting your dev box took like >30min. --- Side note - the other strangest thing was some of these tools people agreed were terrible (restart takes 30min). So you'd expect thousands of engineers to be swarming any system with any UI bug, edge-case, or whatever. But it just didn't work that way.

typon 3y ago
It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?
- kevinventullo 3y ago
  FWIW I am a Meta-to-Google transplant and I feel the opposite way. There’s a lot of internal tooling and functionality I miss dearly. Most of all, Workplace.
  - mattnewton 3y ago
    FWIW I did the opposite leap and felt the opposite way- I really disliked workplace’s stream of seemingly randomly sorted information compared to e-mail lists I could filter, control, and search better. I really dislike workplace as a store of institutional knowledge. The chat was definitely way better than hangouts chat though. I also felt like a lot of the developer tools looked nicer than their google counterparts at the surface but had major reliability problems under the hood where you would need to do a lot of turn it off and on again style operations.

kernal 3y ago

Sapling looks interesting. Git has a horrible user experience.

jonathankoren 3y ago

I stopped reading when I reached that they made their own CVS. This is a solved problem, and even if it isn’t, there’s entire communities dedicated to it. There’s literally no reason why Facebook needs to be in the business of reinventing the wheel from scratch beyond some dude trying to show “impact”. It’s a distraction from core business problems.

rigelbm 3y ago
> This is a solved problem. Hosting a gigantic monorepo for 25K concurrent users is so far from a solved problem.
- jonathankoren 3y ago
  [flagged]
  - sangnoir 3y ago
    For a moment, let's assume they do abandon the monorepo. What solution would you recommend for managing code dependencies and coordinating releases between thousands of teams (at a modest 5 repos per team) - git tags?
    - jonathankoren 3y ago
      This coordinating releases across teams is not a unique a problem. In fact, every large software organization solves this problem. They don’t usually do it in a assbackwards way due to institutional blindness.
      - sangnoir 3y ago
        You're deflecting. What solution would you recommend, since you disapprove of monorepo as a solution to this problem we both agree exists. If you're not simultaneously updating the code and all it's references (i.e. a monorepo), you will need a version dependency graph system (with integrated with your build system). I'm yet to encounter one such tool that isn't awful to use[1]: monorepos are an improvement when you grow beyond a couple dozen repos. Git submodules aren't a good solution either. If you familiar with a decent tool/workflow that is not "institutionally blind", I'd love to learn more about it. 1. Gradle, Android's "repo", home-grown git-submodule-based build systems.

kayson 3y ago

I know git is complex, and the UX is sometimes messy, but after really, really learning it (shoutout to the Github training folks), I've never had any problems that couldn't be solved. I understand the desire to simplify some things, and their log looks way better than gits, but I wish they'd contribute back instead of rolling their own entire VCS. The one thing that is super exciting to me is the stacked pull request support. Using Github for this kind of workflow is enormously painful. Conversations constantly get outdated, and its nearly impossible to track whether comments have been addressed. I know they're working on an improved UX/experience there, but it seems like it'll be a good long while, especially for enterprise server customers.

dmoy 3y ago
I think the biggest issue is that back around 2011/2012 ish, when Facebook devs went to git core devs and asked how they could get git to scale to the size of their predicted monorepo, the response was roughly "no, shard it". git falls over and dies really, really badly when the repo gets stupidly large. There's an article alluding to the discussion here: https://engineering.fb.com/2014/01/07/core-data/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-data/scaling-merc..., but I can't find the original thread on the git mailing list.
- 4ggr0 3y ago
  Maybe this is the mailing list you were searching for?[0] [0] https://web.archive.org/web/20210119051414/http://git.661346.n2.nabble.com/Git-performance-results-on-a-large-repository-td7250867.html https://web.archive.org/web/20210119051414/http://git.661346...
  - dmoy 3y ago
    Ah thanks! Yes I think that was it.
jeffbee 3y ago
"Contribute back to our piece of crap that is 99% antagonistic to your use case" is not realistic. No amount of third party contributions to git will relieve git of its opinions about how development workflow should be done, and those opinions are not shared with every organization.

cletus 3y ago

I've worked for both Facebook and Google so can make informed comments on this with two exceptions: Buck2 came after I left and I'm honestly not sure what sapling is. Is it some Mercurial-like re-implementation a bit like how Google's Piper is a re-implementation of Perforce? The tl;dr is that Google's developer tooling and ifnrastructure is superior in almost every way. Examples: - When I started at FB we used Nuclide, an internal fork of the Atom editor. While I was there it was replaced by VS Code. It's better but honestly they should've built their tooling off of Jetbrains products. Jetbrains make IDEs. VS Code is a text editor like vim or emacs. There's a massive difference; - Buck should've been killed and replaced by Bazel. I can't speak to Buck2 but this seems like a pointless investment; - Thrift should be killed and replaced with gRPC/Protobuf. Same deal; - FB's code search is just grep. It's literally called BigGrep. Grep can get you pretty far but it's just not the same as something with semantic understanding. Google has codesearch, which does understand code, and it's miles ahead. This has all sorts of weird side effects too, like Hack code at FB can't use namespaces or type aliasing because then grep wouldn't be able to find it. When there were name conflicts you'd sometimes be forced to rename something to get something to compile; - Tupperware (FB's container system) is a pale shadow of Borg; - Pushing www code at FB is a very good experience overall. You commit something and it'll get pushed to production possibly within an hour or two or, at busier times, it might take to the next day. This requires no release process or manual build. It's basically automatic; Google's build and release process tends to be way more onerous; - The big achillees heel in FB's www code is that it is one giant binary. There's no dependency declaration at all. This means there's an automatic system to detect if your change affects other things and that process often fails. This leads to trunk getting broken. A lot. - Because of the above problem there is a system to determine what tests to run for a given commit. This is partially about what the affected components are but also longer-running tests aren't run-on-commit and often those tests would've found the problem. There is no way to say "if this file is modified, run this test". That's a huge problem; - FB has a consistent system for running experiements and having features behind flags (ie gatekeeper). This wasn't the case when I was at Google. It may well have changed; - Creating a UI for an internal tool or a new page is incredibly easy at FB. There are standard components with the correct styling for everything. If you want to write an internal tool, you can start at 9am and have it in production by noon if it's not terribly complicated; - The build system for C++ at FB is, well, trash. For Buck (and Bazel), the build system creates a DAG of the build artifacts to decide what to build. FB C++ might take 2 minutes just to load the DAG before it builds anything. This is essentially instant at Google because a lot of infrastructure has been built to solve this problem. This is a combination of SrcFS and ObjFS. Incremental builds at FB to run tests doesn't really work as a workflow; - All non-www builds at FB are local builds. Nothing at Google (on Google3 at least) is built locally, including mobile apps. This is way faster because of build artifact cachcing and you have beefier build machines. - There tends to be less choices as to what to use for FB code (eg storage systems). I consider this largely a good thing. You will typically find 5 different way of doing anything at Google and then need to consider why. You will often find different teams solving the same problem in slightly different ways or even the exact same way. - There are people at FB who work on system-wide refactors (eg Web security, storage). These people can often only commit their diffs that might touch thousnads of files on weekends. - A lot of generated code is committed at FB that isn't at Google. This exacerbates the previous problem. FB has a ton of partially and completely generated files that mean a change to the generating code has a massive effect. At Google, for example, the protobuf generated code is genearted at build time and isn't in the repo. There's probably more but that's what comes to mind.

ghnws 3y ago

The linked article has a section about IDE that this one does not touch. I was surprised they are locking them selves in to one tool with their workflow. I'm a bit worried about the dominance of VS Code. I can't stand the editor and it's popularity just grows and grows.

lostmsu 3y ago
What are you using and why?
- ghnws 3y ago
  IntelliJ. To name a few features I've not been able to get in VS Code or have had a worse experience: - debugger - refactoring (extracting functions and variables, renaming across entire project etc.) - context aware selection (expand selection from cursor logically) - stack trace / error parsing - comparing anything (diff selection with clipboard for example) - DB schema support even in SQL formatted as strings (e.g. when using psycopg) directly from DB by just connecting to a DB - find in files (I've never seen a VS code user have an easy time finding stuff) Etc.
charcircuit 3y ago
They aren't locking themselves in. It just makes sense to support less IDEs than more IDEs. You can move faster if you don't have to duplicate your work between vscode, intellij, android studio, emacs, vim, etc. Nothing is stop developers from using ed, the standard editor, if they wanted to.

ffpip 3y ago

Archive link for people who block facebook - https://web.archive.org/web/20230628131034/https://engineering.fb.com/2023/06/27/developer-tools/meta-developer-tools-open-source/ https://web.archive.org/web/20230628131034/https://engineeri...

GingerBoats 3y ago

[dead]

qwertywert_ 3y ago

If Meta acquires a new company codebase, do they just move it into their monorepo immediately?

escapecharacter 3y ago
Context: I was part of a company acquired by FB in 2019, worked there until Dec 2022. It really depends, based on how independent the acquired company needs to be, and how useful it would be to have overlap with code components & engineering resources with the main monorepo. In our case, up until Dec 2022, parts our pre-acquisition monorepo were still separate, but gradually components and workflows, such as tests and reviews, were moved into the main repo. From my awareness, we didn't even start merging into the main monorepo until more than a year after we were acquired, though of course there were exploratory efforts before. In general, I'm pro-monorepo, it makes sense to be able to update multiple interconnected components in lockstep. For the startup, we were still in research mode, so it was less urgent to spend eng/sci/TPM to incorporate with anything on the FB monorepo side...until it was.

TechBro8615 3y ago

Off topic, but why haven't they migrated these kind of sites to the meta.com domain? I mean, these are Meta tools, right? Not FB tools? Unless... the rename was more of an exercise in liability obfuscation than it was an indication of any kind of reorganization...

baggiponte 3y ago

I am genuinely curious about wasabi, the python LSP they announced on Meta open source but is not available anywhere. Would love to try that out, there is not enough competition in the LSP space in Python and it would foster new development https://developers.facebook.com/blog/post/2022/07/18/enabling-faster-python-authoring-with-wasabi/ https://developers.facebook.com/blog/post/2022/07/18/enablin...

ngai_aku 3y ago
In the “Offline + Online Processing” section, are they talking about an external service that needs to run alongside the processing that occurs on your machine? Or am I misunderstanding that?

Maxff 3y ago

Efficiency! - Where are you going? - I'm going on vacation. - Have you finished your project? - Not yet. Just submitted the diff. [^_^]

zdgeier 3y ago

If anyone is interested in an open source VCS that's trying to solve similar problems to these internal tools check out https://jamhub.dev https://jamhub.dev. (I am the author)

rochak 3y ago

I just love talking about build systems, editors and developer tooling. Anyone has resources on where I can read about how different companies do it?

phyrex 3y ago
Google describes theirs in their software engineering at Google book. It’s available for free online

vkaku 3y ago

Might as well release the Tupperware/Twine part and complete the picture :)

roland35 3y ago

Best tool: macros. You can easily tie in gifs and memes into just about anywhere