10 ms·

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think m

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only tool they had that nobody disliked was their log-searching thing (can't remember what it's called) but it was still lightyears behind something likes splunk. It was my conclusion that "Wow you have 15 people working on a dev-tool compared to a public company with 100 building the industry-standard version over 10 years with actual product managers and UI experts, no wonder ours looks like crap... wouldn't it be cheaper just to take .01% of your salary and buy standard dev tools" I guess "Clunky" is the word I'm looking for. "Blow it away and make a new one" was a phrase that happened with some regularity for dev-envs, repo-checkouts, etc. And iirc restarting your dev box took like >30min. --- Side note - the other strangest thing was some of these tools people agreed were terrible (restart takes 30min). So you'd expect thousands of engineers to be swarming any system with any UI bug, edge-case, or whatever. But it just didn't work that way.

typon 3y ago
It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?
- kevinventullo 3y ago
  FWIW I am a Meta-to-Google transplant and I feel the opposite way. There’s a lot of internal tooling and functionality I miss dearly. Most of all, Workplace.
  - mattnewton 3y ago
    FWIW I did the opposite leap and felt the opposite way- I really disliked workplace’s stream of seemingly randomly sorted information compared to e-mail lists I could filter, control, and search better. I really dislike workplace as a store of institutional knowledge. The chat was definitely way better than hangouts chat though. I also felt like a lot of the developer tools looked nicer than their google counterparts at the surface but had major reliability problems under the hood where you would need to do a lot of turn it off and on again style operations.
  - deleted 3y ago
    [deleted]
  - theptip 3y ago
    I do really like Workplace, I think it provides a much better information architecture than docs/slack alone. It’s possible to run a really disciplined Slack where updates go in a ready-only channel, and people broadcast their work appropriately - but it’s really hard. Most people default to non-discoverable DMs and then stream-of-thought in public channels, which are both bad for discovery in their own way. Posts end up being a bit more considered than a slack message, but still lightweight, and more discoverable than docs. Shame Google nixed their “Google plus for workplace”, I would love to see this workflow used more widely.
- loeg 3y ago
  I think GP's opinion is fairly contrarian. I would say the in-house tools at Meta are substantially better than typical open source or commercial tools used at smaller companies.
taeric 3y ago
I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had to mirror any repository system locally and learn how to set that up. Probably getting it wrong in the process. Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be. We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.
- p_l 3y ago
  I remember when Cisco mentioned potential acquisition of Splunk for $20B, the jokes about whether it was to buy the company or to pay the running Splunk bill wrote themselves.
- zug_zug 3y ago
  I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal. So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors. Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.
  - taeric 3y ago
    Even in a fang, most money spent on log storage is fully wasted. The article yesterday about monitoring being a pain was spot on. More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill. Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates. Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons. At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time. Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.
Macha 3y ago
Yeah, I've worked at another large tech company who had accumulated a lot of in house tooling with a lot of invested SMEs who had been there a long time and missed that the state of the art out in the world had passed them by.
- ericbarrett 3y ago
  I left FB in the early-mid 2010s and can attest to this. It took a year or so to adjust to the "real world." Things like, no, you don't have a fancy scheduler; you get SSH, ansible, and systemd timers. Docker, Jenkins, and Kubernetes (RIP Mesos) were all new to me when other devs/devops folks had years of experience.
idkyall 3y ago
Yeah, that's fair. I think another thing is that a lot of these internal tooling projects date back 5-10 years when open source alternatives may not have been viable yet. For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet. Now it's reaching growing pains as we've scaled, and at this point it's a bit of a sunk cost fallacy to not migrate off of it. Another thing is that sometimes it's truly impossible to use one-size fits all for some tools if your company scale is large enough. I've heard from friends at Amazon that the dev experience and tools are totally different in the hardware space than it is for those working in the retail or AWS orgs.
- tm-guimaraes 3y ago
  > For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet. That's why big orgs are now opensourcing their tools. But it's not to clean up all "too internal" stuff from a tool.
quadrifoliate 3y ago
I don't know anything about tooling at Meta, but I have appreciated custom-built tooling at all the jobs I have worked at so far -- much smaller companies ranging from about 100 to 5000 people. Some of this is just having worked in the industry for a while. Guess what, 15 years ago "Put all your code in a web service running on AWS" was not nearly as slam-dunk a proposition as it is now; and tools optimized for on-prem were faster and more reliable. The other thing is that you don't always have to reinvent things to the point that Meta does. You can just wrap around an Open Source project, or even contribute to it. This is what all of my previous jobs did -- make custom tooling from industry-standard OSS building blocks; and contribute back when it made sense for the larger community. I actually think Meta was actually trying to do this with Mercurial a while ago [1], you probably have a much better idea than us as to why that didn't work out. ---------------------------------------- [1] https://engineering.fb.com/2014/01/07/core-data/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-data/scaling-merc...
- randall 3y ago
  https://sapling-scm.com/ https://sapling-scm.com/ It did work out.
  - quadrifoliate 3y ago
    I know that, I meant the specific reasons why Sapling diverged from standard Mercurial. https://sapling-scm.com/docs/introduction/differences-hg/ https://sapling-scm.com/docs/introduction/differences-hg/ lays out some of the differences; but not why they exist. One guess that I can make from Sapling docs is that perhaps FB/Meta needed an ability to pull in Git repos where needed; and I can see Mercurial devs not being super enthusiastic about that being a first-class workflow in Mercurial itself.
    - ak67 3y ago
      > I know that, I meant the specific reasons why Sapling diverged from standard Mercurial The needs of a large internal code base are often quite different than what open source projects need. Anther issue is that tools like Git and Mercurial need to be compatible with positively ancient repositories and can't really break backwards compatibility whereas within Meta, it is easier to move faster and deal with breaking format changes since developers's checked out repos on their laptops can be silently upgraded since the machines are well managed with Chef and can be upgraded behind the scenes. In the outside world, you can't assume people will be running the latest version and can't upgrade them. Finally, Google is also moving away from Mercurial. Perhaps the JJ developers want to respond why (and check out Jujitsu -- it's a novel and very interesting system y'all should check out).
bertil 3y ago
I would say it’s quite uneven: most tools were better than the state of the art when they were introduced. For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority. Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.
- barbazoo 3y ago
  My current role for the first time exposed me to amounts of data and variety of data (in type and how/where it's stored) that's difficult to learn organically. I wonder how a FAANG does that at "scale". > most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. Is that the kind of question you're talking about here?
rangledangle 3y ago
No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind their intern tool, and usually built to all just work together without friction.
- pavlov 3y ago
  Code review on stacked diffs was awesome for bigger features.
  - rangledangle 3y ago
    The open-ness of code, visibility, diffs. It was perfection. Something broke in my env suddenly? OH, I just checked recently pushed diffs that affect my realm. Hey there it is, security pushed something weird. I'll just revert the part that affects me and tag them. No meeting, maybe a SEV for visibility and review, maybe not. Easy peasy.
- castlecrasher2 3y ago
  >Slack is straight up painful compared to their chat system. As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?
  - anonymoushn 3y ago
    slack is incredibly, unbearably slow
  - KaiserPro 3y ago
    As a chat system workplace is inferior to slack. Discovery and organisation of chats are hard, especially group chats that would normally be a channel. However the combination of the "newsgroup" and chat makes it kinda better than slack on it's own. Would I use workplace at a new startup? probably not. I would use slack though.
fzeindl 3y ago
The problem with metas approach to tooling from a very high level viewpoint is that they virtualize everything. They abstract all common tasks like building, testing or running by building an entirely new system that builds an abstraction layer. Abstraction layers tend to be slow. What they should focus on is plumbing. Take what exists and connect it in a smart way. Take React: the idea of applying functional programming and spitting out HTML was a great idea. HTML existed before react. Then they also implemented a virtual-DOM, which was the unnecessary part. Or react-native: rendering UIs in a functional way is great, but stuffing JavaScript into everything is not necessary.
- RandallBrown 3y ago
  Maybe it's not necessary anymore, but wasn't the virtual DOM made for performance reasons since updating the DOM used to be really really slow?
  - Jcampuzano2 3y ago
    Virtual DOM in any scenario will always be slower than updating the DOM directly. The virtual DOM will always be overhead. The issue is if you are manually updating the DOM yourself you can easily shoot yourself in the foot by doing so in a way that is less performant, or breaks things like scroll position, focus management etc. Its also normally more tedious to do manually. This is where the virtual DOM came in. You just declaratively update the JSX and it handles the updating itself. But nowadays there are plenty of solutions that do not use virtual DOM (or even use a virtual DOM solution that is just plain faster) but still allow you to use declarative rendering, basically the best of both worlds. They are just not as popular or for some people not as interesting to them as React is. Or they are content with Reacts performance already.
findjashua 3y ago
phabricator & scuba are far better than anything i've used anywhere else.
- thiagocmoraes 3y ago
  I REALLY miss Phabricator, Scuba and (gasp) Tasks. Would love to have those back in industry. Far better than Github, Superset (?) and of course the dreadful Jira I have to use these days.
- rsdbdr203 3y ago
  I was very impressed by Scuba's ability to ingest and search data... and very unimpressed by the interface. I found it terribly unintuitive, and often didn't get you what you wanted without having to run multiple queries. Phabricator was pretty sweet... and the internal version of mercurial was a dream!
optymizer 3y ago
For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?). The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's just a better, more intuitive workflow. Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. The comment about small files is odd to me. Source code are small files. The entire monorepo is just small files. Now if we are going to complain about something, it would be the custom Android environment we've got. Android Studio integration is actually clunky, bordering on non-existent. Compared to writing vanilla Android code, which I have done for years before joining Meta, Meta folks wrapped almost every API, so all your pre-existing Android knowledge is useless. I've never been so unproductive when writing Android code.
- PreachSoup 3y ago
  I agree. The tooling is not as good as Google's(expected, Google's internal ides are incredible). But it's definitely better than the rest
- MikeTheRocker 3y ago
  Having also written Android code at Meta, I 100% agree.
  - Solvency 3y ago
    What's the short version of why Meta did this, wrapping so much up in this internal nonstandard format?
    - nsonha 3y ago
      I'm guessing same to why everyone else does that: they want to abstract away the platform. To some people platform intricacies are uninteresting comparing to the problem they are solving, so their abstractions aim to express their problem domain, and hide everything else. For example for apps you may like to deal with abstractions that can express navigation and pages. They are the same across web, iOS, Android. Unlike Activity, Fragment, View (is that still a thing, I stopped doing android 10 years ago lol) and some slightly different set of abstrations over iOS.
jacksnipe 3y ago
Strongly not my experience. The tooling at meta was so good compared to my previous 5 jobs that I left with changed opinions about the tradeoffs of investing in tooling.
LoulouMonkey 3y ago
As a Meta employee as well, and working in the data analytics / engineering space, I'm finding tooling to be pretty high standards actually. Though we arguably rely on a lot of Apache products, whatever we use that's internal only is great to work with. Daiq** recently started supporting notebooks, which has been a game changer for us as well as for the teams we work with. Phabricator is great as well, and makes shipping stuff super easy. Only Ben**, the internal notebook solution, I find meh. Especially compared to Google Colab. But the rest has simply been a joy to work with. For those interested, a former DE made this nice repo that maps internal tools against "real world" products: https://github.com/thijsessens/xmeta2external https://github.com/thijsessens/xmeta2external
- shostack 3y ago
  Is there a reason you censored parts of tool names? Your link lists the uncensored version.
hyuuu 3y ago
as a former employee who has worked both in many other companies over a decade + starting a few startups, Meta tooling is one of the best if not the best, it's so good to a point where Saas companies came out of facebook simply by replicating their internal tooling, like Asana, Scuba, etc
- shostack 3y ago
  I thought Asana were Xooglers?
  - roland35 3y ago
    Nope started by Dustin Moskovitz who was even in the "social network" movie
jarjoura 3y ago
As an ex-meta mobile engineer, I 100% agree that I found Eden to be impossible to use productively. I always had to pre-warm the cache, pretty much defeating its stated goals. Maybe it has gotten better since I left a year ago, but I had 2 TB of local storage, and mercurial's sparse checkouts were more than enough for me and far more reliable. However, all the other tooling around code were just superior in every way. I miss Phabricator, and mercurial and the way it seamlessly integrates into a team. I also miss all of the command line helpers that let me manage all of it. It used to be the case that legitimate teams could form around and focus just on building internal tools for other teams if it improved engineering velocity. Not sure if that's still the case in this new era of layoff-happy Meta, but it was definitely true when I was there.
- roland35 3y ago
  Eden definitely works better on a dev server and sucks on a laptop, especially arm based mac's (it's slowly getting better but still requires occasional reboots). +1 on the coding tools being great
  - ak67 3y ago
    Please reach out to the Eden team about your M1 issues; they'd love to hear from you - from what I've heard, it works better on M1s.
fuzztester 3y ago
>I think Meta's tooling is inferior to industry standard. s/Meta's tooling/Meta/g s/inferior/very inferior/g
radicality 3y ago
Like a few others already commented, I also disagree. Was at FB for 7 years, and now almost 2 years at a place with the “industry standard” things like slack / datadog / github / sourcegraph / (20+ other tools all disconnected from the rest and all behind SSO). As a backend engineer, the dev experience was just incomparably better at FB. Some things I most most are probably phabricator / stacked diffs workflows, buck dep management with everything in a monorepo, deep integration across all the the tooling etc.
ketchupdebugger 3y ago
Every tool Meta has, Google also has. Google open source theirs so the industry gravitates towards that, but Meta keeps theirs internal. So is Meta's tooling better than Industry/Google open source tools? probably not. The only tool I was impressed by was scuba. But Meta basically build it out of gold (thousands of machines with TB's of ram)

typon 3y ago

It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?

kevinventullo 3y ago
FWIW I am a Meta-to-Google transplant and I feel the opposite way. There’s a lot of internal tooling and functionality I miss dearly. Most of all, Workplace.
- mattnewton 3y ago
  FWIW I did the opposite leap and felt the opposite way- I really disliked workplace’s stream of seemingly randomly sorted information compared to e-mail lists I could filter, control, and search better. I really dislike workplace as a store of institutional knowledge. The chat was definitely way better than hangouts chat though. I also felt like a lot of the developer tools looked nicer than their google counterparts at the surface but had major reliability problems under the hood where you would need to do a lot of turn it off and on again style operations.
- deleted 3y ago
  [deleted]
- theptip 3y ago
  I do really like Workplace, I think it provides a much better information architecture than docs/slack alone. It’s possible to run a really disciplined Slack where updates go in a ready-only channel, and people broadcast their work appropriately - but it’s really hard. Most people default to non-discoverable DMs and then stream-of-thought in public channels, which are both bad for discovery in their own way. Posts end up being a bit more considered than a slack message, but still lightweight, and more discoverable than docs. Shame Google nixed their “Google plus for workplace”, I would love to see this workflow used more widely.
loeg 3y ago
I think GP's opinion is fairly contrarian. I would say the in-house tools at Meta are substantially better than typical open source or commercial tools used at smaller companies.

taeric 3y ago

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had to mirror any repository system locally and learn how to set that up. Probably getting it wrong in the process. Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be. We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.

p_l 3y ago
I remember when Cisco mentioned potential acquisition of Splunk for $20B, the jokes about whether it was to buy the company or to pay the running Splunk bill wrote themselves.
zug_zug 3y ago
I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal. So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors. Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.
- taeric 3y ago
  Even in a fang, most money spent on log storage is fully wasted. The article yesterday about monitoring being a pain was spot on. More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill. Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates. Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons. At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time. Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.

Macha 3y ago

Yeah, I've worked at another large tech company who had accumulated a lot of in house tooling with a lot of invested SMEs who had been there a long time and missed that the state of the art out in the world had passed them by.

ericbarrett 3y ago
I left FB in the early-mid 2010s and can attest to this. It took a year or so to adjust to the "real world." Things like, no, you don't have a fancy scheduler; you get SSH, ansible, and systemd timers. Docker, Jenkins, and Kubernetes (RIP Mesos) were all new to me when other devs/devops folks had years of experience.

idkyall 3y ago

Yeah, that's fair. I think another thing is that a lot of these internal tooling projects date back 5-10 years when open source alternatives may not have been viable yet. For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet. Now it's reaching growing pains as we've scaled, and at this point it's a bit of a sunk cost fallacy to not migrate off of it. Another thing is that sometimes it's truly impossible to use one-size fits all for some tools if your company scale is large enough. I've heard from friends at Amazon that the dev experience and tools are totally different in the hardware space than it is for those working in the retail or AWS orgs.

tm-guimaraes 3y ago
> For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet. That's why big orgs are now opensourcing their tools. But it's not to clean up all "too internal" stuff from a tool.

quadrifoliate 3y ago

I don't know anything about tooling at Meta, but I have appreciated custom-built tooling at all the jobs I have worked at so far -- much smaller companies ranging from about 100 to 5000 people. Some of this is just having worked in the industry for a while. Guess what, 15 years ago "Put all your code in a web service running on AWS" was not nearly as slam-dunk a proposition as it is now; and tools optimized for on-prem were faster and more reliable. The other thing is that you don't always have to reinvent things to the point that Meta does. You can just wrap around an Open Source project, or even contribute to it. This is what all of my previous jobs did -- make custom tooling from industry-standard OSS building blocks; and contribute back when it made sense for the larger community. I actually think Meta was actually trying to do this with Mercurial a while ago [1], you probably have a much better idea than us as to why that didn't work out. ---------------------------------------- [1] https://engineering.fb.com/2014/01/07/core-data/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

randall 3y ago
https://sapling-scm.com/ https://sapling-scm.com/ It did work out.
- quadrifoliate 3y ago
  I know that, I meant the specific reasons why Sapling diverged from standard Mercurial. https://sapling-scm.com/docs/introduction/differences-hg/ https://sapling-scm.com/docs/introduction/differences-hg/ lays out some of the differences; but not why they exist. One guess that I can make from Sapling docs is that perhaps FB/Meta needed an ability to pull in Git repos where needed; and I can see Mercurial devs not being super enthusiastic about that being a first-class workflow in Mercurial itself.
  - ak67 3y ago
    > I know that, I meant the specific reasons why Sapling diverged from standard Mercurial The needs of a large internal code base are often quite different than what open source projects need. Anther issue is that tools like Git and Mercurial need to be compatible with positively ancient repositories and can't really break backwards compatibility whereas within Meta, it is easier to move faster and deal with breaking format changes since developers's checked out repos on their laptops can be silently upgraded since the machines are well managed with Chef and can be upgraded behind the scenes. In the outside world, you can't assume people will be running the latest version and can't upgrade them. Finally, Google is also moving away from Mercurial. Perhaps the JJ developers want to respond why (and check out Jujitsu -- it's a novel and very interesting system y'all should check out).

bertil 3y ago

I would say it’s quite uneven: most tools were better than the state of the art when they were introduced. For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority. Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.

barbazoo 3y ago
My current role for the first time exposed me to amounts of data and variety of data (in type and how/where it's stored) that's difficult to learn organically. I wonder how a FAANG does that at "scale". > most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. Is that the kind of question you're talking about here?

rangledangle 3y ago

No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind their intern tool, and usually built to all just work together without friction.

pavlov 3y ago
Code review on stacked diffs was awesome for bigger features.
- rangledangle 3y ago
  The open-ness of code, visibility, diffs. It was perfection. Something broke in my env suddenly? OH, I just checked recently pushed diffs that affect my realm. Hey there it is, security pushed something weird. I'll just revert the part that affects me and tag them. No meeting, maybe a SEV for visibility and review, maybe not. Easy peasy.
castlecrasher2 3y ago
>Slack is straight up painful compared to their chat system. As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?
- anonymoushn 3y ago
  slack is incredibly, unbearably slow
- KaiserPro 3y ago
  As a chat system workplace is inferior to slack. Discovery and organisation of chats are hard, especially group chats that would normally be a channel. However the combination of the "newsgroup" and chat makes it kinda better than slack on it's own. Would I use workplace at a new startup? probably not. I would use slack though.

fzeindl 3y ago

The problem with metas approach to tooling from a very high level viewpoint is that they virtualize everything. They abstract all common tasks like building, testing or running by building an entirely new system that builds an abstraction layer. Abstraction layers tend to be slow. What they should focus on is plumbing. Take what exists and connect it in a smart way. Take React: the idea of applying functional programming and spitting out HTML was a great idea. HTML existed before react. Then they also implemented a virtual-DOM, which was the unnecessary part. Or react-native: rendering UIs in a functional way is great, but stuffing JavaScript into everything is not necessary.

RandallBrown 3y ago
Maybe it's not necessary anymore, but wasn't the virtual DOM made for performance reasons since updating the DOM used to be really really slow?
- Jcampuzano2 3y ago
  Virtual DOM in any scenario will always be slower than updating the DOM directly. The virtual DOM will always be overhead. The issue is if you are manually updating the DOM yourself you can easily shoot yourself in the foot by doing so in a way that is less performant, or breaks things like scroll position, focus management etc. Its also normally more tedious to do manually. This is where the virtual DOM came in. You just declaratively update the JSX and it handles the updating itself. But nowadays there are plenty of solutions that do not use virtual DOM (or even use a virtual DOM solution that is just plain faster) but still allow you to use declarative rendering, basically the best of both worlds. They are just not as popular or for some people not as interesting to them as React is. Or they are content with Reacts performance already.

findjashua 3y ago

phabricator & scuba are far better than anything i've used anywhere else.

thiagocmoraes 3y ago
I REALLY miss Phabricator, Scuba and (gasp) Tasks. Would love to have those back in industry. Far better than Github, Superset (?) and of course the dreadful Jira I have to use these days.
rsdbdr203 3y ago
I was very impressed by Scuba's ability to ingest and search data... and very unimpressed by the interface. I found it terribly unintuitive, and often didn't get you what you wanted without having to run multiple queries. Phabricator was pretty sweet... and the internal version of mercurial was a dream!

optymizer 3y ago

For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?). The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's just a better, more intuitive workflow. Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. The comment about small files is odd to me. Source code are small files. The entire monorepo is just small files. Now if we are going to complain about something, it would be the custom Android environment we've got. Android Studio integration is actually clunky, bordering on non-existent. Compared to writing vanilla Android code, which I have done for years before joining Meta, Meta folks wrapped almost every API, so all your pre-existing Android knowledge is useless. I've never been so unproductive when writing Android code.

PreachSoup 3y ago
I agree. The tooling is not as good as Google's(expected, Google's internal ides are incredible). But it's definitely better than the rest
MikeTheRocker 3y ago
Having also written Android code at Meta, I 100% agree.
- Solvency 3y ago
  What's the short version of why Meta did this, wrapping so much up in this internal nonstandard format?
  - nsonha 3y ago
    I'm guessing same to why everyone else does that: they want to abstract away the platform. To some people platform intricacies are uninteresting comparing to the problem they are solving, so their abstractions aim to express their problem domain, and hide everything else. For example for apps you may like to deal with abstractions that can express navigation and pages. They are the same across web, iOS, Android. Unlike Activity, Fragment, View (is that still a thing, I stopped doing android 10 years ago lol) and some slightly different set of abstrations over iOS.

jacksnipe 3y ago

Strongly not my experience. The tooling at meta was so good compared to my previous 5 jobs that I left with changed opinions about the tradeoffs of investing in tooling.

LoulouMonkey 3y ago

As a Meta employee as well, and working in the data analytics / engineering space, I'm finding tooling to be pretty high standards actually. Though we arguably rely on a lot of Apache products, whatever we use that's internal only is great to work with. Daiq** recently started supporting notebooks, which has been a game changer for us as well as for the teams we work with. Phabricator is great as well, and makes shipping stuff super easy. Only Ben**, the internal notebook solution, I find meh. Especially compared to Google Colab. But the rest has simply been a joy to work with. For those interested, a former DE made this nice repo that maps internal tools against "real world" products: https://github.com/thijsessens/xmeta2external https://github.com/thijsessens/xmeta2external

shostack 3y ago
Is there a reason you censored parts of tool names? Your link lists the uncensored version.

hyuuu 3y ago

as a former employee who has worked both in many other companies over a decade + starting a few startups, Meta tooling is one of the best if not the best, it's so good to a point where Saas companies came out of facebook simply by replicating their internal tooling, like Asana, Scuba, etc

shostack 3y ago
I thought Asana were Xooglers?
- roland35 3y ago
  Nope started by Dustin Moskovitz who was even in the "social network" movie

jarjoura 3y ago

As an ex-meta mobile engineer, I 100% agree that I found Eden to be impossible to use productively. I always had to pre-warm the cache, pretty much defeating its stated goals. Maybe it has gotten better since I left a year ago, but I had 2 TB of local storage, and mercurial's sparse checkouts were more than enough for me and far more reliable. However, all the other tooling around code were just superior in every way. I miss Phabricator, and mercurial and the way it seamlessly integrates into a team. I also miss all of the command line helpers that let me manage all of it. It used to be the case that legitimate teams could form around and focus just on building internal tools for other teams if it improved engineering velocity. Not sure if that's still the case in this new era of layoff-happy Meta, but it was definitely true when I was there.

roland35 3y ago
Eden definitely works better on a dev server and sucks on a laptop, especially arm based mac's (it's slowly getting better but still requires occasional reboots). +1 on the coding tools being great
- ak67 3y ago
  Please reach out to the Eden team about your M1 issues; they'd love to hear from you - from what I've heard, it works better on M1s.

fuzztester 3y ago

>I think Meta's tooling is inferior to industry standard. s/Meta's tooling/Meta/g s/inferior/very inferior/g

radicality 3y ago

Like a few others already commented, I also disagree. Was at FB for 7 years, and now almost 2 years at a place with the “industry standard” things like slack / datadog / github / sourcegraph / (20+ other tools all disconnected from the rest and all behind SSO). As a backend engineer, the dev experience was just incomparably better at FB. Some things I most most are probably phabricator / stacked diffs workflows, buck dep management with everything in a monorepo, deep integration across all the the tooling etc.

ketchupdebugger 3y ago

Every tool Meta has, Google also has. Google open source theirs so the industry gravitates towards that, but Meta keeps theirs internal. So is Meta's tooling better than Industry/Google open source tools? probably not. The only tool I was impressed by was scuba. But Meta basically build it out of gold (thousands of machines with TB's of ram)