5 ms·

Continuing the trend of Facebook creating competing tools more often than persisting with and improving existing ones, which I feel dilutes effort and has fragm

Continuing the trend of Facebook creating competing tools more often than persisting with and improving existing ones, which I feel dilutes effort and has fragmented a number of ecosystems. We've got a couple of Facebook fans at work, which has left us with a number of our systems using different tools to accomplish essentially the same tasks, and no strong case on either side for us to standardise on one of them. Maybe I'm just being bitter because I've had a bad experience, but it's been a maintenance nightmare for us in the office.

skrebbel 10y ago
That's not Facebook's fault, that's fanboyism's fault. I fail to see how the world gets worse from large companies releasing internal tools as open source.
- quickben 10y ago
  It is very simple. Instead of improving <insert make edition>, they started yet another brand new one. From my years and years at large companies, these things start because they don't want to share anything in the first place, and then it's just pricey to maintain.
BinaryIdiot 10y ago
To be fair a majority of the big technology companies all write their own tooling for many things that already exist because they didn't quite fit the way they needed them to and they had the resources to re-invent whatever they want. Sure when they open source them it further fragments the market and you always get the rush of "Facebook has almost 2 billion users therefore their tools must be the best!" which further exacerbates the issue but I don't blame Facebook for these issues. It sounds like, in your situation, perhaps there are too many tools being introduced into your projects. Too often I see developers abuse the hell out of npm and the like to just include whatever they need with zero regard for the newly introduced dependency tree and the new tooling that needs to now be maintained.
- cwyers 10y ago
  When Facebook wrote this, the tooling that already existed wasn't available to them -- ex-Google engineers at Facebook wanted something like Bazel/Blaze, but Google had yet to release that publicly. So this is a Hadoop situation where Google had this internal tool that someone else really wanted and because Google didn't open-source it they wrote their own version.
  - asymmetric 10y ago
    Isn't there a risk that reimplementing a previous employer's tool constitutes a legal/contractual violation of some kind?
    - Asooka 10y ago
      It's almost certain that it is a legal violation and if a lawsuit was brought, I'd bet a million dollars in favour of Google; however, the tool itself isn't the final product and it's silly, counterproductive and just outright malicious to sabotage efforts at building good tooling in a competitor's company. Plus, it will set precedent and Google would have to vet every single new internal tool against what the previous employers of their employees were doing.
- whywhywhywhy 10y ago
  > To be fair a majority of the big technology companies all write their own tooling for many things that already exist because they didn't quite fit the way they needed them Too many engineers syndrome comes to mind too.
perlgeek 10y ago
It seems you lack technical leadership in your organization. Don't blame facebook for it :-)
mhd 10y ago
I'm veering a bit towards the opposite: Some of the Facebook tools look quite nice, but there's a certain amount of taint that goes with their corporate heritage, so if there's a good alternative...
paulddraper 10y ago
What tool would you have suggested Facebook improve? Saying that Facebook should have improved Make or Maven is like saying Linus should have improved CVS or SVN instead of competing with Git. They're in the same space, but they differ at a very fundamental level.
- jhasse 10y ago
  Waf or Meson
  - paulddraper 10y ago
    Facebook doesn't have timetravel, so contributing to Meson would have been impossible.
    - jhasse 10y ago
      Ah I didn't know that Buck was created before 2013. What about Waf though?
      - paulddraper 10y ago
        TBH, I never heard of Waf before. It looks interesting. Buck, Waf, Meson, Bazel....apparently Python is the language for next-gen hash-based build tools :)
        efficax 10y ago
        Buck is primarily Java, just sayin'
        paulddraper 10y ago
        But the syntax of your build files is Pythonish. Just like Make is in C and the build files are in Make.
- hultner 10y ago
thomaslee 10y ago
I can't speak to the rest of Facebook's stuff, but I think the build tool problem is a special case. Per their docs: > Buck is designed for building multiple deliverables from a single repository (a monorepo) rather than across multiple repositories. It has been Facebook's experience that maintaining dependencies in the same repository makes it easier to ensure that all developers have the correct version of all of the code, and simplifies the process of making atomic commits. Having been on the "other side" of the monorepo argument where we tried to make do with improving/extending existing build tools etc. in a rapidly growing engineering org, let me say that Facebook (with Buck), Twitter (Pants? I think?) and Google (with Bazel/Blaze) almost certainly built these to deal with the problem of scaling build management with an ever-growing organization. The popular model of a dozen or so small repositories in GitHub + Jenkins with Maven/NPM/Rake+Bundler/whatever works fine for maybe a few dozen engineers or more, but one day you wake up and realize there hundreds of repositories spread across dozens of _teams_ and hundreds of developers. Obviously you've then got a big ol' dependency graph between repos to deal with, so if you need to fix something near the root suddenly you need to run off bumping version numbers and/or fixing intermediate libraries all the way down the graph. Plus version incompatibilities between the dependencies of different libraries ... it's a total mess, and it doesn't make for an org that can easily "move fast and break things", so to speak. So then to avoid paralysis your options are basically either to silo up (this team owns their stuff, that team owns their stuff, don't bother with shared dependencies) or you go the monorepo route. If you do, then maybe you go and pull all your hundreds of smaller repos into a monorepo. Having everything in one place makes it easier to police the dependency issues within the org & makes it easier for a single engineer to deal with those sort of "cascading changes" instead of shunting that problem onto the entire organization. But in exchange for this "agility" you've then got the problem that builds take multiple hours & the associated tools are often highly language-centric (Maven+Java, NPM+Node, Ruby+Rake, etc.). They don't typically make any reproducibility guarantees either. Anyway, to make a short story really long: at the time FB, Google and Twitter were hitting these organizational scaling walls, making these decisions and building these tools internally, there really weren't any great tools out there for the monorepo use case. I think that's why all these tools have appeared as side-by-side alternatives rather than improvements on one another or to tools like Maven et al. Whether or not consolidation is warranted, for the folks who have the problems that Buck/Bazel/Pants solve, it's likely to save 'em a hell of a lot of time, effort and money IMO. It's a good thing that they have been published, even if the value's maybe not immediately obvious.
LegNeato 10y ago
When Buck was created (my team created it!) there wasn't anything that supported what Facebook needed in a build tool. There still really isn't as all the Blaze work-a-likes are focusing on various different needs. I would argue Buck actually helps to solve what you are concerned about. At Facebook, everything builds with Buck (and Google with Bazel I hear)...which means you learn it once and you know it for your Objective-C library, your Android app, your Cxx service, your python scripts, etc. It really helps to standardize on a build system, and at many companies you can only do that if it supports windows/mac/linux, it supports many languages and platforms, is fast, and is easy to pick up. Buck is all of those things.
- NeverTrump 10y ago
  Any chance you can compare Pants with Buck?
- moondev 10y ago
  Gradle?
  - sdwilsh 10y ago
    Gradle was not the default Android build tool when Buck was open sourced.
- pjmlp 10y ago
  > At Facebook, everything builds with Buck (and Google with Bazel I hear). Surely not on Android or Skia teams, unless they internally use something other than what AOSP and Skia use.
- fs111 10y ago
  > At Facebook, everything builds with Buck Looking at https://github.com/facebook https://github.com/facebook you seem to be using all sorts of build tools...
  - sdwilsh 10y ago
    What Facebook uses internally in their monorepo and what they publish to open source for external consumption doesn't have to be the same build tool. Facebook uses fbshipit (https://github.com/facebook/fbshipit/ https://github.com/facebook/fbshipit/) to take slices of their monorepo and publish them to GitHub.
scrame 10y ago
Nope. I totally agree. I don't trust a single thing from facebook. I mean, good on them that they open source their tools, but most everything that comes out is just a huge pain in the ass with very little benefit to adopt, other than we can say we're using open-source facebook technology. The funny thing, though, is that keeping up with other web tech, you always have to deal with facebooks API's and sharing and whatnot, and its also, usually a huge pain in the ass. So, like, what's the benefit of this, switching build systems, other than a tick on the resume for someone who wants to get a job at facebook?
jlg23 10y ago
I'm all for using existing tools, but not for the sake of itself. If there is a good reason (and everyone in the team agrees), all now obsolete tools should simply be replaced by the new one. If this did not work in your office, this is most likely a problem of people in your office not communication properly about their choice of tools. Did you have people make a short presentation when they introduced a new tool? Did nobody ever point out that you already have various tools that can (easily/elegantly) accomplish what the next new thing(tm) is introduced for?
jacques_chester 10y ago
Facebook and Google have a problem I am familiar with: Not Invented Yet Syndrome. Typically they face a problem that few people have ever faced. There may be an existing solution, but not in the open. So they have to roll their own. Others in this thread say Buck was inspired by Blaze. That seems reasonable and I hardly think Facebook can be blamed for rolling their own when that was the only available option. You might as well blame Google for fragmenting the Hadoop ecosystem by creating their MapReduce framework.

skrebbel 10y ago

That's not Facebook's fault, that's fanboyism's fault. I fail to see how the world gets worse from large companies releasing internal tools as open source.

quickben 10y ago
It is very simple. Instead of improving <insert make edition>, they started yet another brand new one. From my years and years at large companies, these things start because they don't want to share anything in the first place, and then it's just pricey to maintain.

BinaryIdiot 10y ago

To be fair a majority of the big technology companies all write their own tooling for many things that already exist because they didn't quite fit the way they needed them to and they had the resources to re-invent whatever they want. Sure when they open source them it further fragments the market and you always get the rush of "Facebook has almost 2 billion users therefore their tools must be the best!" which further exacerbates the issue but I don't blame Facebook for these issues. It sounds like, in your situation, perhaps there are too many tools being introduced into your projects. Too often I see developers abuse the hell out of npm and the like to just include whatever they need with zero regard for the newly introduced dependency tree and the new tooling that needs to now be maintained.

cwyers 10y ago
When Facebook wrote this, the tooling that already existed wasn't available to them -- ex-Google engineers at Facebook wanted something like Bazel/Blaze, but Google had yet to release that publicly. So this is a Hadoop situation where Google had this internal tool that someone else really wanted and because Google didn't open-source it they wrote their own version.
- asymmetric 10y ago
  Isn't there a risk that reimplementing a previous employer's tool constitutes a legal/contractual violation of some kind?
  - Asooka 10y ago
    It's almost certain that it is a legal violation and if a lawsuit was brought, I'd bet a million dollars in favour of Google; however, the tool itself isn't the final product and it's silly, counterproductive and just outright malicious to sabotage efforts at building good tooling in a competitor's company. Plus, it will set precedent and Google would have to vet every single new internal tool against what the previous employers of their employees were doing.
whywhywhywhy 10y ago
> To be fair a majority of the big technology companies all write their own tooling for many things that already exist because they didn't quite fit the way they needed them Too many engineers syndrome comes to mind too.

perlgeek 10y ago

It seems you lack technical leadership in your organization. Don't blame facebook for it :-)

mhd 10y ago

I'm veering a bit towards the opposite: Some of the Facebook tools look quite nice, but there's a certain amount of taint that goes with their corporate heritage, so if there's a good alternative...

paulddraper 10y ago

What tool would you have suggested Facebook improve? Saying that Facebook should have improved Make or Maven is like saying Linus should have improved CVS or SVN instead of competing with Git. They're in the same space, but they differ at a very fundamental level.

jhasse 10y ago
Waf or Meson
- paulddraper 10y ago
  Facebook doesn't have timetravel, so contributing to Meson would have been impossible.
  - jhasse 10y ago
    Ah I didn't know that Buck was created before 2013. What about Waf though?
    - paulddraper 10y ago
      TBH, I never heard of Waf before. It looks interesting. Buck, Waf, Meson, Bazel....apparently Python is the language for next-gen hash-based build tools :)
      - efficax 10y ago
        Buck is primarily Java, just sayin'
        paulddraper 10y ago
        But the syntax of your build files is Pythonish. Just like Make is in C and the build files are in Make.
hultner 10y ago

thomaslee 10y ago

I can't speak to the rest of Facebook's stuff, but I think the build tool problem is a special case. Per their docs: > Buck is designed for building multiple deliverables from a single repository (a monorepo) rather than across multiple repositories. It has been Facebook's experience that maintaining dependencies in the same repository makes it easier to ensure that all developers have the correct version of all of the code, and simplifies the process of making atomic commits. Having been on the "other side" of the monorepo argument where we tried to make do with improving/extending existing build tools etc. in a rapidly growing engineering org, let me say that Facebook (with Buck), Twitter (Pants? I think?) and Google (with Bazel/Blaze) almost certainly built these to deal with the problem of scaling build management with an ever-growing organization. The popular model of a dozen or so small repositories in GitHub + Jenkins with Maven/NPM/Rake+Bundler/whatever works fine for maybe a few dozen engineers or more, but one day you wake up and realize there hundreds of repositories spread across dozens of _teams_ and hundreds of developers. Obviously you've then got a big ol' dependency graph between repos to deal with, so if you need to fix something near the root suddenly you need to run off bumping version numbers and/or fixing intermediate libraries all the way down the graph. Plus version incompatibilities between the dependencies of different libraries ... it's a total mess, and it doesn't make for an org that can easily "move fast and break things", so to speak. So then to avoid paralysis your options are basically either to silo up (this team owns their stuff, that team owns their stuff, don't bother with shared dependencies) or you go the monorepo route. If you do, then maybe you go and pull all your hundreds of smaller repos into a monorepo. Having everything in one place makes it easier to police the dependency issues within the org & makes it easier for a single engineer to deal with those sort of "cascading changes" instead of shunting that problem onto the entire organization. But in exchange for this "agility" you've then got the problem that builds take multiple hours & the associated tools are often highly language-centric (Maven+Java, NPM+Node, Ruby+Rake, etc.). They don't typically make any reproducibility guarantees either. Anyway, to make a short story really long: at the time FB, Google and Twitter were hitting these organizational scaling walls, making these decisions and building these tools internally, there really weren't any great tools out there for the monorepo use case. I think that's why all these tools have appeared as side-by-side alternatives rather than improvements on one another or to tools like Maven et al. Whether or not consolidation is warranted, for the folks who have the problems that Buck/Bazel/Pants solve, it's likely to save 'em a hell of a lot of time, effort and money IMO. It's a good thing that they have been published, even if the value's maybe not immediately obvious.

LegNeato 10y ago

When Buck was created (my team created it!) there wasn't anything that supported what Facebook needed in a build tool. There still really isn't as all the Blaze work-a-likes are focusing on various different needs. I would argue Buck actually helps to solve what you are concerned about. At Facebook, everything builds with Buck (and Google with Bazel I hear)...which means you learn it once and you know it for your Objective-C library, your Android app, your Cxx service, your python scripts, etc. It really helps to standardize on a build system, and at many companies you can only do that if it supports windows/mac/linux, it supports many languages and platforms, is fast, and is easy to pick up. Buck is all of those things.

NeverTrump 10y ago
Any chance you can compare Pants with Buck?
moondev 10y ago
Gradle?
- sdwilsh 10y ago
  Gradle was not the default Android build tool when Buck was open sourced.
pjmlp 10y ago
> At Facebook, everything builds with Buck (and Google with Bazel I hear). Surely not on Android or Skia teams, unless they internally use something other than what AOSP and Skia use.
fs111 10y ago
> At Facebook, everything builds with Buck Looking at https://github.com/facebook https://github.com/facebook you seem to be using all sorts of build tools...
- sdwilsh 10y ago
  What Facebook uses internally in their monorepo and what they publish to open source for external consumption doesn't have to be the same build tool. Facebook uses fbshipit (https://github.com/facebook/fbshipit/ https://github.com/facebook/fbshipit/) to take slices of their monorepo and publish them to GitHub.

scrame 10y ago

Nope. I totally agree. I don't trust a single thing from facebook. I mean, good on them that they open source their tools, but most everything that comes out is just a huge pain in the ass with very little benefit to adopt, other than we can say we're using open-source facebook technology. The funny thing, though, is that keeping up with other web tech, you always have to deal with facebooks API's and sharing and whatnot, and its also, usually a huge pain in the ass. So, like, what's the benefit of this, switching build systems, other than a tick on the resume for someone who wants to get a job at facebook?

jlg23 10y ago

I'm all for using existing tools, but not for the sake of itself. If there is a good reason (and everyone in the team agrees), all now obsolete tools should simply be replaced by the new one. If this did not work in your office, this is most likely a problem of people in your office not communication properly about their choice of tools. Did you have people make a short presentation when they introduced a new tool? Did nobody ever point out that you already have various tools that can (easily/elegantly) accomplish what the next new thing(tm) is introduced for?

jacques_chester 10y ago

Facebook and Google have a problem I am familiar with: Not Invented Yet Syndrome. Typically they face a problem that few people have ever faced. There may be an existing solution, but not in the open. So they have to roll their own. Others in this thread say Buck was inspired by Blaze. That seems reasonable and I hardly think Facebook can be blamed for rolling their own when that was the only available option. You might as well blame Google for fragmenting the Hadoop ecosystem by creating their MapReduce framework.