6 ms·

Used to work at fb in an infra team. Their abstraction for job scheduling (Tupperware) is about 5 years behind - something like Borg or EC2/EMR. Something like

Used to work at fb in an infra team. Their abstraction for job scheduling (Tupperware) is about 5 years behind - something like Borg or EC2/EMR. Something like that is a fundamental reason why fb can’t do cloud as it is right now. Plus, the infra teams do operate like product - impact at all costs. Which to most management translates to short-term impact over technical quality. It would be a true 180 in terms of eng culture if they could pull off a cloud platform. An example is how they bought Parse and killed it, while firebase at google is doing extremely well. Having said all that, I think focusing on impact over technical quality is probably the right business decision for what they were trying to do at the time - drive engagement and revenue.

thedance 7y ago
When you say that Tupperware is 5 years behind Borg, what do you mean? It would take 5 years to add current Borg features? It lacks features Borg had 5 years ago? Or it would take 5 years to reach parity with Borg as launched, i.e. it's actually 17+ years behind?
- justicezyx 7y ago
  The last statement is factually wrong. Borg was launched 2005, 5 years behind would be 20 years.
ipsum2 7y ago
I think 5 years behind may be an exaggeration. People can read https://engineering.fb.com/data-center-engineering/tupperware/ https://engineering.fb.com/data-center-engineering/tupperwar... and make up their own mind.
- dodobirdlord 7y ago
  5 years might actually be right on the mark? I think Borg and EC2 et al. had all of this functionality 5 years ago, but I'm not sure it was all there more than 5 years ago.
deleted 7y ago
[deleted]
anaisbetts 7y ago
Yep, I have to agree. Facebook in some ways is incredibly ahead of the game, and in other ways, is shockingly behind - I was baffled when I started working there.
- tonymet 7y ago
  say more
  - ericlewis 7y ago
    From what I’ve seen their dev tools internally are incredible- Facebook engineers constantly are saying that github is crazy behind and I’ve witnessed things that they can do that would make that true. Other times... they still have pretty old school hookups and getting stuff provisioned can be a pain (but if you can make a compelling business case this can all change)
  - kerng 7y ago
    Does Facebook still use Phabricator?
    - objclxt 7y ago
      Yes, but it has forked so much that it bears very little resemblance to the public OSS version.
nappy-doo 7y ago
Also, the majority of their resources are web machines, running HHVM. These machines are just serving monkeys, which are highly optimized for that workload.
mlinsey 7y ago
this is true, but a lot of resources put into a new team that was completely walled off from the rest of the company, in terms of roadmaps and eng culture, could overcome this problem. I think someone at Amazon in the mid-2000s would not have necessarily looked at their internal stack vs. other companies and picked them to become the breakout cloud winner, but when I was there (late 2000s), the AWS team was kept quite separate, and aside from things like S3 the retail side of the company wasn't really using much in the way of AWS infrastructure yet.
- harikb 7y ago
  Amazon starting a cloud offering is quite different from anyone coming later. Even though lot of open source infra exists, expectations from dev/companies in 2015+ are sky high. If FB really wanted to have an offering, it needs to be specialized in something specific. May be Oculus game server hosting or something tied to FB/WhatsApp/Insta features
Abishek_Muthian 7y ago
Killing of Parse definitely showed that Facebook didn't want to be in PaaS/IaaS. Now, there's going to be severe trust issues with any PaaS/IaaS with a Facebook branding. Parse was ahead of its time, good Engineering, nice documentations and possible the first platform to make building backend/database system for an app seamless. Even when Facebook killed it, they did a great job at open-sourcing the platform.[1] I had migrated my existing services to open-source Parse and even built a stateless application with >250,000 users with it few years back. But if I had to build the same backend now, I'll probably do it with Go/pgSQL instead of NodeJS/MongoDB as in Parse. [1]https://parseplatform.org/ https://parseplatform.org/
- mst 7y ago
  I mean, yes, but also a reputation for killing things off hasn't entirely stopped people from using GCP.
  - Abishek_Muthian 7y ago
    I should have been more clearer, FB's reputation loss is not just from killing things!
    - weare138 7y ago
      Honestly this was the first thing I thought when I read the headline. No one using enterprise cloud services is going to trust Facebook with their data and mission critical systems.
  - lowdose 7y ago
    Well Google has invested more than $30 billion in the could last 3 years. This wait until Google is going to pull the plug is getting a bit annoying. I welcome the competition with AWS.
haolez 7y ago
Is Parse dead?
- angott 7y ago
  Yes.
pier25 7y ago
> firebase at google is doing extremely well Source? It's certainly popular but everyone I know (me included) is running away from it asap.
- batter 7y ago
  Our managers keep saying GCP is the best, and a lot of developers hate it. But who cares. And we also tried Firebase, luckily idea was abandoned.
  - jsolson 7y ago
    > a lot of developers hate it Any particular complaints? (I work on Compute Engine)
    - smt88 7y ago
      Not GP, but GCP's GUI is incredibly confusing and difficult to use. It is clearly a variety of products (poorly) stitched together. That has nothing to do with Compute Engine, and AWS is just as bad (if not worse). My new projects will all be on Azure because Microsoft has pivoted to a company with reliable investment in UI, developer-friendliness, and long-term support. Serving businesses has also been a core of their company from the beginning, whereas Google seems unwilling to provide the human support required and AWS is clearly at odds with Amazon's primary culture. It does seem like the underlying products in GCP are very good, but they mostly seem to replicate other offerings from Azure and AWS.
      - etaioinshrdlu 7y ago
        There are also those such as myself who instead feel that Azure's UI is by far by worst of the big three, and that GCP is not bad at all, and that AWS is big, complex, and ugly, yet practical. The long term support story of AWS is also great.
        smt88 7y ago
einarvollset 7y ago
Killing Parse really surprised me at the time.

thedance 7y ago

When you say that Tupperware is 5 years behind Borg, what do you mean? It would take 5 years to add current Borg features? It lacks features Borg had 5 years ago? Or it would take 5 years to reach parity with Borg as launched, i.e. it's actually 17+ years behind?

justicezyx 7y ago
The last statement is factually wrong. Borg was launched 2005, 5 years behind would be 20 years.

ipsum2 7y ago

I think 5 years behind may be an exaggeration. People can read https://engineering.fb.com/data-center-engineering/tupperware/ https://engineering.fb.com/data-center-engineering/tupperwar... and make up their own mind.

dodobirdlord 7y ago
5 years might actually be right on the mark? I think Borg and EC2 et al. had all of this functionality 5 years ago, but I'm not sure it was all there more than 5 years ago.

deleted 7y ago

[deleted]

anaisbetts 7y ago

Yep, I have to agree. Facebook in some ways is incredibly ahead of the game, and in other ways, is shockingly behind - I was baffled when I started working there.

tonymet 7y ago
say more
- ericlewis 7y ago
  From what I’ve seen their dev tools internally are incredible- Facebook engineers constantly are saying that github is crazy behind and I’ve witnessed things that they can do that would make that true. Other times... they still have pretty old school hookups and getting stuff provisioned can be a pain (but if you can make a compelling business case this can all change)
- kerng 7y ago
  Does Facebook still use Phabricator?
  - objclxt 7y ago
    Yes, but it has forked so much that it bears very little resemblance to the public OSS version.

nappy-doo 7y ago

Also, the majority of their resources are web machines, running HHVM. These machines are just serving monkeys, which are highly optimized for that workload.

mlinsey 7y ago

this is true, but a lot of resources put into a new team that was completely walled off from the rest of the company, in terms of roadmaps and eng culture, could overcome this problem. I think someone at Amazon in the mid-2000s would not have necessarily looked at their internal stack vs. other companies and picked them to become the breakout cloud winner, but when I was there (late 2000s), the AWS team was kept quite separate, and aside from things like S3 the retail side of the company wasn't really using much in the way of AWS infrastructure yet.

harikb 7y ago
Amazon starting a cloud offering is quite different from anyone coming later. Even though lot of open source infra exists, expectations from dev/companies in 2015+ are sky high. If FB really wanted to have an offering, it needs to be specialized in something specific. May be Oculus game server hosting or something tied to FB/WhatsApp/Insta features

Abishek_Muthian 7y ago

Killing of Parse definitely showed that Facebook didn't want to be in PaaS/IaaS. Now, there's going to be severe trust issues with any PaaS/IaaS with a Facebook branding. Parse was ahead of its time, good Engineering, nice documentations and possible the first platform to make building backend/database system for an app seamless. Even when Facebook killed it, they did a great job at open-sourcing the platform.[1] I had migrated my existing services to open-source Parse and even built a stateless application with >250,000 users with it few years back. But if I had to build the same backend now, I'll probably do it with Go/pgSQL instead of NodeJS/MongoDB as in Parse. [1]https://parseplatform.org/ https://parseplatform.org/

mst 7y ago
I mean, yes, but also a reputation for killing things off hasn't entirely stopped people from using GCP.
- Abishek_Muthian 7y ago
  I should have been more clearer, FB's reputation loss is not just from killing things!
  - weare138 7y ago
    Honestly this was the first thing I thought when I read the headline. No one using enterprise cloud services is going to trust Facebook with their data and mission critical systems.
- lowdose 7y ago
  Well Google has invested more than $30 billion in the could last 3 years. This wait until Google is going to pull the plug is getting a bit annoying. I welcome the competition with AWS.

haolez 7y ago

Is Parse dead?

angott 7y ago
Yes.

pier25 7y ago

> firebase at google is doing extremely well Source? It's certainly popular but everyone I know (me included) is running away from it asap.

batter 7y ago
Our managers keep saying GCP is the best, and a lot of developers hate it. But who cares. And we also tried Firebase, luckily idea was abandoned.
- jsolson 7y ago
  > a lot of developers hate it Any particular complaints? (I work on Compute Engine)
  - smt88 7y ago
    Not GP, but GCP's GUI is incredibly confusing and difficult to use. It is clearly a variety of products (poorly) stitched together. That has nothing to do with Compute Engine, and AWS is just as bad (if not worse). My new projects will all be on Azure because Microsoft has pivoted to a company with reliable investment in UI, developer-friendliness, and long-term support. Serving businesses has also been a core of their company from the beginning, whereas Google seems unwilling to provide the human support required and AWS is clearly at odds with Amazon's primary culture. It does seem like the underlying products in GCP are very good, but they mostly seem to replicate other offerings from Azure and AWS.
    - etaioinshrdlu 7y ago
      There are also those such as myself who instead feel that Azure's UI is by far by worst of the big three, and that GCP is not bad at all, and that AWS is big, complex, and ugly, yet practical. The long term support story of AWS is also great.
      - smt88 7y ago

einarvollset 7y ago

Killing Parse really surprised me at the time.