7 ms·

Pretty much that. FB has a strong culture of trying to use FOSS 1st, no matter what. The problem is that at the scale FB operates this simply produces 2nd rate

Pretty much that. FB has a strong culture of trying to use FOSS 1st, no matter what. The problem is that at the scale FB operates this simply produces 2nd rate solutions with a ton of duct tape. If you're into solving hard problems Google is a way better place to be. In FB it feels like you're solving a lot of problems so it kind of feels way more productive but you quickly realise that most of these are obvious self-inflicted wounds. There just isn't much political will to invest in large non-customer-facing projects (like Google's GFS and its successor, BigTable, Spanner etc.) and perhaps talent is spread a bit too thin. That being said Google inside too often feels like a cult and I don't miss hearing the word 'googliness' at least 10 times a day.

walterbell 11y ago
> There just isn't much political will to invest in large non-customer-facing projects How about OpenCompute? It was promoted by FB (even if inspired by Google) and had so much impact on the industry that even HP is now having FoxConn build HP-branded OpenCompute-style servers.
- eitally 11y ago
  My company (Newisys, under Sanmina) helped with hardware refinements because they were having cooling/airflow problems. They then contracted with Compal to manufacture the servers and then us to do system assembly. Compal's QC was horrifically bad -- we had about a 10% failure rate of servers they sent us. I'm happy OpenCompute is starting to receive traction, but it's not because FB is awesome at hardware. They've gotten a ton of both raw engineering and also DFM assistance from the pros.
  - nickpsecurity 11y ago
    I'm very grateful for you confirming my intuition on how they're doing hardware. It's so specialist and difficult I figured they had to be outsourcing most of it to pro's. Not a negative on their part, though, as it's what I'd do as well. There's just some holdovers in dicussions that think these companies are all doing (rather than funding) hardware development. EDIT to add: The Horus product is interesting. Formerly, I only knew about the NUMAscale for AMD NUMA. Too bad competition rarely survives in this segment. Things were more interesting in the MPP and DSM days.
- simonebrunozzi 11y ago
  Jonathan Heiliger launched the OpenCompute initiative while at Facebook: https://www.facebook.com/notes/facebook-engineering/building-efficient-data-centers-with-the-open-compute-project/10150144039563920 https://www.facebook.com/notes/facebook-engineering/building... I had the pleasure to meet him now that he is a VC at Vertex Ventures. Very smart and nice guy.
nocarrier 11y ago
It would be cool if you could share when you worked at FB and what general area of the infrastructure you worked on. Your statements don’t match the Facebook infrastructure I know. I recently left after spending 8.5+ years on infra, so I saw it both in the really early days, as well as how it is today. Facebook actually built many major pieces of its infrastructure itself after finding no suitable solution in open source or commercial vendors. Efforts like this were happening even back in the early days of FB infra. I could list out many publicly known Facebook infrastructure projects that solved Hard Problems. And not just in recent years either. :-)
- nickpsecurity 11y ago
  Hard problems like massive scale with strong consistency which three other firms solved? Or "hard" problems like managing COTS components or accelerating PHP that just require time + labor to pull off? Feel free to name projects solving Hard problems whose capabilities had little to no precedent outside Facebook. I mean that straight up rather than sarcastically in case it comes off that way. I'd like to know the best Facebook pulled off.
  - nocarrier 11y ago
    I’ll talk about one abstract thing and two concrete things that I think show that Facebook has been able to solve Hard problems that required the development of new capabilities rarely or never seen before. I find these three things to be very impressive and notable myself, although I am obviously very biased. I should note that this is not an exhaustive list, it’s more just the three examples I am most familiar with and wanted to take the time to write up. First, the abstract thing. I don’t know if most engineers outside Facebook appreciate how much activity there is on the site each day. Hundreds of billions of likes and comments. Billions of photos uploaded. Trillions of photos consumed. And growing each day. And this is for an “online application” in the sense that the data is live and constantly changing. We’re not talking about crawling the web, storing it, doing offline processing, and then building a bunch of indices (which is a different but still legit kind of hard). This is an immense amount of live data producing an even more immense stream of live events; trillions and trillions and trillions that need processing, live and on the fly, every day. It is hard to underestimate how difficult it was to build an application and backend that could drive this kind of social platform. In terms of liveness and scale, there really is nothing out there that can touch Facebook, by orders of magnitude. That is a solved Hard problem in my opinion, if not a meta one. But in my opinion, the largest one. Here’s two concrete things that I thought are good examples of some Hard problems that Facebook has solved: 1. A global media storage platform that each day is capable of ingesting billions of new photos and videos and delivering trillions as well. This includes Haystack, F4, and Cold Storage, which store hot/warm/cold media objects for Facebook. Each storage layer has specialized, custom software running on distinct storage hardware designed to take advantage of the requirements for each layer. Facebook is the largest photo sharing site in the world by orders of magnitude, and they had to build a very custom photos backend to handle the immense load. I’m not even mentioning their terabit class Edge platform which has a global constellation of POPs that accelerates application traffic and caches popular content close to users. Facebook’s global media storage and delivery platform is truly a unique asset. 2. A datacenter architecture focused on high power efficiency and flexible workloads, comprised of custom built: datacenters, racks, servers, storage nodes, network fabric, rack switches, and aggregation switches. Plus some other things that haven’t been made public yet. This architecture let application developers stop working around various physical performance bottlenecks with compute, storage, cache, and network, and instead just focus on optimizing what was best for the application. At the same time, this new architecture also greatly reduced infrastructure costs. There are other solved Hard problems I can talk about that are public, and others that I wish I could talk about that aren’t yet. However, I’m not trying to exhaustively defend what Facebook has built, I more just wanted to respond and show a few examples of the very special scale Facebook has and some of the extremely hard problems they’ve had to solve.
nickpsecurity 11y ago
That matches what I've seen in descriptions of their architectures. Duct tape especially. They are pretty good at gluing together a bunch of FOSS stuff and expanding on it, though. I'll give them that.

walterbell 11y ago

> There just isn't much political will to invest in large non-customer-facing projects How about OpenCompute? It was promoted by FB (even if inspired by Google) and had so much impact on the industry that even HP is now having FoxConn build HP-branded OpenCompute-style servers.

eitally 11y ago
My company (Newisys, under Sanmina) helped with hardware refinements because they were having cooling/airflow problems. They then contracted with Compal to manufacture the servers and then us to do system assembly. Compal's QC was horrifically bad -- we had about a 10% failure rate of servers they sent us. I'm happy OpenCompute is starting to receive traction, but it's not because FB is awesome at hardware. They've gotten a ton of both raw engineering and also DFM assistance from the pros.
- nickpsecurity 11y ago
  I'm very grateful for you confirming my intuition on how they're doing hardware. It's so specialist and difficult I figured they had to be outsourcing most of it to pro's. Not a negative on their part, though, as it's what I'd do as well. There's just some holdovers in dicussions that think these companies are all doing (rather than funding) hardware development. EDIT to add: The Horus product is interesting. Formerly, I only knew about the NUMAscale for AMD NUMA. Too bad competition rarely survives in this segment. Things were more interesting in the MPP and DSM days.
simonebrunozzi 11y ago
Jonathan Heiliger launched the OpenCompute initiative while at Facebook: https://www.facebook.com/notes/facebook-engineering/building-efficient-data-centers-with-the-open-compute-project/10150144039563920 https://www.facebook.com/notes/facebook-engineering/building... I had the pleasure to meet him now that he is a VC at Vertex Ventures. Very smart and nice guy.

nocarrier 11y ago

It would be cool if you could share when you worked at FB and what general area of the infrastructure you worked on. Your statements don’t match the Facebook infrastructure I know. I recently left after spending 8.5+ years on infra, so I saw it both in the really early days, as well as how it is today. Facebook actually built many major pieces of its infrastructure itself after finding no suitable solution in open source or commercial vendors. Efforts like this were happening even back in the early days of FB infra. I could list out many publicly known Facebook infrastructure projects that solved Hard Problems. And not just in recent years either. :-)

nickpsecurity 11y ago
Hard problems like massive scale with strong consistency which three other firms solved? Or "hard" problems like managing COTS components or accelerating PHP that just require time + labor to pull off? Feel free to name projects solving Hard problems whose capabilities had little to no precedent outside Facebook. I mean that straight up rather than sarcastically in case it comes off that way. I'd like to know the best Facebook pulled off.
- nocarrier 11y ago
  I’ll talk about one abstract thing and two concrete things that I think show that Facebook has been able to solve Hard problems that required the development of new capabilities rarely or never seen before. I find these three things to be very impressive and notable myself, although I am obviously very biased. I should note that this is not an exhaustive list, it’s more just the three examples I am most familiar with and wanted to take the time to write up. First, the abstract thing. I don’t know if most engineers outside Facebook appreciate how much activity there is on the site each day. Hundreds of billions of likes and comments. Billions of photos uploaded. Trillions of photos consumed. And growing each day. And this is for an “online application” in the sense that the data is live and constantly changing. We’re not talking about crawling the web, storing it, doing offline processing, and then building a bunch of indices (which is a different but still legit kind of hard). This is an immense amount of live data producing an even more immense stream of live events; trillions and trillions and trillions that need processing, live and on the fly, every day. It is hard to underestimate how difficult it was to build an application and backend that could drive this kind of social platform. In terms of liveness and scale, there really is nothing out there that can touch Facebook, by orders of magnitude. That is a solved Hard problem in my opinion, if not a meta one. But in my opinion, the largest one. Here’s two concrete things that I thought are good examples of some Hard problems that Facebook has solved: 1. A global media storage platform that each day is capable of ingesting billions of new photos and videos and delivering trillions as well. This includes Haystack, F4, and Cold Storage, which store hot/warm/cold media objects for Facebook. Each storage layer has specialized, custom software running on distinct storage hardware designed to take advantage of the requirements for each layer. Facebook is the largest photo sharing site in the world by orders of magnitude, and they had to build a very custom photos backend to handle the immense load. I’m not even mentioning their terabit class Edge platform which has a global constellation of POPs that accelerates application traffic and caches popular content close to users. Facebook’s global media storage and delivery platform is truly a unique asset. 2. A datacenter architecture focused on high power efficiency and flexible workloads, comprised of custom built: datacenters, racks, servers, storage nodes, network fabric, rack switches, and aggregation switches. Plus some other things that haven’t been made public yet. This architecture let application developers stop working around various physical performance bottlenecks with compute, storage, cache, and network, and instead just focus on optimizing what was best for the application. At the same time, this new architecture also greatly reduced infrastructure costs. There are other solved Hard problems I can talk about that are public, and others that I wish I could talk about that aren’t yet. However, I’m not trying to exhaustively defend what Facebook has built, I more just wanted to respond and show a few examples of the very special scale Facebook has and some of the extremely hard problems they’ve had to solve.

nickpsecurity 11y ago

That matches what I've seen in descriptions of their architectures. Duct tape especially. They are pretty good at gluing together a bunch of FOSS stuff and expanding on it, though. I'll give them that.