5 ms·

Would be interesting to learn, if at all possible, how these kinds of contributions are justified to upper management/financial folks. For example, how (and if

Would be interesting to learn, if at all possible, how these kinds of contributions are justified to upper management/financial folks. For example, how (and if) they estimated the cost of this new load balancing technology in terms of money saved/user experience/security. BTW, if I am a corporation betting my sales on F5 load balancers, I would be worried (and would probably notify my shareholders of the risk).

wpietri 8y ago
I'd love to see the full breakdown, but I think hints of it are there in the text. E.g., for the load balancing they say, it "helped improve the performance and scalability of network load balancing while drastically reducing inefficiencies." At Facebook's scale, those are pretty easy to translate into dollars by, e.g., looking at how many machines you now didn't have to buy to keep up with growth.
tjkrusinski 8y ago
Relevant: https://code.fb.com/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ https://code.fb.com/production-engineering/introducing-data-...
Jnr 8y ago
I haven't checked what they have released so I could be wrong but my guess is that by open sourcing their tools they might hope to merge it into Linux kernel so that they would not have to maintain their kernel patches. Additionally they could get improvements from others.
- mikekchar 8y ago
  I'm not quite sure if the OP meant justifying open sourcing their tools or writing them in the first place. However, for open sourcing, I'm pretty sure I could justify that pretty easily just on reducing the cost of recruitment. There was a time that Facebook had (at least in my circles) a terrible reputation as an employer. Working at Facebook would be a laughable job. Now it's one of the most sought after jobs in the industry. Not only that, but people regularly spend time getting familiar with their in house tools before they even send in a resume. For me, it's basically a no-brainer.
diegs 8y ago
I don't have firsthand knowledge of obtaining such justifications, but a couple things come to mind that seem reasonable to me as an engineer who has to propose any sort of project (and I worked at Google that also makes upstream contributions and was able to glimpse into some of the "contribute to open source" justifications): - If you're already using gnu/linux, building functionality you need on top of what you have is most likely the easiest and least risky way to go (see also: worse is better[1]). Google and now apparently FB definitely built stuff on top of Linux to meet their specific use cases (Google was a big contributor to the original cgroups IIRC), and it worked out quite well for them. - I would imagine that the way this might go is that a bright engineer/team identifies an opportunity to make a significant improvement, proposes it, does some back-of-the-envelope calculations of the potential benefits it brings, and convinces management that the risk/reward ratio is satisfactory. This is, generally speaking, a critical skill to possess as an engineer in any organization. But especially at the scale of Google/Facebook, if you can propose something that will save even single-digit percent in CPU/RAM/DISK usage or other operational considerations, that translates to many millions of dollars. (Side story--and hopefully I'm not violating any NDAs by saying this--when I was at Google Jeff Dean was leading a grassroots effort to identify and remove wasteful code, such as extraneous log statements and unnecessary string formatting, and maintained a document that cataloged how much $/year (hint: often more than most people's salary's) each improvement saved) - Generally, I feel like most of these projects likely come about incrementally, perhaps with some patches here or there, some ideas that come about based on experience or academic/PhD research from employees, and then eventually that accumulates into something that ends up being cleaned up and upstreamed. - I also think that generally speaking, a lot of engineers do want to express pride in their work, and open sourcing it is one way to achieve that. Furthermore, there is likely a tradeoff of competitive advantage vs community goodwill and also getting other people outside of your company to maintain, debug, and improve what you made, and I think Google and Facebook have an engineering culture of preferring the latter to the former. At the end of the day, Google is profitable because of its search/ad algorithms and Facebook because of its social graph and ad platform. Their tech stacks are necessary but not sufficient, and furthermore are often don't necessarily provide that much benefit to others who aren't running at the same scale (realistically speaking, you can get a lot of mileage out of a single, relatively modest server running OpenBSD or something) [1] https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html

wpietri 8y ago

I'd love to see the full breakdown, but I think hints of it are there in the text. E.g., for the load balancing they say, it "helped improve the performance and scalability of network load balancing while drastically reducing inefficiencies." At Facebook's scale, those are pretty easy to translate into dollars by, e.g., looking at how many machines you now didn't have to buy to keep up with growth.

tjkrusinski 8y ago

Relevant: https://code.fb.com/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ https://code.fb.com/production-engineering/introducing-data-...

Jnr 8y ago

I haven't checked what they have released so I could be wrong but my guess is that by open sourcing their tools they might hope to merge it into Linux kernel so that they would not have to maintain their kernel patches. Additionally they could get improvements from others.

mikekchar 8y ago
I'm not quite sure if the OP meant justifying open sourcing their tools or writing them in the first place. However, for open sourcing, I'm pretty sure I could justify that pretty easily just on reducing the cost of recruitment. There was a time that Facebook had (at least in my circles) a terrible reputation as an employer. Working at Facebook would be a laughable job. Now it's one of the most sought after jobs in the industry. Not only that, but people regularly spend time getting familiar with their in house tools before they even send in a resume. For me, it's basically a no-brainer.

diegs 8y ago

I don't have firsthand knowledge of obtaining such justifications, but a couple things come to mind that seem reasonable to me as an engineer who has to propose any sort of project (and I worked at Google that also makes upstream contributions and was able to glimpse into some of the "contribute to open source" justifications): - If you're already using gnu/linux, building functionality you need on top of what you have is most likely the easiest and least risky way to go (see also: worse is better[1]). Google and now apparently FB definitely built stuff on top of Linux to meet their specific use cases (Google was a big contributor to the original cgroups IIRC), and it worked out quite well for them. - I would imagine that the way this might go is that a bright engineer/team identifies an opportunity to make a significant improvement, proposes it, does some back-of-the-envelope calculations of the potential benefits it brings, and convinces management that the risk/reward ratio is satisfactory. This is, generally speaking, a critical skill to possess as an engineer in any organization. But especially at the scale of Google/Facebook, if you can propose something that will save even single-digit percent in CPU/RAM/DISK usage or other operational considerations, that translates to many millions of dollars. (Side story--and hopefully I'm not violating any NDAs by saying this--when I was at Google Jeff Dean was leading a grassroots effort to identify and remove wasteful code, such as extraneous log statements and unnecessary string formatting, and maintained a document that cataloged how much $/year (hint: often more than most people's salary's) each improvement saved) - Generally, I feel like most of these projects likely come about incrementally, perhaps with some patches here or there, some ideas that come about based on experience or academic/PhD research from employees, and then eventually that accumulates into something that ends up being cleaned up and upstreamed. - I also think that generally speaking, a lot of engineers do want to express pride in their work, and open sourcing it is one way to achieve that. Furthermore, there is likely a tradeoff of competitive advantage vs community goodwill and also getting other people outside of your company to maintain, debug, and improve what you made, and I think Google and Facebook have an engineering culture of preferring the latter to the former. At the end of the day, Google is profitable because of its search/ad algorithms and Facebook because of its social graph and ad platform. Their tech stacks are necessary but not sufficient, and furthermore are often don't necessarily provide that much benefit to others who aren't running at the same scale (realistically speaking, you can get a lot of mileage out of a single, relatively modest server running OpenBSD or something) [1] https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html