4 ms·

Very cool article. Amazing how much tooling Netflix has built themselves.

by mattiemass 11y ago

hitekker 11y ago
I believe at least some parts of the tooling were developed by an embedded team from Pivotal Labs.
- bmoyles 11y ago
  Not that I am aware of. It's possible they contributed to some of our OSS efforts, but I don't believe anything used internally was developed in that manner.
  - lintoma 11y ago
    Pivotal ( the company that engulfed Pivotal Labs and Spring ) did contribute the Cloud Foundry integration in Spinnaker. This is not used at Netflix, though.
- cryowaffle 11y ago
  Huh? The article describes a massive system and there is a lot of abstraction here, are you suggesting Pivotal was a part of the large abstraction or just grabbing credit for some minor part?
riskable 11y ago
What's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.
- chatmasta 11y ago
  It helps that they have repeatedly proven this strategy works. When the CTO goes to the CEO and says "we're going to build this, not buy it," the CEO obviously trusts his decision making because he's seen him successfully execute his strategy so many times. It sounds like Netflix engineering culture is built a lot around trust. Management trusts that only top engineering talent works there, and the sink-or-swim culture, coupled with performance bonuses and a rising stock price, ensures that the engineers are making the best decisions for the company. It's a lot easier to approve a "build" vs "buy" decision when you know that the interests of the engineers pushing for it are actually aligned with the company business interests. Contrast this to a company with a large separation of incentives between the stock price and the engineers. In that scenario you end up with a bunch of engineers who are bored and want to prematurely optimize systems because there is no clear personal cost to doing so.
  - riskable 11y ago
    > It helps that they have repeatedly proven this strategy works. I laughed at this because you can't "repeatedly prove" you're capable of developing your own solutions if you're never allowed to do it in the first place! The enterprise mantra is: COTS or it doesn't happen.
- jdblair 11y ago
  Fairly new Netflix employee here (started last November). From what I can see so far the "highly aligned, loosely coupled" culture here works very well. The company has a clear focus, we all get it, and we all work to make it happen. Teams have freedom to accomplish their goals they best way they see fit. There is quite simply not a lot of focus on particular means, the focus is on outcomes.
- draw_down 11y ago
  I've been in so many places where that response would be the correct one. I've spent a lot of time decommissioning some rather horrible outcomes of "can't we just build it ourselves".
  - riskable 11y ago
    When companies treat engineers like disposable commodities they should expect solutions that will be disposed.
    - draw_down 11y ago
      NIH is the opposite of "disposable engineers". It's saying your engineers are so much more precious and able than the barbarians outside the gate, that they should write their own test harness/DOM selection library/insert commodity software genre here.
      - riskable 11y ago
        To be fair most COTS solutions made for enterprises are absolute crap. It doesn't take much to improve upon them. Enterprise tools are well-known to have absolutely horrible user experiences, extremely complicated configuration and architecture, and even worse documentation. I don't blame any manager/PM for thinking, "we can do better" with an internal team.
- ArkyBeagle 11y ago
  Netflix cares about "it works" and they have to have tools.
  - riskable 11y ago
    They also seem to have technical management as well. That makes a huge difference in building out and maintaining your own tools and also in hiring and retaining effective engineering talent.
    - ArkyBeagle 11y ago
      I think it's more interesting to note that they couldn't exactly buy that from anyone else.
- goalieca 11y ago
  "let's wait for the community to fix the opensource code we rely on". I've heard this a lot in my career (even academia)
- jschmitz28 11y ago
  Do they? The only big company I've been at (Amazon) has an absolute monster of an inhouse build and deployment/pipelining system. Although it could just be that they've been around longer than open source alternatives.

Very cool article. Amazing how much tooling Netflix has built themselves.

hitekker 11y ago

I believe at least some parts of the tooling were developed by an embedded team from Pivotal Labs.

bmoyles 11y ago
Not that I am aware of. It's possible they contributed to some of our OSS efforts, but I don't believe anything used internally was developed in that manner.
- lintoma 11y ago
  Pivotal ( the company that engulfed Pivotal Labs and Spring ) did contribute the Cloud Foundry integration in Spinnaker. This is not used at Netflix, though.
cryowaffle 11y ago
Huh? The article describes a massive system and there is a lot of abstraction here, are you suggesting Pivotal was a part of the large abstraction or just grabbing credit for some minor part?

riskable 11y ago

What's amazing is that Netflix let their teams develop solutions from scratch. Usually at big companies when their developers say something like, "can't we just build a solution ourselves?" they're laughed out of the room or, more likely, marked down as candidates in the next round of layoffs.

chatmasta 11y ago
It helps that they have repeatedly proven this strategy works. When the CTO goes to the CEO and says "we're going to build this, not buy it," the CEO obviously trusts his decision making because he's seen him successfully execute his strategy so many times. It sounds like Netflix engineering culture is built a lot around trust. Management trusts that only top engineering talent works there, and the sink-or-swim culture, coupled with performance bonuses and a rising stock price, ensures that the engineers are making the best decisions for the company. It's a lot easier to approve a "build" vs "buy" decision when you know that the interests of the engineers pushing for it are actually aligned with the company business interests. Contrast this to a company with a large separation of incentives between the stock price and the engineers. In that scenario you end up with a bunch of engineers who are bored and want to prematurely optimize systems because there is no clear personal cost to doing so.
- riskable 11y ago
  > It helps that they have repeatedly proven this strategy works. I laughed at this because you can't "repeatedly prove" you're capable of developing your own solutions if you're never allowed to do it in the first place! The enterprise mantra is: COTS or it doesn't happen.
jdblair 11y ago
Fairly new Netflix employee here (started last November). From what I can see so far the "highly aligned, loosely coupled" culture here works very well. The company has a clear focus, we all get it, and we all work to make it happen. Teams have freedom to accomplish their goals they best way they see fit. There is quite simply not a lot of focus on particular means, the focus is on outcomes.
draw_down 11y ago
I've been in so many places where that response would be the correct one. I've spent a lot of time decommissioning some rather horrible outcomes of "can't we just build it ourselves".
- riskable 11y ago
  When companies treat engineers like disposable commodities they should expect solutions that will be disposed.
  - draw_down 11y ago
    NIH is the opposite of "disposable engineers". It's saying your engineers are so much more precious and able than the barbarians outside the gate, that they should write their own test harness/DOM selection library/insert commodity software genre here.
    - riskable 11y ago
      To be fair most COTS solutions made for enterprises are absolute crap. It doesn't take much to improve upon them. Enterprise tools are well-known to have absolutely horrible user experiences, extremely complicated configuration and architecture, and even worse documentation. I don't blame any manager/PM for thinking, "we can do better" with an internal team.
ArkyBeagle 11y ago
Netflix cares about "it works" and they have to have tools.
- riskable 11y ago
  They also seem to have technical management as well. That makes a huge difference in building out and maintaining your own tools and also in hiring and retaining effective engineering talent.
  - ArkyBeagle 11y ago
    I think it's more interesting to note that they couldn't exactly buy that from anyone else.
goalieca 11y ago
"let's wait for the community to fix the opensource code we rely on". I've heard this a lot in my career (even academia)
jschmitz28 11y ago
Do they? The only big company I've been at (Amazon) has an absolute monster of an inhouse build and deployment/pipelining system. Although it could just be that they've been around longer than open source alternatives.