8 ms·

Netflix VP of IT on the Future of Infrastructure

78 points by dataisfun 13y ago

numlocked 13y ago
The jargon and acronyms in this interview are intense. It's pretty clearly an industry interview so it's my fault that I don't know the phrases, but I'm a little surprised by how impenetrable it is to me (a software engineer who has worked in large corp environments). Anyway care to expand on some of the less Googleable acronyms? - MDM/MAM - NAC - EDW (synonymous with ETL?)
- dataisfun 13y ago
  Thanks for tip. These are now expanded in interview text :)
- martinald 13y ago
  MDM/MAM = mobile device management/mobile application management (managing and provisioning mobile devices and their applications, generally automatically) NAC = network access control EDW = enterprise data warehouse (archiving old information while preserving access)
obblekk 13y ago
Forgive my ignorance. Is `IT` the same as `engineering` at other companies, or is this something else?
- justizin 13y ago
  His actual title is "VP of IT Operations", which is usually part of an Engineering department, but very close to the executive team because of things like budget and business guarantees. Note the Unix / Networking background.
- bri3d 13y ago
  "IT" is too vaguely defined to mean much anymore, but based on this interview I suspect it's internal infrastructure at Netflix. Stuff like staff PCs, internal data warehouses, sales, finance, and marketing software support, WiFi APs, routers, keeping the backoffice servers and network up, ensuring reliable WAN and LAN connectivity so engineers can reach production securely, intrusion detection and analysis, and so on. Generally in smaller software companies I hear R+D and consumer-facing applications referred to as "engineering" with external-facing infrastructure (like the production datacenter) referred to as "operations," with "IT" being reserved for this internal backoffice kind of stuff. In other places, especially larger corporations, I've often heard everything having to do with a computer lumped in as "IT."
  - xivzgrev 13y ago
    Yea I was surprised. When I saw the title I thought infrastructure meant network infrastructure, which seems like a white-hot area of innovation given how much internet traffic they consume. But no, this interview was on internal IT.
- dpritchett 13y ago
  IT is usually the catchall infrastructure/support function. For some companies (like Netflix) you probably also have a product engineering group.
- puppetmaster3 13y ago
  It's office automation. As opposed to Netops, that operates their business such as video streaming.
zobzu 13y ago
Reading the slides make me think this is full of nothing :| How is 802.11ac speed making things "more cloud"? Because you get slightly more bandwidth -maybe- if you have a new laptop and also you dont have everyone using it? I don't get it. Requiring VPN everywhere, how is that cloudy? Finally, using stuff like AWS is nice, but unless they have a specific contract (which they may since they advertise them a lot), its a LOT more expensive when you start having a lot of processing (ie big companies like netflix)
- cwp 13y ago
  Well, if anybody could get a good deal from AWS, it's Netflix.
purephase 13y ago
I thought the 2014 Technology Roadmap [1] was an interesting read. For an organization as "young" as Netflix, I was surprised by the technology debts that they've accumulated and the aggressive tone that they've set to transition. I think it's amazing the decisions that get made with explosive growth/hiring that end-up on roadmaps that read similarly to organizations that have been around much longer. There's no criticism here. I think Netflix is an amazing company and it is the this sort of strategic vision (and the openess of both it and the organization overall) that reminds me that we're all on this rocky ship together and it's amazing that any of it works sometimes. [1] http://www.slideshare.net/mdkail/it-ops-2014-technology-roadmap http://www.slideshare.net/mdkail/it-ops-2014-technology-road...
teacup50 13y ago
> The notion that something needs to remain on-premise is really an Old World way of thinking and feels more like someone wanting control as opposed to there being a valid argument. No, it's the business continuity way of thinking. Outsourcing commodities -- such as servers, virtual or otherwise -- is one thing. Outsourcing your core operational tools, software, and all your data is another matter entirely. Preferring SaaS at a company large enough to afford on-premise solutions is just nonsensical, and I expect it'll either blow up in his face, or just create a never-ending tax on end users who are constantly dealing with a mishmash of vendors, accounts, disappearing services, broken software, and instability. At scale, stability and continuity is worth more than the opex/capex costs of internal IT.
- couradical 13y ago
  I can see both sides of the coin though - for a company that is as heavily cloud-invested as Netflix is - it might make less sense to maintain a cage for internal IT in a few datacenters. That said, I do get worried about sensitive data in the cloud. I didn't get that he was talking about moving to cloud/SaaS vendors as a whole, but more the IaaS/PaaS space - the "Hey, this app runs RoR, can I just run it from a Heroku dyno rather than a VMWare box in a cage" type of move. Granted, there's still a cost associated with that, and a need to plan the move, but it's less of an issue. They are making heavy use of SAML SSO it appears, so the account/vendor thing probably wouldn't be as much of an issue.
- binarytrees 13y ago
  You have a point, but having services on-premise can still achieve the disfunction, mishmash and overall goods times you're describing with a SaaS service. You might get better response times knowing it's a local problem vs let's say an office in NYC hosting your ticketing system. Regardless, I'm sure they have put a lot of thought into this set up and to me sounds amazing. No ties to a physical location = win.
- e12e 13y ago
  > Preferring SaaS at a company large enough to afford on-premise solutions is just nonsensical Yes and no. First off, I don't think Netflix can afford to build the kind of infrastructure they're using -- certainly not without changing focus of their engineering resources. They famously do a lot of work to move data closer to the end users globally -- they're not "just" a US company. Essentially, they'd have to have an operation that would be "qualitatively" similar to AWS in order to be able to do what they do (and at a smaller scale overall, I think that would end up being quite expensive). I'd argue Netflix is one of the few companies I can think of where this "all cloud all the time" idea for infrastructure might actually make sense. I agree it's a big risk though -- and probably not good advice for most companies. It would also appear that Netflix is planning on actually selling a product (video rental) and make money off that, rather than have that merely as a vehicle to drive other, sometimes tangential, innovation. That remains to be seen, of course.
qthrul 13y ago
TLDR: The approach we take for IT works (for us at this point in time in the scope defined as IT by me and/or our internal customers). Netflix talks generally can be fascinating and inspiring. However, when considering IT it's also important to consider the charter and challenges of Netflix IT. i.e. it's no more valid or invalid that the talks of how IT is delivered in so-called build vs. broker models in other companies in other industries http://dilbert.com/strips/comic/2013-07-05/ http://dilbert.com/strips/comic/2013-07-05/
nessup 13y ago
Why was there no discussion of the ethics of the Comcast deal?
- cdcarter 13y ago
  Because this was a conversation about internal infrastructures?
- wookiefeet 13y ago
  Netflix IT would be an odd group to ask about the deal, probably wouldn't have made sense in this article
welder 13y ago
> zero-trust network architecture This makes me think of http://meldium.com http://meldium.com
e12e 13y ago
It'd be great if Netflix (or some other company) manages to do some heavy lifting in creating a viable, modern, certificate-based authentication and authorization stack, that's easier to deploy. Essentially an upgraded take on kerberos (move off shared secrets, perhaps), AFS (I still don't know what a viable way forward for secure, distributed, locally cacheable network filesystem is -- maybe DAV+TLS+regular caching?). I suppose LDAP might be fine as a user/principal/authorization database, but some distribution that uses internal CA and demands TLS as default would be a good start. The last "innovation" I'm aware of in this area, is skolelinux/edulinux work with packaging samba/ldap/kerberos/lts in a easy(ier) to manage package for Debian: https://wiki.debian.org/DebianEdu/Documentation/Wheezy/Architecture https://wiki.debian.org/DebianEdu/Documentation/Wheezy/Archi...
- zobzu 13y ago
  kerberos is very good. everyone reinvents kerberos every month. it doesn't have to be new to be "modern". Kerberos is still modern by today's standards, in fact. the problem is having tools that communicate with each others and an easy setup. yeah, SAML kinda sucks to use too.. and works like kerberos anyway. OpenID, Hawk, etc - also in fact work exactly the same.
  - e12e 13y ago
    IIRC there are a couple of things that might be tweaked in the protocol wrt sign before encrypt? Also, with non-encumbered public key cryptography available, and no longer prohibitively resource intensive, we can do better than NxN shared secrets. I do indeed like kerberos - but I still want a straight forward and reasonably robust framework built on certificates.
  - eropple 13y ago
    Seriously. At my day job we're investigating better ways to handle single sign on and authentication and every road leads back to Kerberos.
    - e12e 13y ago
vespaceballs6 13y ago
I really wanted to share this article with my friends, but it was so filled with buzzwords that even my dev friends wouldn't ascertain much. Keep in mind I'm an idiot, and I have idiot friends.
- dataisfun 13y ago
  haha. Well, I doubt you're an idiot. Buzzwords serve a purpose insofar as they're a good shorthand for defining categories of product offerings. E.g., ETL, Data Warehouse, Mobile Device Management, etc. all have pretty well understood parameters within which the vendors, buyers, analysts, etc. operate.
cottonseed 13y ago
One thing that stuck out to me: > We are implementing “certificate-based authentication” instead of the standard username/password auth against Active Directory. I wish we were all doing this. How long is it going to take to get a usable certificate-based client/user authentication mechanism on the web? edit: Also see e12e's comment.

78 points by dataisfun 13y ago

The jargon and acronyms in this interview are intense. It's pretty clearly an industry interview so it's my fault that I don't know the phrases, but I'm a little surprised by how impenetrable it is to me (a software engineer who has worked in large corp environments). Anyway care to expand on some of the less Googleable acronyms? - MDM/MAM - NAC - EDW (synonymous with ETL?)

dataisfun 13y ago
Thanks for tip. These are now expanded in interview text :)
martinald 13y ago
MDM/MAM = mobile device management/mobile application management (managing and provisioning mobile devices and their applications, generally automatically) NAC = network access control EDW = enterprise data warehouse (archiving old information while preserving access)

obblekk 13y ago

Forgive my ignorance. Is `IT` the same as `engineering` at other companies, or is this something else?

justizin 13y ago
His actual title is "VP of IT Operations", which is usually part of an Engineering department, but very close to the executive team because of things like budget and business guarantees. Note the Unix / Networking background.
bri3d 13y ago
"IT" is too vaguely defined to mean much anymore, but based on this interview I suspect it's internal infrastructure at Netflix. Stuff like staff PCs, internal data warehouses, sales, finance, and marketing software support, WiFi APs, routers, keeping the backoffice servers and network up, ensuring reliable WAN and LAN connectivity so engineers can reach production securely, intrusion detection and analysis, and so on. Generally in smaller software companies I hear R+D and consumer-facing applications referred to as "engineering" with external-facing infrastructure (like the production datacenter) referred to as "operations," with "IT" being reserved for this internal backoffice kind of stuff. In other places, especially larger corporations, I've often heard everything having to do with a computer lumped in as "IT."
- xivzgrev 13y ago
  Yea I was surprised. When I saw the title I thought infrastructure meant network infrastructure, which seems like a white-hot area of innovation given how much internet traffic they consume. But no, this interview was on internal IT.
dpritchett 13y ago
IT is usually the catchall infrastructure/support function. For some companies (like Netflix) you probably also have a product engineering group.
puppetmaster3 13y ago
It's office automation. As opposed to Netops, that operates their business such as video streaming.

zobzu 13y ago

Reading the slides make me think this is full of nothing :| How is 802.11ac speed making things "more cloud"? Because you get slightly more bandwidth -maybe- if you have a new laptop and also you dont have everyone using it? I don't get it. Requiring VPN everywhere, how is that cloudy? Finally, using stuff like AWS is nice, but unless they have a specific contract (which they may since they advertise them a lot), its a LOT more expensive when you start having a lot of processing (ie big companies like netflix)

cwp 13y ago
Well, if anybody could get a good deal from AWS, it's Netflix.

purephase 13y ago

I thought the 2014 Technology Roadmap [1] was an interesting read. For an organization as "young" as Netflix, I was surprised by the technology debts that they've accumulated and the aggressive tone that they've set to transition. I think it's amazing the decisions that get made with explosive growth/hiring that end-up on roadmaps that read similarly to organizations that have been around much longer. There's no criticism here. I think Netflix is an amazing company and it is the this sort of strategic vision (and the openess of both it and the organization overall) that reminds me that we're all on this rocky ship together and it's amazing that any of it works sometimes. [1] http://www.slideshare.net/mdkail/it-ops-2014-technology-roadmap http://www.slideshare.net/mdkail/it-ops-2014-technology-road...

teacup50 13y ago

> The notion that something needs to remain on-premise is really an Old World way of thinking and feels more like someone wanting control as opposed to there being a valid argument. No, it's the business continuity way of thinking. Outsourcing commodities -- such as servers, virtual or otherwise -- is one thing. Outsourcing your core operational tools, software, and all your data is another matter entirely. Preferring SaaS at a company large enough to afford on-premise solutions is just nonsensical, and I expect it'll either blow up in his face, or just create a never-ending tax on end users who are constantly dealing with a mishmash of vendors, accounts, disappearing services, broken software, and instability. At scale, stability and continuity is worth more than the opex/capex costs of internal IT.

couradical 13y ago
I can see both sides of the coin though - for a company that is as heavily cloud-invested as Netflix is - it might make less sense to maintain a cage for internal IT in a few datacenters. That said, I do get worried about sensitive data in the cloud. I didn't get that he was talking about moving to cloud/SaaS vendors as a whole, but more the IaaS/PaaS space - the "Hey, this app runs RoR, can I just run it from a Heroku dyno rather than a VMWare box in a cage" type of move. Granted, there's still a cost associated with that, and a need to plan the move, but it's less of an issue. They are making heavy use of SAML SSO it appears, so the account/vendor thing probably wouldn't be as much of an issue.
binarytrees 13y ago
You have a point, but having services on-premise can still achieve the disfunction, mishmash and overall goods times you're describing with a SaaS service. You might get better response times knowing it's a local problem vs let's say an office in NYC hosting your ticketing system. Regardless, I'm sure they have put a lot of thought into this set up and to me sounds amazing. No ties to a physical location = win.
e12e 13y ago
> Preferring SaaS at a company large enough to afford on-premise solutions is just nonsensical Yes and no. First off, I don't think Netflix can afford to build the kind of infrastructure they're using -- certainly not without changing focus of their engineering resources. They famously do a lot of work to move data closer to the end users globally -- they're not "just" a US company. Essentially, they'd have to have an operation that would be "qualitatively" similar to AWS in order to be able to do what they do (and at a smaller scale overall, I think that would end up being quite expensive). I'd argue Netflix is one of the few companies I can think of where this "all cloud all the time" idea for infrastructure might actually make sense. I agree it's a big risk though -- and probably not good advice for most companies. It would also appear that Netflix is planning on actually selling a product (video rental) and make money off that, rather than have that merely as a vehicle to drive other, sometimes tangential, innovation. That remains to be seen, of course.

qthrul 13y ago

TLDR: The approach we take for IT works (for us at this point in time in the scope defined as IT by me and/or our internal customers). Netflix talks generally can be fascinating and inspiring. However, when considering IT it's also important to consider the charter and challenges of Netflix IT. i.e. it's no more valid or invalid that the talks of how IT is delivered in so-called build vs. broker models in other companies in other industries http://dilbert.com/strips/comic/2013-07-05/ http://dilbert.com/strips/comic/2013-07-05/

nessup 13y ago

Why was there no discussion of the ethics of the Comcast deal?

cdcarter 13y ago
Because this was a conversation about internal infrastructures?
wookiefeet 13y ago
Netflix IT would be an odd group to ask about the deal, probably wouldn't have made sense in this article

welder 13y ago

> zero-trust network architecture This makes me think of http://meldium.com http://meldium.com

e12e 13y ago

It'd be great if Netflix (or some other company) manages to do some heavy lifting in creating a viable, modern, certificate-based authentication and authorization stack, that's easier to deploy. Essentially an upgraded take on kerberos (move off shared secrets, perhaps), AFS (I still don't know what a viable way forward for secure, distributed, locally cacheable network filesystem is -- maybe DAV+TLS+regular caching?). I suppose LDAP might be fine as a user/principal/authorization database, but some distribution that uses internal CA and demands TLS as default would be a good start. The last "innovation" I'm aware of in this area, is skolelinux/edulinux work with packaging samba/ldap/kerberos/lts in a easy(ier) to manage package for Debian: https://wiki.debian.org/DebianEdu/Documentation/Wheezy/Architecture https://wiki.debian.org/DebianEdu/Documentation/Wheezy/Archi...

zobzu 13y ago
kerberos is very good. everyone reinvents kerberos every month. it doesn't have to be new to be "modern". Kerberos is still modern by today's standards, in fact. the problem is having tools that communicate with each others and an easy setup. yeah, SAML kinda sucks to use too.. and works like kerberos anyway. OpenID, Hawk, etc - also in fact work exactly the same.
- e12e 13y ago
  IIRC there are a couple of things that might be tweaked in the protocol wrt sign before encrypt? Also, with non-encumbered public key cryptography available, and no longer prohibitively resource intensive, we can do better than NxN shared secrets. I do indeed like kerberos - but I still want a straight forward and reasonably robust framework built on certificates.
- eropple 13y ago
  Seriously. At my day job we're investigating better ways to handle single sign on and authentication and every road leads back to Kerberos.
  - e12e 13y ago

vespaceballs6 13y ago

I really wanted to share this article with my friends, but it was so filled with buzzwords that even my dev friends wouldn't ascertain much. Keep in mind I'm an idiot, and I have idiot friends.

dataisfun 13y ago
haha. Well, I doubt you're an idiot. Buzzwords serve a purpose insofar as they're a good shorthand for defining categories of product offerings. E.g., ETL, Data Warehouse, Mobile Device Management, etc. all have pretty well understood parameters within which the vendors, buyers, analysts, etc. operate.

cottonseed 13y ago

One thing that stuck out to me: > We are implementing “certificate-based authentication” instead of the standard username/password auth against Active Directory. I wish we were all doing this. How long is it going to take to get a usable certificate-based client/user authentication mechanism on the web? edit: Also see e12e's comment.