7 ms·

The Rise of Platform Engineering

154 points by godelmachine 6y ago

sam_lowry_ 6y ago
I had a really bad experience in an organisation possessed by a platform team. A small number of individuals were rolling change after change that impacted hundreds of engineers, halving their productivity and halting all development to a grind once a month.
- botto 6y ago
  I'm sorry to hear that as someone who has done this role in the past I can tell you that it's the opposite of what a platform team should be doing. When done right a platform team should basically not be noticed except that the tooling, developer experience and overall reliability of a system goes up.
  - sam_lowry_ 6y ago
    «When done right a platform team should basically not be noticed except that the tooling, developer experience and overall reliability of a system goes up.» In memories!
- geofft 6y ago
  Did the platform team use their own platform? I have a theory that that's the right way to set up incentives, but I'm curious whether it works in practice.
- fizx 6y ago
  You know how we have that "libraries are better than frameworks" discussion once a month? It sounds like your platform team was a framework. Ideally, a platform team should give you reliable, self-service components to build upon, like databases, caches, rate-limiters, api-gateways, etc. If they're framework-ish, they're mandating you build your app in exactly their preferred way in order to receive the benefits.
  - fuckyah 6y ago
    I’m in a large Corp that does the latter. It’s an absolute disgrace every single day. I’m sick and tired of it, but stuck with golden handcuffs... Can you recommend a public company that follows the former approach?
  - k__ 6y ago
    ideally 99% of all companies shouldn't even have a platform team, but buy a platform from professionals.
    - user5994461 6y ago
      99% of companies should have a platform team, whose role is to make services from AWS/Google/Azure available internally and easy to use.
      - k__ 6y ago
        Yes, or that.
- SanderKnape 6y ago
  This is a typical challenge for platform teams. Because they are responsible for the underlying foundation on which applications are build, a mistake can easily impact all those applications. An important realization in my opinion is that a platform team is just another development team. They should consider their platform as a product and the developers as their customers. To minimize any downtime they should use the typical mechanisms that developers are also using: automated tests, pair programming, rolling out changes to test environments first, etc.
sl1ck731 6y ago
The author describes Terraform and CloudFormation as "imperative". This doesn't seem correct to me, although you can force a sort of imperative flow by manually defining your dependencies in a specific order. I have only a little experience with Ansible but I would say that is the only major imperative-ish IaC (at least the way I used it) aside from bash scripts or working with SDK's directly.
- cbsmith 6y ago
  Yeah, that's the wrong distinction between Terraform/CloudFormation and Kubernetes. Terraform/CloudFormation try to be declarative. The distinction is more that Terraform/CloudFormation are about provisioning infrastructure.
- halbritt 6y ago
  Terraform in and of itself is declarative, but it behaves in an imperative sort of way with the various backends that it supports. These shortcomings all manifest themselves in how state is managed. Terraform state is declaratively described, and it may or may not match the state of the backend. Once this state drift exists, it becomes difficult to correct. This is my primary criticism of Terraform and one of the reasons I prefer Kubernetes. I know it's an apple to orange comparison, but in Kubernetes there is both declarative configuration and active reconciliation. You have both current state and desired state and a set of controllers seeking to make them match. I'd love to see this implemented with Terraform.
  - mwarkentin 6y ago
    Terraform attempts to refresh its state from the source of truth (eg aws apis) before planning. It’s not always possible, but often it should work just fine even if you’ve modified a resource outside of terraform.
  - borplk 6y ago
    Terraform mostly can make the current state match the desired state but the challenge is the real world side effects such as the state that exists in the database that is about to get destroyed or down-time of services that depend on the resources being managed and so on. So you can't blindly allow it to do what it wants.
dutch3000 6y ago
Platform engineering = Site reliability engineering? That’s how I see it.
- halbritt 6y ago
  People view SRE in many difference ways. If you were to go through the Google SRE book, there's nothing explicit in there about building "platform" or doing any infrastructure engineering. It's is a given that SRE build tooling, but most of the SRE-focused work, as described in that book is around improving the resilience of a given application or product. Addressing production readiness, defining SLOs, and handling incident response. There are platform specific SRE teams at Google, but there's not much published about how they get about creating platform. The book "Seeking SRE" makes it clear that in most places, the notion of "platform engineering" varies tremendously. I don't know of any authors that have addressed this explicitly other than Susan Fowler in "Production Ready Microservices" who writes: Another important part of microservice adoption is the creation of a microservice ecosystem. Typically (or, at least, hopefully), a company running a large monolithic application will have a dedicated infrastructure organization that is responsible for designing, building, and maintaining the infrastructure that the application runs on. When a monolith is split into microservices, the responsibilities of the infrastructure organization for providing a stable platform for microservices to be developed and run on grows drastically in importance. The infrastructure teams must provide microservice teams with stable infrastructure that abstracts away the majority of the complexity of the interactions between microservices.
  - SanderKnape 6y ago
    The book Team Topologies discusses Platform Engineering. They consider four core team types, and the platform team is one of those. Some more information about those types can be found here: https://teamtopologies.com/key-concepts-content/what-are-the-core-team-types-in-team-topologies https://teamtopologies.com/key-concepts-content/what-are-the... I can definitely recommend this book for a (new) perspective on how platform teams fit into organizations.
spaetzleesser 6y ago
I sort of work on platform engineering in the form of a test automation framework used by more than a hundred testers. In general it works fine but you have to be ready to scale up your platform team at the same speed or even higher than the users of the platform. You also have to be very careful allowing the users to expand the platform to their needs or the platform team will be a permanent bottleneck. I have seen this in companies where the SAP people had a backlog of several years.
- andendau 6y ago
  There's a good blog on scaling a platform team and how to keep up with growth. https://medium.com/adobetech/why-do-organizations-need-a-platform-team-910d79893e0a https://medium.com/adobetech/why-do-organizations-need-a-pla...
aogl 6y ago
I actually wrote about this months ago at https://ao.gl/what-it-takes-to-be-a-platform-engineer-in-2020/ https://ao.gl/what-it-takes-to-be-a-platform-engineer-in-202... Interestingly also had quite a bit of traction on HN, which makes me think many are moving in this direction..
- user5994461 6y ago
  +1 on this. Started to get contacted on linkedin and finding job postings for platform engineering last year. I have a strong feeling this could be the next buzzword of the 2020 decade.
staysaasy 6y ago
IMO one of the keys to a platform engineering group's success (alluded to in this post) is having a mindset in which other engineering teams are the customer. Once you flip into this mode it becomes a lot more clear how your platform is really a product, and the platform engineering team fits much more seamlessly into the overall organization.
- andendau 6y ago
  yes good product management is key even though your customers may not have a choice
  - staysaasy 6y ago
    Yup, for sure. Having PMs who understand infrastructure and distributed systems can be a secret weapon for high-scale businesses, especially in enterprise SaaS where competition is more directly head-to-head.
- pawsk 6y ago
  I think this is a good idea. You should treat the consumers of your services as customers, regardless of whether or not it's an actual end-user.
  - staysaasy 6y ago
    Exactly. Once you view other teams as your customers, it helps you focus on things like quality, satisfaction, and adding value in a very clarifying way. And honestly other eng teams kind of are your customers. They may not pay you directly, but they also help to build your product and the company that cuts your (and everyone's) paycheck.
    - pawsk 6y ago
      Regarding treating the other eng. teams as customers, I fully agree. I know it's not something everyone does (Been there, argued that), but when you treat them that way and when you deal with their problems (Service xyz requests are too slow, feature xyz would really solve our problems ...) you will, in the end, help the customers. If my coworkers need my services, then that's because they are developing something that a customer needs. I think of it like a dependency-tree. As long as you trust your company to not have multiple teams develop something that will never see the light of day, or something the user doesn't want or need, then this mindset is absolutely a good one (Sadly been there, done that too).
abhayb 6y ago
All infra teams eventually become platforms. All product teams eventually become experiences. When viewed negatively this is called scope creep. I don't know what it's called when viewed positively but I expect the word "holistic" to be used unironically. Org charts that ship a platform are default stable because everybody it a team or group is doing approximately the same things. Growth is less uncomfortable, advancement feels more objective, and individual developers are relatively interchangeable. But what if a company needs to change? Now the stable org chart resists that change. By rejecting requests from client teams that are responsible for a new set of objectives. This recurses. One layer of platform can simultaneously be moving too slowly for the layer above and too quickly for the one below. Shear forces tear it apart and the organization finds itself with n (3 < n < 6) fewer platform engineers.
- user5994461 6y ago
  In my experience having a formal platform structure helps tremendously to roll new solutions. When the platform officially provides something, everybody in the company can make use of it right away. Unlike the pet project that's being pushed by a random manager. Breakthrough can be pushed by rolling a second library/framework/platform. Like AWS ELB and ALB. Then developers can adopt the later if it's so much greater, but they won't because it's 90% of the same and who wants to work on migrations? Large organizations are fundamentally split apart. First part of the org wants A and B. Second part wants B and C. The developer team next floor is rolling their own thing to do C and D. All while features A and C are incompatible so it's impossible to satisfy everyone. There is no solution to resolve internal conflicts (except maybe reducing a large company to 20% of its current workforce).
mr_tristan 6y ago
My mind went immediately to a Netflix presentation about "the role of central teams in devops" a couple of years ago: https://www.youtube.com/watch?v=xA1qPQg3WTc https://www.youtube.com/watch?v=xA1qPQg3WTc The big thing I remember about their approach that surprised everyone: there was no mandate to use the platform/central team's tools. It made me chuckle how many times the presenter was grilled by everyone about that. It was like some of the audience straight up thought he was lying about that. But basically, if you have a platform team, and a mandate to use that team's tools, well, the other teams aren't really "customers", in the sense you can leverage choice as a signal. So you have to make up for that. In my experience, you need very good management, and constant, multifaceted communication. Which... might work, might not. It's probably best to delay any kind of centralized/platform work until you have a _very_ clear pain that defines a very clear set of roles and requirements. Unless everyone says "Oh shit this is amazing" ... just say no.
- closeparen 6y ago
  Mandate makes more sense the lower in the stack you go. “This is the cluster scheduler you must use to run code in the datacenters” Ok. “Since part of your feature communicates with end users, it must be implemented in this visual programming language we created for workflows that interact with end users.” Not fine. This is how you end up with abominations of workarounds on workarounds. Let me use my regular tools and give me a damn API. Your decision to invent a shitty half baked programming environment is not my problem. When you try to make it my problem you are only creating more damage.
  - user5994461 6y ago
    It doesn't make sense at the datacenter level either. How are ops/developers supposed to use ansible or salt or docker or kubernetes, when the only available solution to access/deploy to the servers is with the one centrally approved tool.

154 points by godelmachine 6y ago

I had a really bad experience in an organisation possessed by a platform team. A small number of individuals were rolling change after change that impacted hundreds of engineers, halving their productivity and halting all development to a grind once a month.

botto 6y ago
I'm sorry to hear that as someone who has done this role in the past I can tell you that it's the opposite of what a platform team should be doing. When done right a platform team should basically not be noticed except that the tooling, developer experience and overall reliability of a system goes up.
- sam_lowry_ 6y ago
  «When done right a platform team should basically not be noticed except that the tooling, developer experience and overall reliability of a system goes up.» In memories!
geofft 6y ago
Did the platform team use their own platform? I have a theory that that's the right way to set up incentives, but I'm curious whether it works in practice.
fizx 6y ago
You know how we have that "libraries are better than frameworks" discussion once a month? It sounds like your platform team was a framework. Ideally, a platform team should give you reliable, self-service components to build upon, like databases, caches, rate-limiters, api-gateways, etc. If they're framework-ish, they're mandating you build your app in exactly their preferred way in order to receive the benefits.
- fuckyah 6y ago
  I’m in a large Corp that does the latter. It’s an absolute disgrace every single day. I’m sick and tired of it, but stuck with golden handcuffs... Can you recommend a public company that follows the former approach?
- k__ 6y ago
  ideally 99% of all companies shouldn't even have a platform team, but buy a platform from professionals.
  - user5994461 6y ago
    99% of companies should have a platform team, whose role is to make services from AWS/Google/Azure available internally and easy to use.
    - k__ 6y ago
      Yes, or that.
SanderKnape 6y ago
This is a typical challenge for platform teams. Because they are responsible for the underlying foundation on which applications are build, a mistake can easily impact all those applications. An important realization in my opinion is that a platform team is just another development team. They should consider their platform as a product and the developers as their customers. To minimize any downtime they should use the typical mechanisms that developers are also using: automated tests, pair programming, rolling out changes to test environments first, etc.

sl1ck731 6y ago

The author describes Terraform and CloudFormation as "imperative". This doesn't seem correct to me, although you can force a sort of imperative flow by manually defining your dependencies in a specific order. I have only a little experience with Ansible but I would say that is the only major imperative-ish IaC (at least the way I used it) aside from bash scripts or working with SDK's directly.

cbsmith 6y ago
Yeah, that's the wrong distinction between Terraform/CloudFormation and Kubernetes. Terraform/CloudFormation try to be declarative. The distinction is more that Terraform/CloudFormation are about provisioning infrastructure.
halbritt 6y ago
Terraform in and of itself is declarative, but it behaves in an imperative sort of way with the various backends that it supports. These shortcomings all manifest themselves in how state is managed. Terraform state is declaratively described, and it may or may not match the state of the backend. Once this state drift exists, it becomes difficult to correct. This is my primary criticism of Terraform and one of the reasons I prefer Kubernetes. I know it's an apple to orange comparison, but in Kubernetes there is both declarative configuration and active reconciliation. You have both current state and desired state and a set of controllers seeking to make them match. I'd love to see this implemented with Terraform.
- mwarkentin 6y ago
  Terraform attempts to refresh its state from the source of truth (eg aws apis) before planning. It’s not always possible, but often it should work just fine even if you’ve modified a resource outside of terraform.
- borplk 6y ago
  Terraform mostly can make the current state match the desired state but the challenge is the real world side effects such as the state that exists in the database that is about to get destroyed or down-time of services that depend on the resources being managed and so on. So you can't blindly allow it to do what it wants.

dutch3000 6y ago

Platform engineering = Site reliability engineering? That’s how I see it.

halbritt 6y ago
People view SRE in many difference ways. If you were to go through the Google SRE book, there's nothing explicit in there about building "platform" or doing any infrastructure engineering. It's is a given that SRE build tooling, but most of the SRE-focused work, as described in that book is around improving the resilience of a given application or product. Addressing production readiness, defining SLOs, and handling incident response. There are platform specific SRE teams at Google, but there's not much published about how they get about creating platform. The book "Seeking SRE" makes it clear that in most places, the notion of "platform engineering" varies tremendously. I don't know of any authors that have addressed this explicitly other than Susan Fowler in "Production Ready Microservices" who writes: Another important part of microservice adoption is the creation of a microservice ecosystem. Typically (or, at least, hopefully), a company running a large monolithic application will have a dedicated infrastructure organization that is responsible for designing, building, and maintaining the infrastructure that the application runs on. When a monolith is split into microservices, the responsibilities of the infrastructure organization for providing a stable platform for microservices to be developed and run on grows drastically in importance. The infrastructure teams must provide microservice teams with stable infrastructure that abstracts away the majority of the complexity of the interactions between microservices.
- SanderKnape 6y ago
  The book Team Topologies discusses Platform Engineering. They consider four core team types, and the platform team is one of those. Some more information about those types can be found here: https://teamtopologies.com/key-concepts-content/what-are-the-core-team-types-in-team-topologies https://teamtopologies.com/key-concepts-content/what-are-the... I can definitely recommend this book for a (new) perspective on how platform teams fit into organizations.

spaetzleesser 6y ago

I sort of work on platform engineering in the form of a test automation framework used by more than a hundred testers. In general it works fine but you have to be ready to scale up your platform team at the same speed or even higher than the users of the platform. You also have to be very careful allowing the users to expand the platform to their needs or the platform team will be a permanent bottleneck. I have seen this in companies where the SAP people had a backlog of several years.

andendau 6y ago
There's a good blog on scaling a platform team and how to keep up with growth. https://medium.com/adobetech/why-do-organizations-need-a-platform-team-910d79893e0a https://medium.com/adobetech/why-do-organizations-need-a-pla...

aogl 6y ago

I actually wrote about this months ago at https://ao.gl/what-it-takes-to-be-a-platform-engineer-in-2020/ https://ao.gl/what-it-takes-to-be-a-platform-engineer-in-202... Interestingly also had quite a bit of traction on HN, which makes me think many are moving in this direction..

user5994461 6y ago
+1 on this. Started to get contacted on linkedin and finding job postings for platform engineering last year. I have a strong feeling this could be the next buzzword of the 2020 decade.

staysaasy 6y ago

IMO one of the keys to a platform engineering group's success (alluded to in this post) is having a mindset in which other engineering teams are the customer. Once you flip into this mode it becomes a lot more clear how your platform is really a product, and the platform engineering team fits much more seamlessly into the overall organization.

andendau 6y ago
yes good product management is key even though your customers may not have a choice
- staysaasy 6y ago
  Yup, for sure. Having PMs who understand infrastructure and distributed systems can be a secret weapon for high-scale businesses, especially in enterprise SaaS where competition is more directly head-to-head.
pawsk 6y ago
I think this is a good idea. You should treat the consumers of your services as customers, regardless of whether or not it's an actual end-user.
- staysaasy 6y ago
  Exactly. Once you view other teams as your customers, it helps you focus on things like quality, satisfaction, and adding value in a very clarifying way. And honestly other eng teams kind of are your customers. They may not pay you directly, but they also help to build your product and the company that cuts your (and everyone's) paycheck.
  - pawsk 6y ago
    Regarding treating the other eng. teams as customers, I fully agree. I know it's not something everyone does (Been there, argued that), but when you treat them that way and when you deal with their problems (Service xyz requests are too slow, feature xyz would really solve our problems ...) you will, in the end, help the customers. If my coworkers need my services, then that's because they are developing something that a customer needs. I think of it like a dependency-tree. As long as you trust your company to not have multiple teams develop something that will never see the light of day, or something the user doesn't want or need, then this mindset is absolutely a good one (Sadly been there, done that too).

abhayb 6y ago

All infra teams eventually become platforms. All product teams eventually become experiences. When viewed negatively this is called scope creep. I don't know what it's called when viewed positively but I expect the word "holistic" to be used unironically. Org charts that ship a platform are default stable because everybody it a team or group is doing approximately the same things. Growth is less uncomfortable, advancement feels more objective, and individual developers are relatively interchangeable. But what if a company needs to change? Now the stable org chart resists that change. By rejecting requests from client teams that are responsible for a new set of objectives. This recurses. One layer of platform can simultaneously be moving too slowly for the layer above and too quickly for the one below. Shear forces tear it apart and the organization finds itself with n (3 < n < 6) fewer platform engineers.

user5994461 6y ago
In my experience having a formal platform structure helps tremendously to roll new solutions. When the platform officially provides something, everybody in the company can make use of it right away. Unlike the pet project that's being pushed by a random manager. Breakthrough can be pushed by rolling a second library/framework/platform. Like AWS ELB and ALB. Then developers can adopt the later if it's so much greater, but they won't because it's 90% of the same and who wants to work on migrations? Large organizations are fundamentally split apart. First part of the org wants A and B. Second part wants B and C. The developer team next floor is rolling their own thing to do C and D. All while features A and C are incompatible so it's impossible to satisfy everyone. There is no solution to resolve internal conflicts (except maybe reducing a large company to 20% of its current workforce).

mr_tristan 6y ago

My mind went immediately to a Netflix presentation about "the role of central teams in devops" a couple of years ago: https://www.youtube.com/watch?v=xA1qPQg3WTc https://www.youtube.com/watch?v=xA1qPQg3WTc The big thing I remember about their approach that surprised everyone: there was no mandate to use the platform/central team's tools. It made me chuckle how many times the presenter was grilled by everyone about that. It was like some of the audience straight up thought he was lying about that. But basically, if you have a platform team, and a mandate to use that team's tools, well, the other teams aren't really "customers", in the sense you can leverage choice as a signal. So you have to make up for that. In my experience, you need very good management, and constant, multifaceted communication. Which... might work, might not. It's probably best to delay any kind of centralized/platform work until you have a _very_ clear pain that defines a very clear set of roles and requirements. Unless everyone says "Oh shit this is amazing" ... just say no.

closeparen 6y ago
Mandate makes more sense the lower in the stack you go. “This is the cluster scheduler you must use to run code in the datacenters” Ok. “Since part of your feature communicates with end users, it must be implemented in this visual programming language we created for workflows that interact with end users.” Not fine. This is how you end up with abominations of workarounds on workarounds. Let me use my regular tools and give me a damn API. Your decision to invent a shitty half baked programming environment is not my problem. When you try to make it my problem you are only creating more damage.
- user5994461 6y ago
  It doesn't make sense at the datacenter level either. How are ops/developers supposed to use ansible or salt or docker or kubernetes, when the only available solution to access/deploy to the servers is with the one centrally approved tool.