4 ms·

Six Sins of Platform Teams

58 points by kaycebasques 2y ago

com 2y ago
The “internal customer” thing has been the worst driver of waste and frustration I’ve seen in my career. It’s like the MBAs looked at the Soviet Union’s fake markets, and thought to themselves “we want some of that, please”.
bvrmn 2y ago
Platform teams could be highly toxic especially in small startups during initial secops/audit era. You really need like hundreds of product engineers to gain velocity/value from platform teams.
- tuananh 2y ago
  that's when they dont see internal teams are customers. because if they do, they will do everything to make the internal teams life easier. enabling, not gatekeeping.
- SerCe 2y ago
  In my mind, platform teams wouldn't necessarily be toxic in a small-ish companies, it's just very unlikely they'd qualify to be a platform team. In the absence of cross-cutting logic to take care of, it sounds like they'd effectively be a product team focusing on the infra things?
  - master_crab 2y ago
    That’s generally how they start. Most companies don’t really start with a “platform” team. It’s usually a guy who manages the infrastructure, or is the go-to for your IaC questions.
    - SerCe 2y ago
      That's true, but they don't always have to evolve to follow the platform team model. They could evolve into more of an SRE-style team and/or traditional ops team.
  - bvrmn 2y ago
    Let me elaborate. The idea of "we have multiple product teams and they do similar stuff, let's optimize it" visits CTO/CEO brain on quite early stages. And "platform" team could evolve according to: 1) devops/sre start to provide some guides on top of some cloud, nowdays it's k8s. Like default service templates. 2) service templates transforms into custom DSL with side configuration and k8s abstraction things. 3) Abstraction/libraries on top of secrets management. 4) Service configuration per enviroment. At with point it's all good. But startup grows, and needs a secops team to get some internal audit. Or it could be a platform team initiative. 5) Audit shows critical issues with permissions and platform team starts to think about how to restrict access. Mismatch with "old freedom" could be quite high for unprepared product teams. Platform becomes "inconvenient". It takes huge amount of resources to make it actually usable.
    - SerCe 2y ago
      I see, thank you for clarifying. That's an interesting observation, I feel like there is a story behind it. :) I definitely agree that as any small start-up grows, the security controls become very painful, especially for those folks who felt "the freedom" before the controls were established. That said, would the picture look better without platform teams? The security controls would need to be there anyway, and I'd personally prefer to use some self-serve, platform-ish solution built by a team of software engineers that would do call auditing, verification, etc., rather than raising a JIRA ticket with some ops folks who'd do the security-sensitive thing for you.
      - bvrmn 2y ago
        > I feel like there is a story behind it. :) Ahah. 100%. Honestly I am in a great conflict right now. My current role is in a platform team (first time here, previously it was only product positions). And I'm responsible for implementing unified company wide authorization including client libraries and integration with all web frameworks in use. I feel vast empathy for product engineers having to migrate and integrate the new system. It has artificial intended bumps I have to enforce and doesn't like personally.
jillesvangurp 2y ago
I worked in Nokia Research and Nokia Maps around the time that Apple started kicking their ass. Platforms became a dirty word in that company. Equating that to devops is a mistake. Any kind of platform is a problem. What's a platform: anything that only serves internal customers that is treated like a product but just isn't. Nokia had lots of platforms. Several Linux based operating systems (meego/maemo, meltemi, and briefly Android even), several flavors of Symbian (S60, S80, S90), and several flavors of non symbian legacy proprietary platforms (S30 and S40). And that's just operating systems. And that was before MS got involved and added two flavors of Windows Phone with completely different kernels. Nokia also had UI platforms (multiple). And it dabbled with online/cloud platforms as well (OVI, if you remember that). And indeed devops and operations platforms. I witnessed what the introduction of AWS did to those. It was hilarious. First Nokia IT was outraged that people dared to use something from Amazon, then shocked it was actually cheaper and better, and then the layoffs started and that team was one of the first things that got impacted for the obvious reason that the AWS deployment was a raging success and they tried blocking that for several years at great cost to the business. Once they were off the critical path to deploying stuff, that whole unit became redundant. They had forgotten about end users and they were pissing off their internal customers to the point that those started engineering around them. That's the problem with internal platforms: they compete with external ones and if they aren't good enough, they become a problem. You get people digging in, add internal politics to the mix, and now you have epic amounts of money being wasted. I think Google and Facebook are good examples of becoming serial offenders on being too absorbed by their own internal platform teams. I recognize a lot of Nokia in modern day Google. Lots of headless chickens in their leadership, endless waffle about internal platforms that then flop (fuchsia being a prime example), endless reinvention of wheels (UI frameworks, compilers, operating systems, chat protocols, social networking, IOT, LLM/AI, etc.) and a long list of stuff that they discontinue because it just didn't work, and way too many variants of just about anything they do.
- SerCe 2y ago
  That's a very interesting story, can't find those in places other than HN, thanks for sharing. Out of interest, what was the reason they didn't start rebuilding their stack on top of AWS? If it was cheaper and better, then it would make sense for them to effectively migrate all teams to their AWS-plus solution? Was there some anti-cloud sentiment?
fifilura 2y ago
My experience with platform teams is, once they are formed, they suck up the best people. Doing k8s/technical stuff. People that should have been working on building products.
- zactato 2y ago
  If those people were unhappy in their previous product engineering roles wouldn't they have left he company?
  - fifilura 2y ago
    It is not black and white. In my experience, what people want is heavily influenced by the company culture. If building platforms is cool then everyone wants to be there. But if building products is the thing then people want to work with this.
- anonzzzies 2y ago
  Many technical people find platforms far more interesting than trying to convert an excel sheet to a user viable product. We mixed up platform with product again after a few years as platform not only got the best people but also started to slow down the product team by taking decisions they saw as better (objectively they were) which the product team didn't agree andor didn't have experience with.
  - fifilura 2y ago
    You describe the toxicity I have experienced working in the product team vs a platform team. If you don't find it interesting to build products with whatever tool is most adequate (excel some times) maybe you should not work at a company building products. At least you should understand that you are part of the support team not the main guy. And yes, moving to GitHub from Jenkins is maybe objectively a good choice. But if it stalls product development for 6 months, is that objectively-objectively a smart move? It could of course be, but it requires more situational awareness than "is this tool better".

58 points by kaycebasques 2y ago

com 2y ago

The “internal customer” thing has been the worst driver of waste and frustration I’ve seen in my career. It’s like the MBAs looked at the Soviet Union’s fake markets, and thought to themselves “we want some of that, please”.

bvrmn 2y ago

Platform teams could be highly toxic especially in small startups during initial secops/audit era. You really need like hundreds of product engineers to gain velocity/value from platform teams.

tuananh 2y ago
that's when they dont see internal teams are customers. because if they do, they will do everything to make the internal teams life easier. enabling, not gatekeeping.
SerCe 2y ago
In my mind, platform teams wouldn't necessarily be toxic in a small-ish companies, it's just very unlikely they'd qualify to be a platform team. In the absence of cross-cutting logic to take care of, it sounds like they'd effectively be a product team focusing on the infra things?
- master_crab 2y ago
  That’s generally how they start. Most companies don’t really start with a “platform” team. It’s usually a guy who manages the infrastructure, or is the go-to for your IaC questions.
  - SerCe 2y ago
    That's true, but they don't always have to evolve to follow the platform team model. They could evolve into more of an SRE-style team and/or traditional ops team.
- bvrmn 2y ago
  Let me elaborate. The idea of "we have multiple product teams and they do similar stuff, let's optimize it" visits CTO/CEO brain on quite early stages. And "platform" team could evolve according to: 1) devops/sre start to provide some guides on top of some cloud, nowdays it's k8s. Like default service templates. 2) service templates transforms into custom DSL with side configuration and k8s abstraction things. 3) Abstraction/libraries on top of secrets management. 4) Service configuration per enviroment. At with point it's all good. But startup grows, and needs a secops team to get some internal audit. Or it could be a platform team initiative. 5) Audit shows critical issues with permissions and platform team starts to think about how to restrict access. Mismatch with "old freedom" could be quite high for unprepared product teams. Platform becomes "inconvenient". It takes huge amount of resources to make it actually usable.
  - SerCe 2y ago
    I see, thank you for clarifying. That's an interesting observation, I feel like there is a story behind it. :) I definitely agree that as any small start-up grows, the security controls become very painful, especially for those folks who felt "the freedom" before the controls were established. That said, would the picture look better without platform teams? The security controls would need to be there anyway, and I'd personally prefer to use some self-serve, platform-ish solution built by a team of software engineers that would do call auditing, verification, etc., rather than raising a JIRA ticket with some ops folks who'd do the security-sensitive thing for you.
    - bvrmn 2y ago
      > I feel like there is a story behind it. :) Ahah. 100%. Honestly I am in a great conflict right now. My current role is in a platform team (first time here, previously it was only product positions). And I'm responsible for implementing unified company wide authorization including client libraries and integration with all web frameworks in use. I feel vast empathy for product engineers having to migrate and integrate the new system. It has artificial intended bumps I have to enforce and doesn't like personally.

jillesvangurp 2y ago

I worked in Nokia Research and Nokia Maps around the time that Apple started kicking their ass. Platforms became a dirty word in that company. Equating that to devops is a mistake. Any kind of platform is a problem. What's a platform: anything that only serves internal customers that is treated like a product but just isn't. Nokia had lots of platforms. Several Linux based operating systems (meego/maemo, meltemi, and briefly Android even), several flavors of Symbian (S60, S80, S90), and several flavors of non symbian legacy proprietary platforms (S30 and S40). And that's just operating systems. And that was before MS got involved and added two flavors of Windows Phone with completely different kernels. Nokia also had UI platforms (multiple). And it dabbled with online/cloud platforms as well (OVI, if you remember that). And indeed devops and operations platforms. I witnessed what the introduction of AWS did to those. It was hilarious. First Nokia IT was outraged that people dared to use something from Amazon, then shocked it was actually cheaper and better, and then the layoffs started and that team was one of the first things that got impacted for the obvious reason that the AWS deployment was a raging success and they tried blocking that for several years at great cost to the business. Once they were off the critical path to deploying stuff, that whole unit became redundant. They had forgotten about end users and they were pissing off their internal customers to the point that those started engineering around them. That's the problem with internal platforms: they compete with external ones and if they aren't good enough, they become a problem. You get people digging in, add internal politics to the mix, and now you have epic amounts of money being wasted. I think Google and Facebook are good examples of becoming serial offenders on being too absorbed by their own internal platform teams. I recognize a lot of Nokia in modern day Google. Lots of headless chickens in their leadership, endless waffle about internal platforms that then flop (fuchsia being a prime example), endless reinvention of wheels (UI frameworks, compilers, operating systems, chat protocols, social networking, IOT, LLM/AI, etc.) and a long list of stuff that they discontinue because it just didn't work, and way too many variants of just about anything they do.

SerCe 2y ago
That's a very interesting story, can't find those in places other than HN, thanks for sharing. Out of interest, what was the reason they didn't start rebuilding their stack on top of AWS? If it was cheaper and better, then it would make sense for them to effectively migrate all teams to their AWS-plus solution? Was there some anti-cloud sentiment?

fifilura 2y ago

My experience with platform teams is, once they are formed, they suck up the best people. Doing k8s/technical stuff. People that should have been working on building products.

zactato 2y ago
If those people were unhappy in their previous product engineering roles wouldn't they have left he company?
- fifilura 2y ago
  It is not black and white. In my experience, what people want is heavily influenced by the company culture. If building platforms is cool then everyone wants to be there. But if building products is the thing then people want to work with this.
anonzzzies 2y ago
Many technical people find platforms far more interesting than trying to convert an excel sheet to a user viable product. We mixed up platform with product again after a few years as platform not only got the best people but also started to slow down the product team by taking decisions they saw as better (objectively they were) which the product team didn't agree andor didn't have experience with.
- fifilura 2y ago
  You describe the toxicity I have experienced working in the product team vs a platform team. If you don't find it interesting to build products with whatever tool is most adequate (excel some times) maybe you should not work at a company building products. At least you should understand that you are part of the support team not the main guy. And yes, moving to GitHub from Jenkins is maybe objectively a good choice. But if it stalls product development for 6 months, is that objectively-objectively a smart move? It could of course be, but it requires more situational awareness than "is this tool better".