5 ms·

Why Everyone Builds Internal Kubernetes Platforms

73 points by thiry 6y ago

Just1689 6y ago
Anyone have a link where we can read this?
- deleted 6y ago
  [deleted]
- thiry 6y ago
  You should be able to read it here: https://itnext.io/why-everyone-builds-internal-kubernetes-platforms-284c2cf76226?source=friends_link&sk=03aee58977f09eb9738b0119468a84dd https://itnext.io/why-everyone-builds-internal-kubernetes-pl...
  - bityard 6y ago
    Looks to me like there's a whole lotta tracking going on there. We have GET parameters, a redirect to medium, then a redirect _back_ to itnext.io. Sorry, but I'm giving this a miss.
    - thiry 6y ago
      I didn't know what Medium does here but that is good to know, thanks. Next time, I will be more careful submitting the right link here.
YarickR2 6y ago
Because it works ? I did that for our business unit to fully leverage our internal network infrastructure (SDN) , so I could have pods as first-class citizens working along with components deployed on bare metal, without all that vxlan/calico fuzz ; and did some other enhancements, all in less than 1000 lines of Python and Go.
- slow_donkey 6y ago
  (because it's an ad) Although fairly well written content marketing.
tantalor 6y ago
Mods: this is redirecting to itnext.io
- thiry 6y ago
  Yes, sorry I posted it before it was accepted by the medium publication. Here is the link to the article on itnext: https://itnext.io/why-everyone-builds-internal-kubernetes-platforms-284c2cf76226 https://itnext.io/why-everyone-builds-internal-kubernetes-pl...
junon 6y ago
How come a medium.com link is redirecting me to some unaffiliated website?
- thiry 6y ago
  It is not an unaffiliated website but a medium publication. I accidentally posted the link here before the article was accepted by the publication.
  - junon 6y ago
    But I thought Medium was a self-contained site... Why is it redirecting to another site? I don't understand.
    - thiry 6y ago
      If you have a publication on Medium, you can apparently get a custom URL: https://help.medium.com/hc/en-us/articles/360036093373-Changing-the-URL-of-your-Medium-publication https://help.medium.com/hc/en-us/articles/360036093373-Chang...
esoterae 6y ago
This seems relatively well written, but I consider the cost analysis worse than a dead fish handshake. It's just plain facile to explain in one sentence that it took two years of development to bring an internal tool into service, and then in another suggest that it is cost effective without any kind of rational assignment of figures; as well without comparison of the same environment without the tool.
- thiry 6y ago
  I see your point. The problem with exact numbers is that they are hardly (publicly) available as on the one side, there is the cost of building and running the platform, and on the other, there is potentially lost productivity (which is generally hard to measure). So, for the 2-year development example, I could only rely on the KubeCon talk, and it just mentions costs briefly (saying that it was cheaper for dev teams to use the platform). Another problem is that it can be very different depending on your situation: If you are currently working with local clusters that are free to use, the direct cost benefits will be much lower than if you use individual clusters for each developer, which are quite expensive even for small teams. So, I just wanted to give a rough overview of what that drives cost and where you potentially could save money.
insomniacity 6y ago
I feel like this completely misses the security and compliance reasons - it may in fact be easier to build a Kubernetes platform internally than to do all the work necessary to safely host confidential/HIPAA/mission critical information in the public cloud. And of course some organisations do both - public cloud for new stuff which can be secure from day 1, and internal platforms for the keys to the kingdom or legacy stuff.
- xenophonf 6y ago
  Considering how many security controls my FISMA Moderate-classed SaaS inherits from my cloud vendor, I very much doubt it's easier to build any platform internally, if we're only considering regulatory compliance.
  - Terretta 6y ago
    There’s paperwork compliance, and then there’s security and risk controls performance validated compliance. It may be more difficult if “we signed a contrast, trust the compliance report” is not an acceptable answer for a particular risk management audit or regulator.
    - xenophonf 6y ago
      If we're in "we can't rely the 3PAO's assessment or the JAB's (or DISA's) review of this cloud vendor" territory, then we're probably dealing with workloads far more sensitive than FISMA High or Secret, in which case it absolutely makes sense to DIY. But most workloads aren't that special.
thinkingkong 6y ago
Having built a bunch of these, the article more or less skips past the hard stuff. When you build an internal PaaS you aren't building the k8s part, you're building and prescribing the entire developer workflow to the team. If you do it well then it makes self-service super easy but the opinions on how to do everything are baked in. For example, you get opinionated versions of a build pipeline, SPA hosting, deployment models, yaml templating (or some other config language), dependency trees, monitoring and alerting systems, identity, secrets management, etc. All of those things end up being baked into the system if it's going to be successful. It's a tall order and I'd love to read more about internal PaaS failures than successes; thats where the learning is going to be.
- deleted 6y ago
  [deleted]
- thiry 6y ago
  I agree that the main challenge is to get the developer workflows right and being prescrptive here often should not be a problem from the developers' perspective. And yes, it would be great to generally hear more about failures but naturally people are less willing to share these stories, I guess.
peterwwillis 6y ago
mostly it's Conway's Law
vp8989 6y ago
People who don't write code should not dictate what the dev experience in a company should be. Kludging up some YAML that you run directly in Prod from your laptop is not writing code, Im sorry to say. The problem is that SREs are tasked with solving these problems and they have horrible taste and sensibilities around dev experience because they are not devs. All these remote dev Kubernetes solutions sound good in theory, but suck royally in practice. Remote debugging just isn't as good and the idea that I have to be online to get any work done is such a step back that it's baffling that anyone could even propose this.
- longcommonname 6y ago
  Im an SRE. I started off my career as a dev. I write code, yaml, hcl every day. You sound incredibly bitter.
nojvek 6y ago
At first I thought Kubernetes was a massive overkill. However as I learn more about it, I’ve been a big fan of having a local minikube cluster for dev, jsonnet for template wrangling, and being able to ship to stage and prod the same built images. Once you get over the learning curve it makes a ton of sense why k8s is growing like wildfire.
longcommonname 6y ago
Most developers just want to work on tickets other non infra related tasks. For the majority of those contributors its really helpful to give them a happy path. This happy path also makes supporting much easier. This happy path comes out as a platform and series of tools on top of the k8s layer. I love working on infra automation, and I love enabling other teams. When I did nothing but dev work the feedback loop for when I helped someone could be months. As an SRE I help dozens of people every week with near real-time feedback.

73 points by thiry 6y ago

Just1689 6y ago

Anyone have a link where we can read this?

deleted 6y ago
[deleted]
thiry 6y ago
You should be able to read it here: https://itnext.io/why-everyone-builds-internal-kubernetes-platforms-284c2cf76226?source=friends_link&sk=03aee58977f09eb9738b0119468a84dd https://itnext.io/why-everyone-builds-internal-kubernetes-pl...
- bityard 6y ago
  Looks to me like there's a whole lotta tracking going on there. We have GET parameters, a redirect to medium, then a redirect _back_ to itnext.io. Sorry, but I'm giving this a miss.
  - thiry 6y ago
    I didn't know what Medium does here but that is good to know, thanks. Next time, I will be more careful submitting the right link here.

YarickR2 6y ago

Because it works ? I did that for our business unit to fully leverage our internal network infrastructure (SDN) , so I could have pods as first-class citizens working along with components deployed on bare metal, without all that vxlan/calico fuzz ; and did some other enhancements, all in less than 1000 lines of Python and Go.

slow_donkey 6y ago
(because it's an ad) Although fairly well written content marketing.

tantalor 6y ago

Mods: this is redirecting to itnext.io

thiry 6y ago
Yes, sorry I posted it before it was accepted by the medium publication. Here is the link to the article on itnext: https://itnext.io/why-everyone-builds-internal-kubernetes-platforms-284c2cf76226 https://itnext.io/why-everyone-builds-internal-kubernetes-pl...

junon 6y ago

How come a medium.com link is redirecting me to some unaffiliated website?

thiry 6y ago
It is not an unaffiliated website but a medium publication. I accidentally posted the link here before the article was accepted by the publication.
- junon 6y ago
  But I thought Medium was a self-contained site... Why is it redirecting to another site? I don't understand.
  - thiry 6y ago
    If you have a publication on Medium, you can apparently get a custom URL: https://help.medium.com/hc/en-us/articles/360036093373-Changing-the-URL-of-your-Medium-publication https://help.medium.com/hc/en-us/articles/360036093373-Chang...

esoterae 6y ago

This seems relatively well written, but I consider the cost analysis worse than a dead fish handshake. It's just plain facile to explain in one sentence that it took two years of development to bring an internal tool into service, and then in another suggest that it is cost effective without any kind of rational assignment of figures; as well without comparison of the same environment without the tool.

thiry 6y ago
I see your point. The problem with exact numbers is that they are hardly (publicly) available as on the one side, there is the cost of building and running the platform, and on the other, there is potentially lost productivity (which is generally hard to measure). So, for the 2-year development example, I could only rely on the KubeCon talk, and it just mentions costs briefly (saying that it was cheaper for dev teams to use the platform). Another problem is that it can be very different depending on your situation: If you are currently working with local clusters that are free to use, the direct cost benefits will be much lower than if you use individual clusters for each developer, which are quite expensive even for small teams. So, I just wanted to give a rough overview of what that drives cost and where you potentially could save money.

insomniacity 6y ago

I feel like this completely misses the security and compliance reasons - it may in fact be easier to build a Kubernetes platform internally than to do all the work necessary to safely host confidential/HIPAA/mission critical information in the public cloud. And of course some organisations do both - public cloud for new stuff which can be secure from day 1, and internal platforms for the keys to the kingdom or legacy stuff.

xenophonf 6y ago
Considering how many security controls my FISMA Moderate-classed SaaS inherits from my cloud vendor, I very much doubt it's easier to build any platform internally, if we're only considering regulatory compliance.
- Terretta 6y ago
  There’s paperwork compliance, and then there’s security and risk controls performance validated compliance. It may be more difficult if “we signed a contrast, trust the compliance report” is not an acceptable answer for a particular risk management audit or regulator.
  - xenophonf 6y ago
    If we're in "we can't rely the 3PAO's assessment or the JAB's (or DISA's) review of this cloud vendor" territory, then we're probably dealing with workloads far more sensitive than FISMA High or Secret, in which case it absolutely makes sense to DIY. But most workloads aren't that special.

thinkingkong 6y ago

Having built a bunch of these, the article more or less skips past the hard stuff. When you build an internal PaaS you aren't building the k8s part, you're building and prescribing the entire developer workflow to the team. If you do it well then it makes self-service super easy but the opinions on how to do everything are baked in. For example, you get opinionated versions of a build pipeline, SPA hosting, deployment models, yaml templating (or some other config language), dependency trees, monitoring and alerting systems, identity, secrets management, etc. All of those things end up being baked into the system if it's going to be successful. It's a tall order and I'd love to read more about internal PaaS failures than successes; thats where the learning is going to be.

deleted 6y ago
[deleted]
thiry 6y ago
I agree that the main challenge is to get the developer workflows right and being prescrptive here often should not be a problem from the developers' perspective. And yes, it would be great to generally hear more about failures but naturally people are less willing to share these stories, I guess.

peterwwillis 6y ago

mostly it's Conway's Law

vp8989 6y ago

People who don't write code should not dictate what the dev experience in a company should be. Kludging up some YAML that you run directly in Prod from your laptop is not writing code, Im sorry to say. The problem is that SREs are tasked with solving these problems and they have horrible taste and sensibilities around dev experience because they are not devs. All these remote dev Kubernetes solutions sound good in theory, but suck royally in practice. Remote debugging just isn't as good and the idea that I have to be online to get any work done is such a step back that it's baffling that anyone could even propose this.

longcommonname 6y ago
Im an SRE. I started off my career as a dev. I write code, yaml, hcl every day. You sound incredibly bitter.

nojvek 6y ago

At first I thought Kubernetes was a massive overkill. However as I learn more about it, I’ve been a big fan of having a local minikube cluster for dev, jsonnet for template wrangling, and being able to ship to stage and prod the same built images. Once you get over the learning curve it makes a ton of sense why k8s is growing like wildfire.

longcommonname 6y ago

Most developers just want to work on tickets other non infra related tasks. For the majority of those contributors its really helpful to give them a happy path. This happy path also makes supporting much easier. This happy path comes out as a platform and series of tools on top of the k8s layer. I love working on infra automation, and I love enabling other teams. When I did nothing but dev work the feedback loop for when I helped someone could be months. As an SRE I help dozens of people every week with near real-time feedback.