4 ms·

A Pipeline Made of Airbags

198 points by packetlost 2y ago

swiftcoder 2y ago
It's a real shame that we are steadily losing all the lessons of Erlang/SmallTalk/Lisp machines.
- nine_k 2y ago
  What are the specific lessons worth preserving, but being lost? (I assume that "keep an image, it's too costly to rebuild everything from version-controlled sources" is not such a lesson.)
  - igouy 2y ago
    Yes specific would be better. Of course "keep an image" and "version-controlled sources" are not mutually exclusive. https://www.google.com/books/edition/Mastering_ENVY_Developer/ld6E19QIMo4C https://www.google.com/books/edition/Mastering_ENVY_Develope...
  - swiftcoder 2y ago
    The biggest common lesson is being able to inspect/interrogate/modify the running system. Debugging distributed system failures purely based on logs/metrics output is not a particularly pleasant job, and most immutable software stacks don't offer a log more than that. However, for Erlang specifically, the lesson is pushing statelessness as far down the system as you possibly can. Stateless immutable containers that we can kill at will are great - but what if we could do the same thing at the request handler level?
  - tonyg 2y ago
    > (I assume that "keep an image, it's too costly to rebuild everything from version-controlled sources" is not such a lesson.) Exactly - that lesson is in no danger of being lost. Docker and other kinds of VM image are ubiquitous.
- agumonkey 2y ago
  society often over powers good ideas.. if the zeitgeist is about kube, companies will accept its cost not really knowing the lost benefits anyway.. I guess we'll have to wait to reinstate these features later on
dools 2y ago
Ha, I recently wrote a system that does more or less the exact same thing for pushing updates to IoT devices. I can tell the system to update particular nodes to a given git commit, then I can roll it out to a handful of devices, then I can say "update all of them" but wait 30 seconds in between each update and so on.
Annatar 2y ago
[dead]
michael569 2y ago
[flagged]
jamesblonde 2y ago
Joe would be turning in his grave if he knew where industry are right now on the k8s love-bomb.
- p_l 2y ago
  The real issue is that the languages most use do not support, reliably, another approach. k8s is not the issue. Worse Is Better languages and runtimes are.
Sebb767 2y ago
The big thing about immutable infrastructure is that it is reproducible. I've seen both worlds and I do appreciate the simplicity and quickness of the upgrade solution presented in the post. The problem with this manual approach is that it is quite easy to end up with a few undocumented fixes/upgrades/changes to your pet server and suddenly upgrading or even just rebooting the servers/app becomes something scary. Now, for immutable infrastructure, you have a whole different set of problems. All your changes are nicely logged in git, but to deploy you need to rebuild containers and roll them out over a cluster. To do this smoothly, the cluster also needs to have some kind of high availability setup, making everything quite complex and, in the end, you wasted minutes to hours of compute for something that a pet setup can do in a few seconds. But you can be sure that a server going down or a reboot are completely safe operations. What works for you really depends on your situation (team size, importance of the app, etc.), but both approaches do have their uses and reducing the immutable infra approach to "people run k8s because it's hip" misses the point.
- swiftcoder 2y ago
  > undocumented fixes/upgrades/changes to your pet server and suddenly upgrading or even just rebooting the servers/app becomes something scary. You can mostly prevent this by mandating that fresh nodes come up regularly. Have a management process that keeps a rolling window of ~5% of your fleet in connection-drain, and replaces the nodes as soon as they hit low-digits of connections. Whole fleet is replaced every ~3 weeks, you learn about any deployment/startup failures within one day of new code landing in trunk, minimal disruption to client connections.
  - turtlebits 2y ago
    This doesn't prevent anything, it just schedules possible breakages because your infra isn't 100% immutable. IME, this doesn't work because companies won't implement/will deprioritize any infra changes that impact the development cycle.
    - swiftcoder 2y ago
      It's not so very different to dropping your PR into any other automated-CI/CD-all-the-way-to-prod pipeline. Albeit maybe a little easier to justify to management that you are dropping everything to fix the breakage when your CI/CD pipeline stops.
wpietri 2y ago
What a lovely and well-written piece. I think the dev vs ops divide has caused so many problems like this. We just write systems differently when we have to run things versus when it gets thrown over the wall to other people to deal with. Maybe that sounds like I'm blaming developers, but I'm not. I think this is rooted in management theories of work. They optimize for simple top-down understanding, not cross-functional collaboration. If people are rewarded for keeping to over-optimistic managerial plan (or keeping up "velocity"), then they're mostly going to throw things over the wall.
from-nibly 2y ago
I get all of these complaints. Why do I also have to be an infrastrucutre engineer? And why is my infrastructure not bespoke enough to do this weird thing I want to do? Why cant I use 5 different languages at this 30 person company? The thing about immutable infrastructure is that its straightforward. There are a set of assumptions others can make about your app. Immutable infrastructure is boring. Deployments are uncreative. Thats a good thing. Repeat after me, "my creative energy should be spent on my customers"
- ActionHank 2y ago
  I think theres more to it than that. You are correct for 90% of the cases, but this also kills innovation.
  - from-nibly 2y ago
    If you want to do insfrastructure innovation you are more than welcome to. There are lots of engineers dedicated to it. Its also not that hard to go from software engineer to infrastructure engineer thus bringing your experiece and unique perspective. But working at a SMB or startup (the 90%) doesn't justify innovation for innovations sake. 1 acre of corn doesn't justify inventing the combine.
    - andiareso 2y ago
      I love that last line. That’s the best analogy I’ve heard.
  - marcosdumay 2y ago
    The way to enable fast ops evolution is by creating a small bubble with either a mutable facade or the immutability restrictions disabled, and go innovate there. Once you are ready, you can port the changes to the overall environment. And the way to do the thing the article complains about is with partial deployments. Both of those are much better behaved on a large-scale ops than the small-scale counterparts. K8s kinda "supports" both of them, but like almost everything in k8s, it's more work, and there are many foot-guns.
thih9 2y ago
Would a middle ground be possible? E.g. by default use stateless containers, but for certain stacks or popular app frameworks support automated stateful deploys?
- mononcqc 2y ago
  Two years after writing A Pipeline Made of Airbags, I ended up prototyping a minimal way to do hot code loading from kubernetes instances by using generic images and using a sidecar to load pre-built software releases from a manifest in a way that worked both for cold restarts and for hot code loading: https://ferd.ca/my-favorite-erlang-container.html https://ferd.ca/my-favorite-erlang-container.html It's more or less as close to a middle-ground as I could imagine at the time.
specialist 2y ago
Exactly right. OC hits all the points. Fine granularity of failure, (re)warming caches, faster iteration by reducing cost of changes, etc. I too lived this. Albeit with "poor man's Erlang" (aka Java). Our customers were hospitals, ERs, etc. Our stack could not go down. And it had to be correct. So sometimes that means manual human intervention. There's another critial distinction, missed by the "whole freaking docker-meets-kubernetes" herd: Our deployed systems were "pets". Whereas k8s is meant for "cattle". Different tools for different use cases.
turtlebits 2y ago
A well crafted app is great. It is also complex and generally only maintainable/supportable by those who built it.
- kmoser 2y ago
  If the original devs wrote good documentation then pretty much anybody can maintain it easily.
- michaelteter 2y ago
  A truly well crafted app requires very little maintenance or support, and that maintenance/support has already been throught through and made easy to learn and do. These things are possible, and they fit economically somewhere in the 3-5 year maturity of a system. Years 1-3 are usually necessarily focused on features and releases, but far too many orgs just stop at that point and aren't willing to invest that extra year or two in work that will save time and money for many years to follow. I believe this resistance is due to the short-sightedness of buyouts/IPOs or simply leadership churn.
anothername12 2y ago
K8s is a Google thing for Google problems. It’s just not needed for most software delivery. Edit: meanwhile I’m waiting 45 minutes on average for my most recent, single-line change, PR to roll out to k8s cluster at $massive_company that totally does this like everyone else.
- fragmede 2y ago
  Google doesn't use Kubernetes internally. Kubernetes is a simplified version of Borg. If it's taking 45 minutes to deploy a change, that's on your company's platform team, not Google.

198 points by packetlost 2y ago

swiftcoder 2y ago

It's a real shame that we are steadily losing all the lessons of Erlang/SmallTalk/Lisp machines.

nine_k 2y ago
What are the specific lessons worth preserving, but being lost? (I assume that "keep an image, it's too costly to rebuild everything from version-controlled sources" is not such a lesson.)
- igouy 2y ago
  Yes specific would be better. Of course "keep an image" and "version-controlled sources" are not mutually exclusive. https://www.google.com/books/edition/Mastering_ENVY_Developer/ld6E19QIMo4C https://www.google.com/books/edition/Mastering_ENVY_Develope...
- swiftcoder 2y ago
  The biggest common lesson is being able to inspect/interrogate/modify the running system. Debugging distributed system failures purely based on logs/metrics output is not a particularly pleasant job, and most immutable software stacks don't offer a log more than that. However, for Erlang specifically, the lesson is pushing statelessness as far down the system as you possibly can. Stateless immutable containers that we can kill at will are great - but what if we could do the same thing at the request handler level?
- tonyg 2y ago
  > (I assume that "keep an image, it's too costly to rebuild everything from version-controlled sources" is not such a lesson.) Exactly - that lesson is in no danger of being lost. Docker and other kinds of VM image are ubiquitous.
agumonkey 2y ago
society often over powers good ideas.. if the zeitgeist is about kube, companies will accept its cost not really knowing the lost benefits anyway.. I guess we'll have to wait to reinstate these features later on

dools 2y ago

Ha, I recently wrote a system that does more or less the exact same thing for pushing updates to IoT devices. I can tell the system to update particular nodes to a given git commit, then I can roll it out to a handful of devices, then I can say "update all of them" but wait 30 seconds in between each update and so on.

Annatar 2y ago

[dead]

michael569 2y ago

[flagged]

jamesblonde 2y ago

Joe would be turning in his grave if he knew where industry are right now on the k8s love-bomb.

p_l 2y ago
The real issue is that the languages most use do not support, reliably, another approach. k8s is not the issue. Worse Is Better languages and runtimes are.

Sebb767 2y ago

The big thing about immutable infrastructure is that it is reproducible. I've seen both worlds and I do appreciate the simplicity and quickness of the upgrade solution presented in the post. The problem with this manual approach is that it is quite easy to end up with a few undocumented fixes/upgrades/changes to your pet server and suddenly upgrading or even just rebooting the servers/app becomes something scary. Now, for immutable infrastructure, you have a whole different set of problems. All your changes are nicely logged in git, but to deploy you need to rebuild containers and roll them out over a cluster. To do this smoothly, the cluster also needs to have some kind of high availability setup, making everything quite complex and, in the end, you wasted minutes to hours of compute for something that a pet setup can do in a few seconds. But you can be sure that a server going down or a reboot are completely safe operations. What works for you really depends on your situation (team size, importance of the app, etc.), but both approaches do have their uses and reducing the immutable infra approach to "people run k8s because it's hip" misses the point.

swiftcoder 2y ago
> undocumented fixes/upgrades/changes to your pet server and suddenly upgrading or even just rebooting the servers/app becomes something scary. You can mostly prevent this by mandating that fresh nodes come up regularly. Have a management process that keeps a rolling window of ~5% of your fleet in connection-drain, and replaces the nodes as soon as they hit low-digits of connections. Whole fleet is replaced every ~3 weeks, you learn about any deployment/startup failures within one day of new code landing in trunk, minimal disruption to client connections.
- turtlebits 2y ago
  This doesn't prevent anything, it just schedules possible breakages because your infra isn't 100% immutable. IME, this doesn't work because companies won't implement/will deprioritize any infra changes that impact the development cycle.
  - swiftcoder 2y ago
    It's not so very different to dropping your PR into any other automated-CI/CD-all-the-way-to-prod pipeline. Albeit maybe a little easier to justify to management that you are dropping everything to fix the breakage when your CI/CD pipeline stops.

wpietri 2y ago

What a lovely and well-written piece. I think the dev vs ops divide has caused so many problems like this. We just write systems differently when we have to run things versus when it gets thrown over the wall to other people to deal with. Maybe that sounds like I'm blaming developers, but I'm not. I think this is rooted in management theories of work. They optimize for simple top-down understanding, not cross-functional collaboration. If people are rewarded for keeping to over-optimistic managerial plan (or keeping up "velocity"), then they're mostly going to throw things over the wall.

from-nibly 2y ago

I get all of these complaints. Why do I also have to be an infrastrucutre engineer? And why is my infrastructure not bespoke enough to do this weird thing I want to do? Why cant I use 5 different languages at this 30 person company? The thing about immutable infrastructure is that its straightforward. There are a set of assumptions others can make about your app. Immutable infrastructure is boring. Deployments are uncreative. Thats a good thing. Repeat after me, "my creative energy should be spent on my customers"

ActionHank 2y ago
I think theres more to it than that. You are correct for 90% of the cases, but this also kills innovation.
- from-nibly 2y ago
  If you want to do insfrastructure innovation you are more than welcome to. There are lots of engineers dedicated to it. Its also not that hard to go from software engineer to infrastructure engineer thus bringing your experiece and unique perspective. But working at a SMB or startup (the 90%) doesn't justify innovation for innovations sake. 1 acre of corn doesn't justify inventing the combine.
  - andiareso 2y ago
    I love that last line. That’s the best analogy I’ve heard.
- marcosdumay 2y ago
  The way to enable fast ops evolution is by creating a small bubble with either a mutable facade or the immutability restrictions disabled, and go innovate there. Once you are ready, you can port the changes to the overall environment. And the way to do the thing the article complains about is with partial deployments. Both of those are much better behaved on a large-scale ops than the small-scale counterparts. K8s kinda "supports" both of them, but like almost everything in k8s, it's more work, and there are many foot-guns.

thih9 2y ago

Would a middle ground be possible? E.g. by default use stateless containers, but for certain stacks or popular app frameworks support automated stateful deploys?

mononcqc 2y ago
Two years after writing A Pipeline Made of Airbags, I ended up prototyping a minimal way to do hot code loading from kubernetes instances by using generic images and using a sidecar to load pre-built software releases from a manifest in a way that worked both for cold restarts and for hot code loading: https://ferd.ca/my-favorite-erlang-container.html https://ferd.ca/my-favorite-erlang-container.html It's more or less as close to a middle-ground as I could imagine at the time.

specialist 2y ago

Exactly right. OC hits all the points. Fine granularity of failure, (re)warming caches, faster iteration by reducing cost of changes, etc. I too lived this. Albeit with "poor man's Erlang" (aka Java). Our customers were hospitals, ERs, etc. Our stack could not go down. And it had to be correct. So sometimes that means manual human intervention. There's another critial distinction, missed by the "whole freaking docker-meets-kubernetes" herd: Our deployed systems were "pets". Whereas k8s is meant for "cattle". Different tools for different use cases.

turtlebits 2y ago

A well crafted app is great. It is also complex and generally only maintainable/supportable by those who built it.

kmoser 2y ago
If the original devs wrote good documentation then pretty much anybody can maintain it easily.
michaelteter 2y ago
A truly well crafted app requires very little maintenance or support, and that maintenance/support has already been throught through and made easy to learn and do. These things are possible, and they fit economically somewhere in the 3-5 year maturity of a system. Years 1-3 are usually necessarily focused on features and releases, but far too many orgs just stop at that point and aren't willing to invest that extra year or two in work that will save time and money for many years to follow. I believe this resistance is due to the short-sightedness of buyouts/IPOs or simply leadership churn.

anothername12 2y ago

K8s is a Google thing for Google problems. It’s just not needed for most software delivery. Edit: meanwhile I’m waiting 45 minutes on average for my most recent, single-line change, PR to roll out to k8s cluster at $massive_company that totally does this like everyone else.

fragmede 2y ago
Google doesn't use Kubernetes internally. Kubernetes is a simplified version of Borg. If it's taking 45 minutes to deploy a change, that's on your company's platform team, not Google.