9 ms·

A Pipeline Made of Airbags

125 points by bshanks 6y ago

gorgoiler 6y ago
I always admired PHP’s runtime for the way it serves web requests. I think it’s a similar ethos to that described in the post, even though Erlang and PHP are quite different. Start from nothing. No shared state with previous requests. Plough right through as many errors as you can. Log them, but don’t stop. Keep executing code until you run out of things to do.
- TylerE 6y ago
  I've always thought that's precisely why PHP is bad. It throws out the context baby with the state bathwater.
- fn1 6y ago
  This is probable the reason for why it's so popular (meaning wordpress), but also why it's so full of security exploits. Fail-fast leads to more stable systems.
  - mping 6y ago
    Generally agree, but if erlang was as popular as wordpress we don't know what kind of issues hackers would find
    - rkangel 6y ago
      I think it would be ok, but part of the question is what you mean by "popular". WhatsApp famously runs on Erlang, as do large chunks of our telecomms infrastructure. It has an enormous number of users. What it doesn't have is an enormous number of developers. One of the good things about the Erlang (and Elixir) ecosystem is that there are a relatively small number of people implementing the bits you care about, and they all seem to be extremely competent engineers. The pieces are designed well, they are reliable and they fit together. They also have good abstractions so if you use good pieces then you get good results (e.g. with Phoenix you'd have a hard time creating a XSRF or SQL injection problem). PHP has a bit of a perfect storm of problems - it is so accessible that anyone can and did use it. This meant a lot of people implemented things using it, with varying levels of competence. This means a lot of problematic implementation of a lot of pieces, built on a language and framework that didn't really try to prevent security problems and so didn't really have the tools to do so. In contrast to Erlang/Elixir above, you can take a set of tools in PHP and easily implement a very insecure website.
- toast0 6y ago
  I love both Erlang and PHP, but the philosophy of what to do with errors in requesrs is like diametrically opposed. For PHP, usually it's log, but keep on going as best you can. For Erlang, it's log and stop, and the request handler will generally be restarted. I guess where they're similar is that an error is generally going to only affect that one request, the next request will have a fresh handler. (Of course, these are generalizations, you can always do things differently in either language).
- vlovich123 6y ago
  I’d recommending reading the post about the difference between erlang and php and reading the article again. This isn’t silently swallow errors. It’s saying start with fail aggressively but also don’t forget to identify situations resulting in failures in the first place and fix those bugs our handle the corner conditions properly. That’s diametrically incompatible with “log and continue” as at no point is that desirable according to this post.
asdfasgasdgasdg 6y ago
I can understand the technical argument against the stateless containerized model. It's not neat. But what is the business argument? That's where this perspective falls down for me. If you're building a database that manages a disk directly, I can see the argument for graceful shutdown and handover. In that case you really need the program to be colocated with the data and if you're down then the data is unavailable. I've built that once and it was hell to get right. But probably 90+% of programs I've ever worked on do not achieve any business advantage from graceful restarts (as opposed to drain and restart i.e. the stateless model).
- bostonsre 6y ago
  Kuberenetes is a leaky abstraction that by default allows the deployment of sloppy unreliable apps if you don't know what you're doing. If something is broken and it just restarts automatically and no one is paying attention, it will do that in perpetuity. Ideally, you would see those issues and iterate to make your app more stable, but it makes it easy to just forget about it. The additional layer of complexity is also harder to understand easily for most devs that haven't spent the time to learn kubernetes and even with devs that do spend the time it can still make understanding how everything works hard. Letting tech debt like unstable apps grow into mud balls can reduce a team's velocity in the long term and in some cases the short term as well. Kubernetes is great for easily sharing and executing recipes, but it can make it tough for people that want to become chefs that are able to use reason and intuition to build and understand complex systems.
  - bweitzman 6y ago
    On the flip side, why bother spending time optimizing a process if nobody notices when it fails? Aren't there more important problems to be solving in at scenario?
    - knl 6y ago
      Maybe people do notice, but it’s not easy to report back? Or they notice the system runs slowly? Or they just accept the fact that all systems (extrapolating from several banking, public service, and publishing sites) are crap, and need to live with it?
winrid 6y ago
Here's a neat comparison that might make your blood boil. You can install Windows (proved with XP) in five minutes or less. How many build pipelines have you worked with that take longer than installing Windows? Granted, they're different things - one is copying a giant set of files and creating a small DB and one is running tests and packaging software. Here's the thing though. Most of the time, these builds are slow due to 1. lack of parallelism and 2. Blindly following "best practices" instead of using our $170k+ Silicon Valley brains. These are problems you can solve. You can get your build to be faster than installing Windows. Will most companies let their developers do things that may not be "best practices" to do that? Not today.
- saagarjha 6y ago
  I work with a project that takes an hour to build because it’s thousands of C++ files. What would you suggest I do to improve my build times?
  - coding_unit_1 6y ago
    Do you need to recompile all multi-thousand files every time you make a change? Modularising the build would be my first step.
  - magicalhippo 6y ago
    A buddy slashed his CI build times by using a RAM disk. But yeah, usually faster to avoid compiling things.
  - froj 6y ago
    We have a similar situation and use ccache and distcc is setup on each dev's workstation. It also helps to use forward declarations instead of including header files whenever possible.
  - rkangel 6y ago
    A good build system and caching. By "a good build system" I mean a hermetic one like Bazel that can be confident of knowing the inputs to each build step so that they can be run or not as appropriate. Sometimes you'll touch a core header file and a lot of things will rebuild, but it should still be a lot better.
gbrown_ 6y ago
Not sure why this wasn't marked a dupe of https://news.ycombinator.com/item?id=24577881 https://news.ycombinator.com/item?id=24577881 which was submitted ~10 hours earlier and I think was briefly on the front page.
blunte 6y ago
At some point most of us (from engineers to CTO) fall for a "silver bullet". Then after a while, reality speaks up loudly and we decide things like, "not everything should be an object", or "not every function should be a microservice", or now "containers aren't the only solution".
dls2016 6y ago
I thought this was going to be about Kramerica Industries.
geofft 6y ago
This is a weird article. If the two systems the author described worked so well, how did the business ever decide to "bulldoze" it and replace it with something inferior? Even if you take the view that some non-technical person came in and demanded a particular technical change (??), any halfway-competent business is going to keep the old thing running while they build the new one so they can continue making money, and they'll notice that the new thing doesn't work as well as the old. Right? I get the sense something is missing, and I get the sense the missing thing was that the replacements were actually better for the business. A couple of highly-talented people on a small team where the original developers owned production operations could run it as designed, and it was "really neat," but the business was limited in how they could use the system. Perhaps you couldn't easily add developers and make them work on new features. Perhaps you couldn't integrate OSS components that worked fine, and you were left behind your competitors. Perhaps the "one or two developers" who operated the system got married and started a family and weren't able to be on-call 24/7/365 any longer. Perhaps this system was one of dozens run by the business, and all the other ones were using crash-only containers, and this system continued to be the odd one out. I'd like to hear from the other side of this - I'd like to hear the stories of the people who pushed the redesign away from the systems. Until then, this seems entirely too much like "We built this awesome prototype with a bunch of neat properties in theory." My suspicion is that the other side of these migrations were also driven by highly-talented technical people who have technical stories to tell, but their story is that they cleared out a bunch of technical debt and replaced it with a higher-reliability system.
- mononcqc 6y ago
  Author here. For the first system, it was deprecated without replacement, and just let to run by managers and people who had moved on to other teams (but used to work on it) who did the minimal maintenance required, and former employees were given emergency contracts in weird circumstances to deal with things. Roughly 3-4 years after, they finally replaced it after 2-3 attempts at rewrites that had failed before. The old design with minimal maintenance for years finally approached limits to how it could be scale without bigger redesigns; I consider this to be extremely successful. It wasn't exactly bulldozed nearly as much as declared "done" and abandoned without adequate replacements while major parts of the business was just being rewritten to use Go and a more "standard" stack. Obviously these migrations always start with something easier and by replacing components that are huge pain points to its contemporaries, and you're left with more legacy stuff in the end that is much harder to replace done in the final pushes. I felt that the blog post would have veered off point if the whole thing became about that, though. The people on these teams left in part because the hiring budget was redirected towards hiring on the new projects. The idea was that everything could be done in Go for these stacks (by normalizing on tools and libraries developed in another project and wanting to have one implementation for both the private and shared platforms), and the rewrites were to start with Ruby components. You knew working on the Erlang side of things that no feature development would ever take place again, that no new hands would be hired to help, and that you would be stuck on call 24/7 with no relief for years. All efforts were redirected to Go and getting rid of Ruby, and your stuff fell in between the cracks. I was one of the people who left on the long tail there. After my departure, I was brought back on a lucrative part-time contract as a sort of retainer for years to help them in case of major outages (got 1 or 2 in 4 years) since that was the only way they could get expert knowledge once they drove us all away. I'm still on good terms with the people there, it's just that "maintaining a self-declared unmaintained legacy stack without budget or headcount until we get to rewrite it in many years" is not where any of us wanted to drive our careers. Interestingly, we tried very hard to add new developers. We wrote manuals, tooling, a book on operating these systems (see https://www.erlang-in-anger.com https://www.erlang-in-anger.com), wanted to set up internal internships so developers from other teams could come and work with us for a while, etc. Whereas our team was very willing, internal politics (which I can't easily get in a public forum) made it unworkable and most attempts were turned down. These things were not always purely business decisions, and organizational dynamics can be very funny things.
losvedir 6y ago
We run a lot of production Elixir apps at work, but we deploy them all in Docker containers on AWS basically. The whole live-upgrade erlang philosophy has always been one I've been curious about and interested in, but also a little skeptical of. It's funny to me that erlang (and Elixir) are so dogmatically immutable / no shared state "in the small" at the module and function level, but apparently not "in the large", at the system level. Whereas you tend to see the reverse with deploying most other languages.
- toast0 6y ago
  Erlang language level immutability is a means to an end, it makes the runtime (especially garbage collection) simpler than a system with mutable state where you can have references to newer and older data. At the systems level, immutability makes operating the system much harder. Especially when you consider Ericcson's basic design requirement of a telephone switch with a redundant pair of control computers; it's safer to deploy changes by making a temporary change to the live system than a permanent change to the backup system and switching over. In case of a bad deploy, you can live reload the old version on the live system, or if that's not possible, promote the backup system and restart the primary back to the known good state. If all goes well though, there's no period of reduced redundancy. In a restart to deploy situation, every deploy has a period with loss of redundancy, as well as a primary/secondary switch, and you can't deploy to degraded systems where one of the redundant pair is not working pending repair (these switches would most likely be deployed to towards the end of the cycle to reduce risk)
- elcritch 6y ago
  A large part of the philosophy is controlling state. Immutability in the small helps control where state lives. It’s difficult to control (or understand) state spread across hundreds of little objects (e.g. most OO languages) and be able to modify a running system. OTP Supervision trees and GenServers are all about partitioning state and organizing it. It’s similar in many ways to micro services, but more advanced (as the article mentioned regarding OTP Supervision trees and k8s).
luord 6y ago
While I still love/docker and containerization and use them (judiciously), this article does describe a couple of situations I've witnessed. One way to describe it, and I forgot where I read it, is that one advantage of interpreted languages used to be that you no longer had to grab a cup of coffee every hour while your work compiled. With containerization, things are even now: everyone gets to wait while the images get built. Of course, I've also worked at places where the environments are set correctly and take advantage of the tech the team is using, instead of having to rebuild all the time. Which is why I don't hate containerization on its own. Everything can be badly applied or used.
theptip 6y ago
If you really need stateful services in k8s, why not just use a StatefulSet and keep your old deploy process?

125 points by bshanks 6y ago

I always admired PHP’s runtime for the way it serves web requests. I think it’s a similar ethos to that described in the post, even though Erlang and PHP are quite different. Start from nothing. No shared state with previous requests. Plough right through as many errors as you can. Log them, but don’t stop. Keep executing code until you run out of things to do.

TylerE 6y ago
I've always thought that's precisely why PHP is bad. It throws out the context baby with the state bathwater.
fn1 6y ago
This is probable the reason for why it's so popular (meaning wordpress), but also why it's so full of security exploits. Fail-fast leads to more stable systems.
- mping 6y ago
  Generally agree, but if erlang was as popular as wordpress we don't know what kind of issues hackers would find
  - rkangel 6y ago
    I think it would be ok, but part of the question is what you mean by "popular". WhatsApp famously runs on Erlang, as do large chunks of our telecomms infrastructure. It has an enormous number of users. What it doesn't have is an enormous number of developers. One of the good things about the Erlang (and Elixir) ecosystem is that there are a relatively small number of people implementing the bits you care about, and they all seem to be extremely competent engineers. The pieces are designed well, they are reliable and they fit together. They also have good abstractions so if you use good pieces then you get good results (e.g. with Phoenix you'd have a hard time creating a XSRF or SQL injection problem). PHP has a bit of a perfect storm of problems - it is so accessible that anyone can and did use it. This meant a lot of people implemented things using it, with varying levels of competence. This means a lot of problematic implementation of a lot of pieces, built on a language and framework that didn't really try to prevent security problems and so didn't really have the tools to do so. In contrast to Erlang/Elixir above, you can take a set of tools in PHP and easily implement a very insecure website.
toast0 6y ago
I love both Erlang and PHP, but the philosophy of what to do with errors in requesrs is like diametrically opposed. For PHP, usually it's log, but keep on going as best you can. For Erlang, it's log and stop, and the request handler will generally be restarted. I guess where they're similar is that an error is generally going to only affect that one request, the next request will have a fresh handler. (Of course, these are generalizations, you can always do things differently in either language).
vlovich123 6y ago
I’d recommending reading the post about the difference between erlang and php and reading the article again. This isn’t silently swallow errors. It’s saying start with fail aggressively but also don’t forget to identify situations resulting in failures in the first place and fix those bugs our handle the corner conditions properly. That’s diametrically incompatible with “log and continue” as at no point is that desirable according to this post.

asdfasgasdgasdg 6y ago

I can understand the technical argument against the stateless containerized model. It's not neat. But what is the business argument? That's where this perspective falls down for me. If you're building a database that manages a disk directly, I can see the argument for graceful shutdown and handover. In that case you really need the program to be colocated with the data and if you're down then the data is unavailable. I've built that once and it was hell to get right. But probably 90+% of programs I've ever worked on do not achieve any business advantage from graceful restarts (as opposed to drain and restart i.e. the stateless model).

bostonsre 6y ago
Kuberenetes is a leaky abstraction that by default allows the deployment of sloppy unreliable apps if you don't know what you're doing. If something is broken and it just restarts automatically and no one is paying attention, it will do that in perpetuity. Ideally, you would see those issues and iterate to make your app more stable, but it makes it easy to just forget about it. The additional layer of complexity is also harder to understand easily for most devs that haven't spent the time to learn kubernetes and even with devs that do spend the time it can still make understanding how everything works hard. Letting tech debt like unstable apps grow into mud balls can reduce a team's velocity in the long term and in some cases the short term as well. Kubernetes is great for easily sharing and executing recipes, but it can make it tough for people that want to become chefs that are able to use reason and intuition to build and understand complex systems.
- bweitzman 6y ago
  On the flip side, why bother spending time optimizing a process if nobody notices when it fails? Aren't there more important problems to be solving in at scenario?
  - knl 6y ago
    Maybe people do notice, but it’s not easy to report back? Or they notice the system runs slowly? Or they just accept the fact that all systems (extrapolating from several banking, public service, and publishing sites) are crap, and need to live with it?

winrid 6y ago

Here's a neat comparison that might make your blood boil. You can install Windows (proved with XP) in five minutes or less. How many build pipelines have you worked with that take longer than installing Windows? Granted, they're different things - one is copying a giant set of files and creating a small DB and one is running tests and packaging software. Here's the thing though. Most of the time, these builds are slow due to 1. lack of parallelism and 2. Blindly following "best practices" instead of using our $170k+ Silicon Valley brains. These are problems you can solve. You can get your build to be faster than installing Windows. Will most companies let their developers do things that may not be "best practices" to do that? Not today.

saagarjha 6y ago
I work with a project that takes an hour to build because it’s thousands of C++ files. What would you suggest I do to improve my build times?
- coding_unit_1 6y ago
  Do you need to recompile all multi-thousand files every time you make a change? Modularising the build would be my first step.
- magicalhippo 6y ago
  A buddy slashed his CI build times by using a RAM disk. But yeah, usually faster to avoid compiling things.
- froj 6y ago
  We have a similar situation and use ccache and distcc is setup on each dev's workstation. It also helps to use forward declarations instead of including header files whenever possible.
- rkangel 6y ago
  A good build system and caching. By "a good build system" I mean a hermetic one like Bazel that can be confident of knowing the inputs to each build step so that they can be run or not as appropriate. Sometimes you'll touch a core header file and a lot of things will rebuild, but it should still be a lot better.

gbrown_ 6y ago

Not sure why this wasn't marked a dupe of https://news.ycombinator.com/item?id=24577881 https://news.ycombinator.com/item?id=24577881 which was submitted ~10 hours earlier and I think was briefly on the front page.

blunte 6y ago

At some point most of us (from engineers to CTO) fall for a "silver bullet". Then after a while, reality speaks up loudly and we decide things like, "not everything should be an object", or "not every function should be a microservice", or now "containers aren't the only solution".

dls2016 6y ago

I thought this was going to be about Kramerica Industries.

geofft 6y ago

This is a weird article. If the two systems the author described worked so well, how did the business ever decide to "bulldoze" it and replace it with something inferior? Even if you take the view that some non-technical person came in and demanded a particular technical change (??), any halfway-competent business is going to keep the old thing running while they build the new one so they can continue making money, and they'll notice that the new thing doesn't work as well as the old. Right? I get the sense something is missing, and I get the sense the missing thing was that the replacements were actually better for the business. A couple of highly-talented people on a small team where the original developers owned production operations could run it as designed, and it was "really neat," but the business was limited in how they could use the system. Perhaps you couldn't easily add developers and make them work on new features. Perhaps you couldn't integrate OSS components that worked fine, and you were left behind your competitors. Perhaps the "one or two developers" who operated the system got married and started a family and weren't able to be on-call 24/7/365 any longer. Perhaps this system was one of dozens run by the business, and all the other ones were using crash-only containers, and this system continued to be the odd one out. I'd like to hear from the other side of this - I'd like to hear the stories of the people who pushed the redesign away from the systems. Until then, this seems entirely too much like "We built this awesome prototype with a bunch of neat properties in theory." My suspicion is that the other side of these migrations were also driven by highly-talented technical people who have technical stories to tell, but their story is that they cleared out a bunch of technical debt and replaced it with a higher-reliability system.

mononcqc 6y ago
Author here. For the first system, it was deprecated without replacement, and just let to run by managers and people who had moved on to other teams (but used to work on it) who did the minimal maintenance required, and former employees were given emergency contracts in weird circumstances to deal with things. Roughly 3-4 years after, they finally replaced it after 2-3 attempts at rewrites that had failed before. The old design with minimal maintenance for years finally approached limits to how it could be scale without bigger redesigns; I consider this to be extremely successful. It wasn't exactly bulldozed nearly as much as declared "done" and abandoned without adequate replacements while major parts of the business was just being rewritten to use Go and a more "standard" stack. Obviously these migrations always start with something easier and by replacing components that are huge pain points to its contemporaries, and you're left with more legacy stuff in the end that is much harder to replace done in the final pushes. I felt that the blog post would have veered off point if the whole thing became about that, though. The people on these teams left in part because the hiring budget was redirected towards hiring on the new projects. The idea was that everything could be done in Go for these stacks (by normalizing on tools and libraries developed in another project and wanting to have one implementation for both the private and shared platforms), and the rewrites were to start with Ruby components. You knew working on the Erlang side of things that no feature development would ever take place again, that no new hands would be hired to help, and that you would be stuck on call 24/7 with no relief for years. All efforts were redirected to Go and getting rid of Ruby, and your stuff fell in between the cracks. I was one of the people who left on the long tail there. After my departure, I was brought back on a lucrative part-time contract as a sort of retainer for years to help them in case of major outages (got 1 or 2 in 4 years) since that was the only way they could get expert knowledge once they drove us all away. I'm still on good terms with the people there, it's just that "maintaining a self-declared unmaintained legacy stack without budget or headcount until we get to rewrite it in many years" is not where any of us wanted to drive our careers. Interestingly, we tried very hard to add new developers. We wrote manuals, tooling, a book on operating these systems (see https://www.erlang-in-anger.com https://www.erlang-in-anger.com), wanted to set up internal internships so developers from other teams could come and work with us for a while, etc. Whereas our team was very willing, internal politics (which I can't easily get in a public forum) made it unworkable and most attempts were turned down. These things were not always purely business decisions, and organizational dynamics can be very funny things.

losvedir 6y ago

We run a lot of production Elixir apps at work, but we deploy them all in Docker containers on AWS basically. The whole live-upgrade erlang philosophy has always been one I've been curious about and interested in, but also a little skeptical of. It's funny to me that erlang (and Elixir) are so dogmatically immutable / no shared state "in the small" at the module and function level, but apparently not "in the large", at the system level. Whereas you tend to see the reverse with deploying most other languages.

toast0 6y ago
Erlang language level immutability is a means to an end, it makes the runtime (especially garbage collection) simpler than a system with mutable state where you can have references to newer and older data. At the systems level, immutability makes operating the system much harder. Especially when you consider Ericcson's basic design requirement of a telephone switch with a redundant pair of control computers; it's safer to deploy changes by making a temporary change to the live system than a permanent change to the backup system and switching over. In case of a bad deploy, you can live reload the old version on the live system, or if that's not possible, promote the backup system and restart the primary back to the known good state. If all goes well though, there's no period of reduced redundancy. In a restart to deploy situation, every deploy has a period with loss of redundancy, as well as a primary/secondary switch, and you can't deploy to degraded systems where one of the redundant pair is not working pending repair (these switches would most likely be deployed to towards the end of the cycle to reduce risk)
elcritch 6y ago
A large part of the philosophy is controlling state. Immutability in the small helps control where state lives. It’s difficult to control (or understand) state spread across hundreds of little objects (e.g. most OO languages) and be able to modify a running system. OTP Supervision trees and GenServers are all about partitioning state and organizing it. It’s similar in many ways to micro services, but more advanced (as the article mentioned regarding OTP Supervision trees and k8s).

luord 6y ago

While I still love/docker and containerization and use them (judiciously), this article does describe a couple of situations I've witnessed. One way to describe it, and I forgot where I read it, is that one advantage of interpreted languages used to be that you no longer had to grab a cup of coffee every hour while your work compiled. With containerization, things are even now: everyone gets to wait while the images get built. Of course, I've also worked at places where the environments are set correctly and take advantage of the tech the team is using, instead of having to rebuild all the time. Which is why I don't hate containerization on its own. Everything can be badly applied or used.

theptip 6y ago

If you really need stateful services in k8s, why not just use a StatefulSet and keep your old deploy process?