11 ms·

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great e

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails all the time at this scale, but heals itself pretty quickly. In over 1 year of operation it hasn't caused a single incident, and it has survived all outages.

kortilla 4y ago
This is just kubernetes right? Declarative desired state model. Containers created and destroyed to get there. Crashes happen, metrics are incremented, load balancers route around the crashing pod until they recover (or are replaced), etc.
arunaugustine 4y ago
Yes, please give us more info about using control theory and how one might think about building such a system please..
- wsc981 4y ago
  My guess would be add assertions everywhere instead of throwing exceptions.
  - Karellen 4y ago
    Why not throw exceptions, and just never use try/catch? That way, all exceptions are uncaught and should terminate the program, in a way that takes advantage of the programming language's native error reporting facilities.
    - hsn915 4y ago
      assertion failures terminate the program immediately. exceptions usually trigger cleanup code. If your cleanup code is mostly closing files and clearing memory, then it's useless because the OS will do that for your crashed program anyway.
  - roeles 4y ago
    I don't know of a way to test this behavior (I mainly code C++ and unit test with Google Test). One could spawn a process and capture the output and return value, but that sounds a bit heavy for just testing if your error handling still works as intended.
- angarg12 4y ago
  These kind of systems are not always appropriate, but when they do, they work wonderfully. Our use case was to build a service to manage the dynamic part of our infrastructure. These are infra pieces that are created/deleted/modified on the fly according to some policies, instead of defining them statically as code. The implementation is simply a lambda function that runs every minute, loads a policy, compares the current state of the system, and then creates/deletes/modify resources as needed. I am currently in the process of writing a talk that I will deliver to the rest of my org. This will help me crystalize my thoughts, but here are some pointer on why I think it worked in this case: * The service is stateless. On each run it just loads a policy, compares it to the current state of the system, and acts accordingly. This avoids handling complicated state or coordinating executions. In theory two policies could contradict each other, but in practice we partition our policies in such a way overlap is not possible. * Operations are idempotent. This is one of the reasons the system converges to a desired state. This makes the service resilient to both failures and eventual consistency. * Deviation from policies doesn't affect correctness. We are fortunate that our system is not directly customer facing. Deviation from policies affects only performance. The system can work several minutes (if not hours) outside policy band without consequences. This probably will be a critical blocker for most production systems. Other than control theory and let it fail, I had the chance to play with other cool concepts while implementing this service. These are some of those: * Parse, don't validate/anti-corruption layer: The service downloads and parses a policy at the beginning of each run. If parsing fails, it errors out. Otherwise, it passes the policy object to the rest of the execution. This makes the system easy to test, and avoids the anti-pattern of peppering your code with instructions reading input, just to find mid-execution that the policy was invalid. * Pluggable policies: The main body of the service is a very simple sense/act loop. For the actors, we use a strategy pattern, where policies can choose what strategy to use. This approach has helped us to introduce new behavior with minimal code changes. * Typescript as configuration language: This service replaces an older, less flexible one. A major pain of the old service is that policies where defined as Jinja templates over plain text files. This became unmaintainable as the number and complexity of policies grew. Our new service defines policies in Typescript. Policies are statically typed, and we use regular programming constructs (functions, loops, variables...) to build them at compile time. The output is still a plain JSON file. Hope that helps.
- Too 4y ago
  This is how Kubernetes works in many ways. Crash a pod and the control loop inside the ReplicaSet will create a new pod for you. Scaling nodes is based on similar principles of desired vs actual values.
fire 4y ago
this is the type of thing I'd love to see code / a post about implementing
tsimionescu 4y ago
This is exactly why I think all the discussions about the importance of error handling paths (and the aversion drive have to exceptions) are usually overblown. The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the error. If you don't, then exceptions are the best approximation of this pattern - much better than result types or error codes, which litter your code with irrelevant error handling details.
- dfee 4y ago
  Totally agree. Now, I’ve actually found functional programming - specifically, Either<Error, Value> - to be of great use in helping me focus only on the happy path.
  - tsimionescu 4y ago
    Yes, with some kind of monadic-like control flow (either actual monads or even Rust's ? macro), those can be also achieve this workflow pretty well. Edit to add: I still think exceptions are better in practice, as you also get a stack trace when the failure happens, whereas Either and ? don't really help track down the error unless you add code to create a manual "stack trace".
    - rklaehn 4y ago
      In rust, you can use the excellent anyhow crate https://docs.rs/anyhow/latest/anyhow/ https://docs.rs/anyhow/latest/anyhow/ . It has various ways to add context to an error, and will automatically attach a stack trace with the backtrace feature. Explicit or implicit panic of course also attaches a backtrace. It can also be caught, although that is a can of worms. So panicing is the closest thing rust has to exceptions - somewhat similar to java.lang.Error on the jvm. https://docs.oracle.com/javase/7/docs/api/java/lang/Error.html https://docs.oracle.com/javase/7/docs/api/java/lang/Error.ht... With anyhow, error handling in rust really is quite pleasant.
- 4y ago
secondcoming 4y ago
> fails and logs What if your logging code was written with the same philosophy?
- angarg12 4y ago
  In another comment I mention that our service has some properties which make this a great solution. It doesn't always work, but when it does, it's awesome. To your point, Erlang has the concept of supervisor trees and handling of errors. Similarly, our supervisor is the Lambda runtime. If everything goes wrong, the runtime will emit a metric error. But what if Lambda itself fails? Or Cloudwatch? or any other supporting service? of course this is possible. But probably at that point the system is so fucked up that the fate of our little service is of little relevance. At least we know from past outages that when normal operations resume, the system will correct itself.

kortilla 4y ago

This is just kubernetes right? Declarative desired state model. Containers created and destroyed to get there. Crashes happen, metrics are incremented, load balancers route around the crashing pod until they recover (or are replaced), etc.

arunaugustine 4y ago

Yes, please give us more info about using control theory and how one might think about building such a system please..

wsc981 4y ago
My guess would be add assertions everywhere instead of throwing exceptions.
- Karellen 4y ago
  Why not throw exceptions, and just never use try/catch? That way, all exceptions are uncaught and should terminate the program, in a way that takes advantage of the programming language's native error reporting facilities.
  - hsn915 4y ago
    assertion failures terminate the program immediately. exceptions usually trigger cleanup code. If your cleanup code is mostly closing files and clearing memory, then it's useless because the OS will do that for your crashed program anyway.
- roeles 4y ago
  I don't know of a way to test this behavior (I mainly code C++ and unit test with Google Test). One could spawn a process and capture the output and return value, but that sounds a bit heavy for just testing if your error handling still works as intended.
angarg12 4y ago
These kind of systems are not always appropriate, but when they do, they work wonderfully. Our use case was to build a service to manage the dynamic part of our infrastructure. These are infra pieces that are created/deleted/modified on the fly according to some policies, instead of defining them statically as code. The implementation is simply a lambda function that runs every minute, loads a policy, compares the current state of the system, and then creates/deletes/modify resources as needed. I am currently in the process of writing a talk that I will deliver to the rest of my org. This will help me crystalize my thoughts, but here are some pointer on why I think it worked in this case: * The service is stateless. On each run it just loads a policy, compares it to the current state of the system, and acts accordingly. This avoids handling complicated state or coordinating executions. In theory two policies could contradict each other, but in practice we partition our policies in such a way overlap is not possible. * Operations are idempotent. This is one of the reasons the system converges to a desired state. This makes the service resilient to both failures and eventual consistency. * Deviation from policies doesn't affect correctness. We are fortunate that our system is not directly customer facing. Deviation from policies affects only performance. The system can work several minutes (if not hours) outside policy band without consequences. This probably will be a critical blocker for most production systems. Other than control theory and let it fail, I had the chance to play with other cool concepts while implementing this service. These are some of those: * Parse, don't validate/anti-corruption layer: The service downloads and parses a policy at the beginning of each run. If parsing fails, it errors out. Otherwise, it passes the policy object to the rest of the execution. This makes the system easy to test, and avoids the anti-pattern of peppering your code with instructions reading input, just to find mid-execution that the policy was invalid. * Pluggable policies: The main body of the service is a very simple sense/act loop. For the actors, we use a strategy pattern, where policies can choose what strategy to use. This approach has helped us to introduce new behavior with minimal code changes. * Typescript as configuration language: This service replaces an older, less flexible one. A major pain of the old service is that policies where defined as Jinja templates over plain text files. This became unmaintainable as the number and complexity of policies grew. Our new service defines policies in Typescript. Policies are statically typed, and we use regular programming constructs (functions, loops, variables...) to build them at compile time. The output is still a plain JSON file. Hope that helps.
Too 4y ago
This is how Kubernetes works in many ways. Crash a pod and the control loop inside the ReplicaSet will create a new pod for you. Scaling nodes is based on similar principles of desired vs actual values.

fire 4y ago

this is the type of thing I'd love to see code / a post about implementing

tsimionescu 4y ago

This is exactly why I think all the discussions about the importance of error handling paths (and the aversion drive have to exceptions) are usually overblown. The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the error. If you don't, then exceptions are the best approximation of this pattern - much better than result types or error codes, which litter your code with irrelevant error handling details.

dfee 4y ago
Totally agree. Now, I’ve actually found functional programming - specifically, Either<Error, Value> - to be of great use in helping me focus only on the happy path.
- tsimionescu 4y ago
  Yes, with some kind of monadic-like control flow (either actual monads or even Rust's ? macro), those can be also achieve this workflow pretty well. Edit to add: I still think exceptions are better in practice, as you also get a stack trace when the failure happens, whereas Either and ? don't really help track down the error unless you add code to create a manual "stack trace".
  - rklaehn 4y ago
    In rust, you can use the excellent anyhow crate https://docs.rs/anyhow/latest/anyhow/ https://docs.rs/anyhow/latest/anyhow/ . It has various ways to add context to an error, and will automatically attach a stack trace with the backtrace feature. Explicit or implicit panic of course also attaches a backtrace. It can also be caught, although that is a can of worms. So panicing is the closest thing rust has to exceptions - somewhat similar to java.lang.Error on the jvm. https://docs.oracle.com/javase/7/docs/api/java/lang/Error.html https://docs.oracle.com/javase/7/docs/api/java/lang/Error.ht... With anyhow, error handling in rust really is quite pleasant.
4y ago

secondcoming 4y ago

> fails and logs What if your logging code was written with the same philosophy?

angarg12 4y ago
In another comment I mention that our service has some properties which make this a great solution. It doesn't always work, but when it does, it's awesome. To your point, Erlang has the concept of supervisor trees and handling of errors. Similarly, our supervisor is the Lambda runtime. If everything goes wrong, the runtime will emit a metric error. But what if Lambda itself fails? Or Cloudwatch? or any other supporting service? of course this is possible. But probably at that point the system is so fucked up that the fate of our little service is of little relevance. At least we know from past outages that when normal operations resume, the system will correct itself.