3 ms·

So here is what I don't get about this stuff. What happens to the in-flight requests? Don't a few users run into random errors whenever a host is killed unexpe

So here is what I don't get about this stuff. What happens to the in-flight requests? Don't a few users run into random errors whenever a host is killed unexpectedly? You could have your loadbalancer retry everything that fails, but then wouldn't every single request in your app have to be idempotent?

LoSboccacc 8y ago
well for example in our systems all api calls only moves from a know state to another known state and any call failure redirects the client/user to the dashboard trough an error handler so they have to reload the last good state saved on the database. not perfect, but having a server crash is not much different than having a connection reset by a wifi status change or an upload timing out due the mobile network going away or the user navigating away or closing the browser.
- Negitivefrags 8y ago
  It sounds like you are saying "The in-flight requests fail" to me. I really don't like the idea of saying that it's simply okay to give random users a bad user experience like that when you are actually killing servers yourself all the time.
  - dbaggerman 8y ago
    It's a different approach to managing risk -- minimizing impact of failure rather than minimizing the likelihood of failure. It's nice to know that you can kill a process and the only impact is that in-flight requests fail, rather than having a more significant outage if a process crashes and the failover doesn't work, or the process doesn't automatically restart, etc. If you accept that requests will fail you can build retries into the system. It's a lot harder to make a system more resilient if you avoid testing the failure scenarios.
    - lklig 8y ago
      Exactly! Chaos engineering is all about thoughtfully planned out experiments, to observe what the user experience will be when something fails. Doing this on your own terms allows you to improve the experience so that your customers aren't affected. You can decide what happens when an in-flight request is dropped, whether you hold onto the state somehow and retry or the client could fail gracefully with a relevant error message.
    - dkersten 8y ago
      Another thing that's not often caught by "normal" testing but that chaos engineering can capture is when multiple things fail together in random ways. It can be surprising how otherwise robust services can fail badly when multiple things go wrong at once.
  - zeckalpha 8y ago
    When you have nested service calls, a single downstream failure shouldn’t fail all the way back to the root request, in most cases.
perfmode 8y ago
Server crashes happen. This forces you to deal with them instead of pretending they won’t.
- Negitivefrags 8y ago
  Well yes, but I would suggest that they are uncommon enough that a few requests failing isn't a problem when those happen. It's an entirely different story when you are killing processes constantly.
  - perfmode 8y ago
    It doesn’t _need_ to be constantly. The important part is that it is done _deliberately_ to understand what happens when failures occur.
  - tylersmith 8y ago
    If "a few requests failing isn't a problem" is a reasonable statement in your company then this type of service isn't really aimed at you. The point is to help shake out fault tolerance issues. If you can already tolerate faults there isn't much of an issue.
flowardnut 8y ago
idempotent requests, stateless services, etc are all parts of a fault tolerant system. your service has a few ways to deal with a dependency going down -- maybe it's a retry, maybe it's opening a circuit breaker and returning a default payload instead of calling that service. It really depends on what specifically the service is and what it's calling (so it's a very case by case issue). One of the very neat features of istio is that you can do this tuning in real time -- spin up your services, simulate faults, and then test your service while tuning your retry logic to see what the best user experience is.

LoSboccacc 8y ago

well for example in our systems all api calls only moves from a know state to another known state and any call failure redirects the client/user to the dashboard trough an error handler so they have to reload the last good state saved on the database. not perfect, but having a server crash is not much different than having a connection reset by a wifi status change or an upload timing out due the mobile network going away or the user navigating away or closing the browser.

Negitivefrags 8y ago
It sounds like you are saying "The in-flight requests fail" to me. I really don't like the idea of saying that it's simply okay to give random users a bad user experience like that when you are actually killing servers yourself all the time.
- dbaggerman 8y ago
  It's a different approach to managing risk -- minimizing impact of failure rather than minimizing the likelihood of failure. It's nice to know that you can kill a process and the only impact is that in-flight requests fail, rather than having a more significant outage if a process crashes and the failover doesn't work, or the process doesn't automatically restart, etc. If you accept that requests will fail you can build retries into the system. It's a lot harder to make a system more resilient if you avoid testing the failure scenarios.
  - lklig 8y ago
    Exactly! Chaos engineering is all about thoughtfully planned out experiments, to observe what the user experience will be when something fails. Doing this on your own terms allows you to improve the experience so that your customers aren't affected. You can decide what happens when an in-flight request is dropped, whether you hold onto the state somehow and retry or the client could fail gracefully with a relevant error message.
  - dkersten 8y ago
    Another thing that's not often caught by "normal" testing but that chaos engineering can capture is when multiple things fail together in random ways. It can be surprising how otherwise robust services can fail badly when multiple things go wrong at once.
- zeckalpha 8y ago
  When you have nested service calls, a single downstream failure shouldn’t fail all the way back to the root request, in most cases.

perfmode 8y ago

Server crashes happen. This forces you to deal with them instead of pretending they won’t.

Negitivefrags 8y ago
Well yes, but I would suggest that they are uncommon enough that a few requests failing isn't a problem when those happen. It's an entirely different story when you are killing processes constantly.
- perfmode 8y ago
  It doesn’t _need_ to be constantly. The important part is that it is done _deliberately_ to understand what happens when failures occur.
- tylersmith 8y ago
  If "a few requests failing isn't a problem" is a reasonable statement in your company then this type of service isn't really aimed at you. The point is to help shake out fault tolerance issues. If you can already tolerate faults there isn't much of an issue.

flowardnut 8y ago

idempotent requests, stateless services, etc are all parts of a fault tolerant system. your service has a few ways to deal with a dependency going down -- maybe it's a retry, maybe it's opening a circuit breaker and returning a default payload instead of calling that service. It really depends on what specifically the service is and what it's calling (so it's a very case by case issue). One of the very neat features of istio is that you can do this tuning in real time -- spin up your services, simulate faults, and then test your service while tuning your retry logic to see what the best user experience is.