5 ms·

E) Your health checks notice the machine is acting up and it is removed from the cluster and another provisioned. In a distributed environment all of your nodes

by moondev 10y ago

E) Your health checks notice the machine is acting up and it is removed from the cluster and another provisioned. In a distributed environment all of your nodes should be designed to be immutable and stateless. That's the advantage of running them at scale horizontally

dsr_ 10y ago
When all of your nodes are immutable and stateless, you have a really tough time doing logging, new account signup, and storing any information at all on behalf of your customers. So not all of the nodes can be immutable and stateless.
- microcolonel 10y ago
  In fact, there is no such thing as an immutable or stateless computer; it violates the theory of computing. Just a bunch of buzzword nonsense people say so that they can justify spending their entire life asking for too much money to try making the machine behave as though it is stateless. With real, physical machines; you could have a cooling issue (which you should fix), a manufacturing defect (which you should RMA), a networking issue (which you should fix). All of these can easily be unique to an instance, and all of them need analytical attention; especially when you're paying for all of those machines "horizontally" and "at scale".
  - amalcon 10y ago
    I mean, you can have an immutable stateless computer, but it's just called a "boolean logic circuit". You give it an input, it gives you an output, that's all. In practice, you'll of course actually be using a computer with RAM. Most likely, it will even have a Von Neumann architecture. That leaves you open to everything from race conditions to hardware failures to cosmic rays putting a machine into a bad state.
    - deleted 10y ago
      [deleted]
  - drdrey 10y ago
    Stateless depends on the context, it doesn't violate any theory of computing. Here are some elements of stateless systems: * all the state needed for serving requests is transient, not something that needs to be persisted * any particular instance can disappear without affecting the observable behavior of the system * the system is infinitely scalable: it doesn't matter if the cluster is comprised of 1, 100 or 1000 instances
  - BraveNewCurency 10y ago
    > In fact, there is no such thing as an immutable or stateless computer Nobody calls the computer immutable/stateless. Immutable/stateless is an architecture pattern, implying that nothing important is stored or shared between calls. It doesn't say that you can't cache info or write log files, etc. > Just a bunch of buzzword nonsense Well, what's the alternative that you propose? Should my HTTP requests leave lots of state-full files laying around? Is it easier to manage deployments when every box has different files laying around depending on how old it is? Also, remember these techniques are for people "at scale". If you only have 1 or 2 boxes, these techniques may not be useful to you, so feel free to ignore them. I've got a 1000 boxes, and have seen first hand why it's a bad idea for each box to be able to be a unique snowflake. Even if you have physical machines, you shouldn't put your production image on it to debug it -- you should put a debug image on it that allows SSH. If it's a hardware problem, it will be easy to find. If it's a software problem -- well, you have 1000 other boxes running that software, so you'll see it again. Just add better logging.
serge2k 10y ago
Then you notice you're churning through machines pretty often and start to wonder if maybe you should actually fix that bug instead of just ignoring it. We aren't talking about using SSH to go in and fix a machine so things stay up. It's about figuring out why something is happening.
amalcon 10y ago
So your plan is to buy a new machine as a result of a software issue, a network issue, or another problem that has nothing to do with the hardware like a corrupted image? When you have 100 machines, you can probably afford to do this. It would be silly, because when you only have 100 machines, most of your problems will be software or network issues. Still, you can do it and it probably won't cost you a kidney. When you have ten thousand, you can't. And this is even assuming that your monitoring system notices the problem. All too frequently, you'll notice the problem in some other way. Monitoring systems are designed to detect the things you knew might happen, and problems similar to those. There will be blind spots, it's just unavoidable.
niij 10y ago
THROW MORE BUZZWORDS AT IT

by moondev 10y ago

dsr_ 10y ago

When all of your nodes are immutable and stateless, you have a really tough time doing logging, new account signup, and storing any information at all on behalf of your customers. So not all of the nodes can be immutable and stateless.

microcolonel 10y ago
In fact, there is no such thing as an immutable or stateless computer; it violates the theory of computing. Just a bunch of buzzword nonsense people say so that they can justify spending their entire life asking for too much money to try making the machine behave as though it is stateless. With real, physical machines; you could have a cooling issue (which you should fix), a manufacturing defect (which you should RMA), a networking issue (which you should fix). All of these can easily be unique to an instance, and all of them need analytical attention; especially when you're paying for all of those machines "horizontally" and "at scale".
- amalcon 10y ago
  I mean, you can have an immutable stateless computer, but it's just called a "boolean logic circuit". You give it an input, it gives you an output, that's all. In practice, you'll of course actually be using a computer with RAM. Most likely, it will even have a Von Neumann architecture. That leaves you open to everything from race conditions to hardware failures to cosmic rays putting a machine into a bad state.
  - deleted 10y ago
    [deleted]
- drdrey 10y ago
  Stateless depends on the context, it doesn't violate any theory of computing. Here are some elements of stateless systems: * all the state needed for serving requests is transient, not something that needs to be persisted * any particular instance can disappear without affecting the observable behavior of the system * the system is infinitely scalable: it doesn't matter if the cluster is comprised of 1, 100 or 1000 instances
- BraveNewCurency 10y ago
  > In fact, there is no such thing as an immutable or stateless computer Nobody calls the computer immutable/stateless. Immutable/stateless is an architecture pattern, implying that nothing important is stored or shared between calls. It doesn't say that you can't cache info or write log files, etc. > Just a bunch of buzzword nonsense Well, what's the alternative that you propose? Should my HTTP requests leave lots of state-full files laying around? Is it easier to manage deployments when every box has different files laying around depending on how old it is? Also, remember these techniques are for people "at scale". If you only have 1 or 2 boxes, these techniques may not be useful to you, so feel free to ignore them. I've got a 1000 boxes, and have seen first hand why it's a bad idea for each box to be able to be a unique snowflake. Even if you have physical machines, you shouldn't put your production image on it to debug it -- you should put a debug image on it that allows SSH. If it's a hardware problem, it will be easy to find. If it's a software problem -- well, you have 1000 other boxes running that software, so you'll see it again. Just add better logging.

serge2k 10y ago

Then you notice you're churning through machines pretty often and start to wonder if maybe you should actually fix that bug instead of just ignoring it. We aren't talking about using SSH to go in and fix a machine so things stay up. It's about figuring out why something is happening.

amalcon 10y ago

So your plan is to buy a new machine as a result of a software issue, a network issue, or another problem that has nothing to do with the hardware like a corrupted image? When you have 100 machines, you can probably afford to do this. It would be silly, because when you only have 100 machines, most of your problems will be software or network issues. Still, you can do it and it probably won't cost you a kidney. When you have ten thousand, you can't. And this is even assuming that your monitoring system notices the problem. All too frequently, you'll notice the problem in some other way. Monitoring systems are designed to detect the things you knew might happen, and problems similar to those. There will be blind spots, it's just unavoidable.

niij 10y ago

THROW MORE BUZZWORDS AT IT