5 ms·

Why is ssh even needed on distributed clusters? Shouldn't provisioning the cluster be automated and the nodes be immutable by design? I can only imagine what a

by moondev 10y ago

Why is ssh even needed on distributed clusters? Shouldn't provisioning the cluster be automated and the nodes be immutable by design? I can only imagine what a nightmare a huge fleet of special sniwflake machines woukd be to manage. Cattle not pets

deleted 10y ago
[deleted]
vasco 10y ago
This is a bit naive. If you've ran a production system with any decent traffic and never needed to SSH into machines, congrats. I haven't and I don't know anyone who has. You might need to go in for anything from auditing to troubleshooting, even if it's rare. PS: How is your automated provisioning system reaching your cluster if not by SSH?
- colemickens 10y ago
  >PS: How is your automated provisioning system reaching your cluster if not by SSH? Not sure about moondev, but Terraform + Cloud-init + Container Orchestrator means that I basically never need to SSH into my nodes, except in extreme/rare circumstances.
  - vacri 10y ago
    > except in extreme/rare circumstances. So you do need ssh after all?
    - colemickens 10y ago
      I said "I basically never need to". Not that I never need to. But yeah, I basically never, ever need to. Short of needing to take a coredump or docker shits the bed, I don't really ever need to log into my nodes. I guess that's an offensive thing to point out judging by my score...
      - johnbellone 10y ago
        So you need SSH.
        colemickens 10y ago
        Are you kidding me? >I said "I basically never need to". Not that I never need to. But yeah, I basically never, ever need to It is interesting to imagine why this innocuous comment has ruffled so many feathers.
        colemickens 10y ago
        Boy, I sure hope I'm helping someone really sad feel better about themselves. Downvote away, it's not going to change a damn thing.
  - Thaxll 10y ago
    Except when you need to troubleshoot complex issues.
  - deleted 10y ago
    [deleted]
  - moondev 10y ago
    I do love me some terraform. Mainly use spinnaker these days
  - deleted 10y ago
    [deleted]
- SysArchitect 10y ago
  OpenStack + cloud-init + SaltStack
  - chetanahuja 10y ago
    "SaltStack" Saltstack is either using SSH to communicate or opening it's own port. I'd much rather trust an open ssh port for securely provisioning/management than allow any other piece of software to keep a port open (upto and including TLS based protocols).
    - SysArchitect 10y ago
      SaltStack has an agent that communicates with a master on a different server. The agents on the clients don't need an open port (other than egress). This allows me to have one central server that is well secured and protected that allows ingress from the remote hosts, and then all the clients reach out to the master to get their tasks.
- moondev 10y ago
  logging and metrics are sent elsewhere to be consumed and queried. I build machine images with packer that get provisioned during the deployment pipeline. That single machine is then put into a cluster with x number of copies. If one dies I don't care, the cluster provisions another automatically.
  - Sanddancer 10y ago
    How do you debug things when you discover that machines are randomly dropping logs for x minutes?
cakeface 10y ago
Direct SSH access can be an invaluable tool for debugging production issues. It is great to be able to SSH in and check the tcp dump, view the running processes in detail, attach gdb and get a memory dump. These types of tasks will never go away. We have great leaps in orchestrating remote application management but when you get down to issues at the bottom of your stack you'll always need to directly access the machine. I like to tell my developers that "the stack goes all the way down". A bug in Linux networking or even a CPU error is still a bug.
- mitchty 10y ago
  Yep, sometimes you have to debug things live with things like perf/gdb echo c > /proc/sysrq_trigger. If not, well congratulations, but if I got told I can't get the above to debug things I'd start question why this "pet" infrastructure lacks basic debugging ability.
- moondev 10y ago
  Smart health checks and logging should take care of that and remove the instance automatically. You can also spin up a canary machine to "live" debug. I'm referring to distributed clusters not a single machine taking on all the traffic.
  - icebraining 10y ago
    Smart health checks and logging should take care of that and remove the instance automatically. What if the bug is corrupting data or sending incorrect results to the client? Even if you detect and kill the instance, you still have a problem to fix. And even if you can cleanly kill the instance and redirect the request, you can't avoid the latency hit from having to re-process it. You can also spin up a canary machine to "live" debug. You can reproduce the code and data, but how can you reproduce the exact state? You can't log everything that happens in a machine.
amalcon 10y ago
Let's say you have a thousand machines sitting around, doing whatever, and suddenly you notice one of them acting strangely. Maybe it's just a little slower than the others, maybe it's crashing, maybe your monitoring system can't pick it up, maybe it's even producing incorrect outputs. How do you tell if it's: A) A hardware issue (thus requiring hardware maintenance) B) A software issue triggered by the particular workload of this machine (thus requiring a software change) C) A network issue (thus requiring network maintenance, possibly phone calls to ISPs, etc) D) Something else (thus requiring who knows what) The most normal way in the real world is, have a sysadmin and a dev sit down together, SSH into the machine, and poke things until they arrive at the root of the problem (possibly with additional process involved if it's a really big system). Then, through normal (non SSH) processes, perform the required maintenance and update your monitoring system so that it identifies problems of that class.
- moondev 10y ago
  E) Your health checks notice the machine is acting up and it is removed from the cluster and another provisioned. In a distributed environment all of your nodes should be designed to be immutable and stateless. That's the advantage of running them at scale horizontally
  - dsr_ 10y ago
    When all of your nodes are immutable and stateless, you have a really tough time doing logging, new account signup, and storing any information at all on behalf of your customers. So not all of the nodes can be immutable and stateless.
    - microcolonel 10y ago
      In fact, there is no such thing as an immutable or stateless computer; it violates the theory of computing. Just a bunch of buzzword nonsense people say so that they can justify spending their entire life asking for too much money to try making the machine behave as though it is stateless. With real, physical machines; you could have a cooling issue (which you should fix), a manufacturing defect (which you should RMA), a networking issue (which you should fix). All of these can easily be unique to an instance, and all of them need analytical attention; especially when you're paying for all of those machines "horizontally" and "at scale".
sciurus 10y ago
Even Netflix, the poster child for this style of managing systems, allows engineers to SSH in to machines. In fact, their system and Facebook's system for performing SSH auth were discussed on HN recently. https://news.ycombinator.com/item?id=11746425 https://news.ycombinator.com/item?id=11746425 https://news.ycombinator.com/item?id=12482212 https://news.ycombinator.com/item?id=12482212
mdeeks 10y ago
I firmly believe No-SSH is a goal you should always strive for but never actually achieve. There are always cases where you need to do really detailed troubleshooting that requires things like tcpdump, a debugger, or even running lsof (or other expensive command that you can't afford to run regularly and log). Some random user with some edge case will put one of your nodes for your service into a bad state. Throwing away the machine will just lose the state and push the user somewhere else. Also centralized log and metrics stores get real big and expensive really fast. Sometimes you simply can't afford to ship everything to it. So you'll find yourself putting detailed debug logs on the "free" ephemeral/local disks while info logs go to your central store.
imsofuture 10y ago
Provisioning the cluster should of course be automated. Does that mean you'll never need to ever access a node for any reason? Of course not.

by moondev 10y ago

deleted 10y ago

[deleted]

vasco 10y ago

This is a bit naive. If you've ran a production system with any decent traffic and never needed to SSH into machines, congrats. I haven't and I don't know anyone who has. You might need to go in for anything from auditing to troubleshooting, even if it's rare. PS: How is your automated provisioning system reaching your cluster if not by SSH?

colemickens 10y ago
>PS: How is your automated provisioning system reaching your cluster if not by SSH? Not sure about moondev, but Terraform + Cloud-init + Container Orchestrator means that I basically never need to SSH into my nodes, except in extreme/rare circumstances.
- vacri 10y ago
  > except in extreme/rare circumstances. So you do need ssh after all?
  - colemickens 10y ago
    I said "I basically never need to". Not that I never need to. But yeah, I basically never, ever need to. Short of needing to take a coredump or docker shits the bed, I don't really ever need to log into my nodes. I guess that's an offensive thing to point out judging by my score...
    - johnbellone 10y ago
      So you need SSH.
      - colemickens 10y ago
        Are you kidding me? >I said "I basically never need to". Not that I never need to. But yeah, I basically never, ever need to It is interesting to imagine why this innocuous comment has ruffled so many feathers.
        colemickens 10y ago
        Boy, I sure hope I'm helping someone really sad feel better about themselves. Downvote away, it's not going to change a damn thing.
- Thaxll 10y ago
  Except when you need to troubleshoot complex issues.
- deleted 10y ago
  [deleted]
- moondev 10y ago
  I do love me some terraform. Mainly use spinnaker these days
- deleted 10y ago
  [deleted]
SysArchitect 10y ago
OpenStack + cloud-init + SaltStack
- chetanahuja 10y ago
  "SaltStack" Saltstack is either using SSH to communicate or opening it's own port. I'd much rather trust an open ssh port for securely provisioning/management than allow any other piece of software to keep a port open (upto and including TLS based protocols).
  - SysArchitect 10y ago
    SaltStack has an agent that communicates with a master on a different server. The agents on the clients don't need an open port (other than egress). This allows me to have one central server that is well secured and protected that allows ingress from the remote hosts, and then all the clients reach out to the master to get their tasks.
moondev 10y ago
logging and metrics are sent elsewhere to be consumed and queried. I build machine images with packer that get provisioned during the deployment pipeline. That single machine is then put into a cluster with x number of copies. If one dies I don't care, the cluster provisions another automatically.
- Sanddancer 10y ago
  How do you debug things when you discover that machines are randomly dropping logs for x minutes?

cakeface 10y ago

Direct SSH access can be an invaluable tool for debugging production issues. It is great to be able to SSH in and check the tcp dump, view the running processes in detail, attach gdb and get a memory dump. These types of tasks will never go away. We have great leaps in orchestrating remote application management but when you get down to issues at the bottom of your stack you'll always need to directly access the machine. I like to tell my developers that "the stack goes all the way down". A bug in Linux networking or even a CPU error is still a bug.

mitchty 10y ago
Yep, sometimes you have to debug things live with things like perf/gdb echo c > /proc/sysrq_trigger. If not, well congratulations, but if I got told I can't get the above to debug things I'd start question why this "pet" infrastructure lacks basic debugging ability.
moondev 10y ago
Smart health checks and logging should take care of that and remove the instance automatically. You can also spin up a canary machine to "live" debug. I'm referring to distributed clusters not a single machine taking on all the traffic.
- icebraining 10y ago
  Smart health checks and logging should take care of that and remove the instance automatically. What if the bug is corrupting data or sending incorrect results to the client? Even if you detect and kill the instance, you still have a problem to fix. And even if you can cleanly kill the instance and redirect the request, you can't avoid the latency hit from having to re-process it. You can also spin up a canary machine to "live" debug. You can reproduce the code and data, but how can you reproduce the exact state? You can't log everything that happens in a machine.

amalcon 10y ago

Let's say you have a thousand machines sitting around, doing whatever, and suddenly you notice one of them acting strangely. Maybe it's just a little slower than the others, maybe it's crashing, maybe your monitoring system can't pick it up, maybe it's even producing incorrect outputs. How do you tell if it's: A) A hardware issue (thus requiring hardware maintenance) B) A software issue triggered by the particular workload of this machine (thus requiring a software change) C) A network issue (thus requiring network maintenance, possibly phone calls to ISPs, etc) D) Something else (thus requiring who knows what) The most normal way in the real world is, have a sysadmin and a dev sit down together, SSH into the machine, and poke things until they arrive at the root of the problem (possibly with additional process involved if it's a really big system). Then, through normal (non SSH) processes, perform the required maintenance and update your monitoring system so that it identifies problems of that class.

moondev 10y ago
E) Your health checks notice the machine is acting up and it is removed from the cluster and another provisioned. In a distributed environment all of your nodes should be designed to be immutable and stateless. That's the advantage of running them at scale horizontally
- dsr_ 10y ago
  When all of your nodes are immutable and stateless, you have a really tough time doing logging, new account signup, and storing any information at all on behalf of your customers. So not all of the nodes can be immutable and stateless.
  - microcolonel 10y ago
    In fact, there is no such thing as an immutable or stateless computer; it violates the theory of computing. Just a bunch of buzzword nonsense people say so that they can justify spending their entire life asking for too much money to try making the machine behave as though it is stateless. With real, physical machines; you could have a cooling issue (which you should fix), a manufacturing defect (which you should RMA), a networking issue (which you should fix). All of these can easily be unique to an instance, and all of them need analytical attention; especially when you're paying for all of those machines "horizontally" and "at scale".

sciurus 10y ago

Even Netflix, the poster child for this style of managing systems, allows engineers to SSH in to machines. In fact, their system and Facebook's system for performing SSH auth were discussed on HN recently. https://news.ycombinator.com/item?id=11746425 https://news.ycombinator.com/item?id=11746425 https://news.ycombinator.com/item?id=12482212 https://news.ycombinator.com/item?id=12482212

mdeeks 10y ago

I firmly believe No-SSH is a goal you should always strive for but never actually achieve. There are always cases where you need to do really detailed troubleshooting that requires things like tcpdump, a debugger, or even running lsof (or other expensive command that you can't afford to run regularly and log). Some random user with some edge case will put one of your nodes for your service into a bad state. Throwing away the machine will just lose the state and push the user somewhere else. Also centralized log and metrics stores get real big and expensive really fast. Sometimes you simply can't afford to ship everything to it. So you'll find yourself putting detailed debug logs on the "free" ephemeral/local disks while info logs go to your central store.

imsofuture 10y ago

Provisioning the cluster should of course be automated. Does that mean you'll never need to ever access a node for any reason? Of course not.