5 ms·

What is that nonsense? I literally restart apps in a rolling fashion on VMs all the time, no Kubernetes needed and no downtime in sight. This includes a full re

What is that nonsense? I literally restart apps in a rolling fashion on VMs all the time, no Kubernetes needed and no downtime in sight. This includes a full reboot of the VM itself.

anamexis 5y ago
They are talking about restarting the single server that all of your VMs are running on.
- bsagdiyev 5y ago
  Which can also be done with zero downtime to the VM.
  - panzagl 5y ago
    It's just like flying- throw yourself at the ground and miss...
    - bsagdiyev 5y ago
      I responded above, either I missed the word "single" or it's been edited (but doesn't show it as being edited?) -- live migration between hypervisors is a thing.
  - RamRodification 5y ago
    How do you restart a single server hosting VMs with zero downtime to the VMs?
    - bsagdiyev 5y ago
      Did I misunderstand that the original post was regarding being able to do it with Kubernetes, not a few physical hosts? I didn't see "single" in the comment when I originally responded.
      - RamRodification 5y ago
        Yeah, Top comment says "You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server." And the comment you replied to is referring to "the server"
    - cyberpunk 5y ago
      VMware can do it by live migrating the vm, you will incur a short pause though, and the networking is a bit tricky to setup.. This of course doesn't happen during an unexpected downtime, it's a cold boot on another node in that case.
      - bpodgursky 5y ago
        I have never seen this go smoothly on a production server; it's always WAY slower than expected (if you use any significant amount of memory) and something always gets f'd up wrt the network connectivity, broken caches, etc.
        dekhn 5y ago
        if I understand correctly, this is happening (nearly) transparently on GCP all the time. https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati... It involves copying the whole VM image over, and rewiring the network connections virtually on the fly. I say (nearly) because it's not 100% transparent, but my understanding is that it works properly the vast majority of time.
        bpodgursky 5y ago
        Probably true but I'm also pretty sure Google isn't using VMware for live migrations under the good.
      - ted_dunning 5y ago
        If you migrate a VM to a second hardware server then you have, by definition, a second server. The question was how you reboot the posited singular hardware server with no downtime to any VMs running on it.
      - ryan_lane 5y ago
        And what do you do when you need to restart the actual VM itself? You know, for when you need to patch the kernel and such...
    - hedora 5y ago
      VMWare has had a high availability mode for over a decade. It keeps checkpoints of system state elsewhere and synchronously replicates them. If the primary catches fire or crashes, the secondary boots the VM quickly without data loss. If the primary reboots, it checkpoints RAM to the secondary pauses the VM and unpauses it on the secondary a few milliseconds later. Note that this transparently handles storage replication, which is something that is notoriously difficult in Kubernetes. If your cluster fits on one machine, you've paid for a lot of (currently) unnecessary complexity up front, both in dev time and in hardware cost. If your app scales to need the cluster, congrats. Sometimes delaying time to market to allow for a smoother ramp makes sense. Sometimes it does not. (Wikipedia is a good example of succeeding without ever needing to scale out the back end. I doubt they'd have won out over their competition if they took an extra 12-24 months to launch.)
      - sockpuppet69 5y ago
        So not 0 down time then
      - felipelemos 5y ago
        > VMWare has had a high availability mode for over a decade. It keeps checkpoints of system state elsewhere and synchronously replicates them. So you don't have one VM's, but two. In case of Op's scenario you are paying for 196 CPUS instead of 98. > If the primary reboots, it checkpoints RAM to the secondary pauses the VM and unpauses it on the secondary a few milliseconds later. So a very small downtime, but not 0.
        hedora 5y ago
        It depends on your SLA. Under 10ms 99.9% of the time is pretty tight. Even if the migration takes a second, you'll meet your SLA for the hour.
  - mountainriver 5y ago
    Also curious how this is possible
    - bsagdiyev 5y ago
      I don't recall seeing "single" in the comment I replied to. You can live migrate VMs between hypervisors. Either I missed it or it's a dirty edit.
      - anamexis 5y ago
        This is the relevant part of the root comment: > You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server.
koffiezet 5y ago
Can you do that without knowing what apps are running on them? Your entire platform say "do a rolling update" and don't look back? I wouldn't want to have paid for the engineering that went into that if you could actually do that.
- bsagdiyev 5y ago
  If things are architected to be updated in a rolling fashion. If it's some single node app you're screwed. But we're talking about K8s so I would hope you're using it for its purpose and not some tiny cluster for a 3 request per minute app.
- secondcoming 5y ago
  > Your entire platform say "do a rolling update" and don't look back? Yes. GCP literally has a button that allows you to Replace/Restart all the instances in an Instance Group. Our deployment system just pushes a new GCP template and GCP does all the work rolling it out. No Kubernetes.
  - koffiezet 5y ago
    Yes there are other solutions out-there that have been pre-engineered, but he was talking about VM's and apps and rebooting them. Doesn't sound like instance groups to me.
- dosethree 5y ago
  for stateless apps (with HA, capacity etc) its easy
- hedora 5y ago
  Yes. You can do this without knowing about or modifying the applications. There was a big push for hardware consolidation a decade or so ago, where legacy apps were crammed on to fewer and fewer machines, and were cheaper/faster/more reliable as a result. However, a permanent machine failure or unexpected reboot implies a few seconds or minutes of downtime. These systems usually use a local disk, a synchronously updated disk across town (low latency, but on a different power grid, outside most natural disaster blast radii), and a far away asynchronously updated disaster recovery disk. The disaster recovery disk provides a crash-consistent state that's less than a few seconds out of date. These days, the "disks" tend to be deduped and compressed SSD's that use parity encoded raid. Their hardware cost is lower than a single copy on ext4, but they come with an enterprise tax / support contract, etc. In practice, it's durable unless corporate HQ is wiped out. If that happens, it won't be the weak link in your business continuity plan (but a few customers' orders might've been dropped). This stuff is tremendously unsexy, but it'll plug along just fine until you have some workload that can't be partitioned, and that needs more than roughly 40-100gbit of network/disk bandwidth, 128 cores, or 1TB of DRAM. (Edit: Forgot to mention that the the apps run inside a VM infrastructure that supports live migration.)

What is that nonsense? I literally restart apps in a rolling fashion on VMs all the time, no Kubernetes needed and no downtime in sight. This includes a full reboot of the VM itself.

anamexis 5y ago

They are talking about restarting the single server that all of your VMs are running on.

bsagdiyev 5y ago
Which can also be done with zero downtime to the VM.
- panzagl 5y ago
  It's just like flying- throw yourself at the ground and miss...
  - bsagdiyev 5y ago
    I responded above, either I missed the word "single" or it's been edited (but doesn't show it as being edited?) -- live migration between hypervisors is a thing.
- RamRodification 5y ago
  How do you restart a single server hosting VMs with zero downtime to the VMs?
  - bsagdiyev 5y ago
    Did I misunderstand that the original post was regarding being able to do it with Kubernetes, not a few physical hosts? I didn't see "single" in the comment when I originally responded.
    - RamRodification 5y ago
      Yeah, Top comment says "You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server." And the comment you replied to is referring to "the server"
  - cyberpunk 5y ago
    VMware can do it by live migrating the vm, you will incur a short pause though, and the networking is a bit tricky to setup.. This of course doesn't happen during an unexpected downtime, it's a cold boot on another node in that case.
    - bpodgursky 5y ago
      I have never seen this go smoothly on a production server; it's always WAY slower than expected (if you use any significant amount of memory) and something always gets f'd up wrt the network connectivity, broken caches, etc.
      - dekhn 5y ago
        if I understand correctly, this is happening (nearly) transparently on GCP all the time. https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati... It involves copying the whole VM image over, and rewiring the network connections virtually on the fly. I say (nearly) because it's not 100% transparent, but my understanding is that it works properly the vast majority of time.
        bpodgursky 5y ago
        Probably true but I'm also pretty sure Google isn't using VMware for live migrations under the good.
    - ted_dunning 5y ago
      If you migrate a VM to a second hardware server then you have, by definition, a second server. The question was how you reboot the posited singular hardware server with no downtime to any VMs running on it.
    - ryan_lane 5y ago
      And what do you do when you need to restart the actual VM itself? You know, for when you need to patch the kernel and such...
  - hedora 5y ago
    VMWare has had a high availability mode for over a decade. It keeps checkpoints of system state elsewhere and synchronously replicates them. If the primary catches fire or crashes, the secondary boots the VM quickly without data loss. If the primary reboots, it checkpoints RAM to the secondary pauses the VM and unpauses it on the secondary a few milliseconds later. Note that this transparently handles storage replication, which is something that is notoriously difficult in Kubernetes. If your cluster fits on one machine, you've paid for a lot of (currently) unnecessary complexity up front, both in dev time and in hardware cost. If your app scales to need the cluster, congrats. Sometimes delaying time to market to allow for a smoother ramp makes sense. Sometimes it does not. (Wikipedia is a good example of succeeding without ever needing to scale out the back end. I doubt they'd have won out over their competition if they took an extra 12-24 months to launch.)
    - sockpuppet69 5y ago
      So not 0 down time then
    - felipelemos 5y ago
      > VMWare has had a high availability mode for over a decade. It keeps checkpoints of system state elsewhere and synchronously replicates them. So you don't have one VM's, but two. In case of Op's scenario you are paying for 196 CPUS instead of 98. > If the primary reboots, it checkpoints RAM to the secondary pauses the VM and unpauses it on the secondary a few milliseconds later. So a very small downtime, but not 0.
      - hedora 5y ago
        It depends on your SLA. Under 10ms 99.9% of the time is pretty tight. Even if the migration takes a second, you'll meet your SLA for the hour.
- mountainriver 5y ago
  Also curious how this is possible
  - bsagdiyev 5y ago
    I don't recall seeing "single" in the comment I replied to. You can live migrate VMs between hypervisors. Either I missed it or it's a dirty edit.
    - anamexis 5y ago
      This is the relevant part of the root comment: > You can easily get a box that has this many (or more) cores. I wouldn't be surprised if our cloud provider analyzed the topology of the connections, and decided to put the whole thing on the same server.

koffiezet 5y ago

Can you do that without knowing what apps are running on them? Your entire platform say "do a rolling update" and don't look back? I wouldn't want to have paid for the engineering that went into that if you could actually do that.

bsagdiyev 5y ago
If things are architected to be updated in a rolling fashion. If it's some single node app you're screwed. But we're talking about K8s so I would hope you're using it for its purpose and not some tiny cluster for a 3 request per minute app.
secondcoming 5y ago
> Your entire platform say "do a rolling update" and don't look back? Yes. GCP literally has a button that allows you to Replace/Restart all the instances in an Instance Group. Our deployment system just pushes a new GCP template and GCP does all the work rolling it out. No Kubernetes.
- koffiezet 5y ago
  Yes there are other solutions out-there that have been pre-engineered, but he was talking about VM's and apps and rebooting them. Doesn't sound like instance groups to me.
dosethree 5y ago
for stateless apps (with HA, capacity etc) its easy
hedora 5y ago
Yes. You can do this without knowing about or modifying the applications. There was a big push for hardware consolidation a decade or so ago, where legacy apps were crammed on to fewer and fewer machines, and were cheaper/faster/more reliable as a result. However, a permanent machine failure or unexpected reboot implies a few seconds or minutes of downtime. These systems usually use a local disk, a synchronously updated disk across town (low latency, but on a different power grid, outside most natural disaster blast radii), and a far away asynchronously updated disaster recovery disk. The disaster recovery disk provides a crash-consistent state that's less than a few seconds out of date. These days, the "disks" tend to be deduped and compressed SSD's that use parity encoded raid. Their hardware cost is lower than a single copy on ext4, but they come with an enterprise tax / support contract, etc. In practice, it's durable unless corporate HQ is wiped out. If that happens, it won't be the weak link in your business continuity plan (but a few customers' orders might've been dropped). This stuff is tremendously unsexy, but it'll plug along just fine until you have some workload that can't be partitioned, and that needs more than roughly 40-100gbit of network/disk bandwidth, 128 cores, or 1TB of DRAM. (Edit: Forgot to mention that the the apps run inside a VM infrastructure that supports live migration.)