3 ms·

My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected ma

by solatic 2mo ago

My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write. Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better. I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.

dilyevsky 2mo ago
What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.
- solatic 2mo ago
  Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler. > basically just one for system and majority of serving workload ran on another priority and the rest was for batch RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
  - dilyevsky 2mo ago
    Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
    - solatic 2mo ago
      The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
      - dastbe 2mo ago
        are these separate services? Are you going to integrate how services preempt into overall system design? My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
        solatic 2mo ago
        > you want to move these decisions into the application tier as much as possible I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support. Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).

by solatic 2mo ago

dilyevsky 2mo ago

What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.

solatic 2mo ago
Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler. > basically just one for system and majority of serving workload ran on another priority and the rest was for batch RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
- dilyevsky 2mo ago
  Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
  - solatic 2mo ago
    The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
    - dastbe 2mo ago
      are these separate services? Are you going to integrate how services preempt into overall system design? My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
      - solatic 2mo ago
        > you want to move these decisions into the application tier as much as possible I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support. Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).