4 ms·

same here. Really interested to read about the popular ways of screwing a k8s cluster

by bsaul 5y ago

stephenr 5y ago
I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
- robertlagrant 5y ago
  Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
  - stephenr 5y ago
    Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
    - robertlagrant 5y ago
      We do it by having services auto-upgrade data migrations, migrations only take a few seconds at most, services maintain old endpoints, newer versions of which will work on new or old database structures (removed in future versions) and doing them at a time that will disrupt basically no one. That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
      - stephenr 5y ago
        My take on that is that you’re creating work in one place to avoid it in another. And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
        robertlagrant 5y ago
        > I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
        stephenr 5y ago
        No there are definitely alternatives, but IME having a maintenance page is a good idea/necessity anyway, and the engineering involved to activate it during deployments is essentially set and forget, whereas the engineering to make your application backwards compatible with any possible schema change is entirely an ongoing issue, and a lot more complex. If your development team is sufficiently experienced to do allow for the no-downtime approach that’s a big win, but my experience so far makes me pessimistic about how widespread that kind of experience is. Edit: I should clarify somewhat: I’ve had to introduce at least one of the following concepts at basically every client: continuous/automatic deployments; automatic migrations; automatic maintenance pages during outages (planned or otherwise); and hard as it is to believe: version control. My clients are typically smallish, so anything I can do to solve problems/provide solutions at the infra layer is a no-brainer because setting up an automated {enable maintenance mode; apply migrations; disable maintenance mode} is a lot more reliable long term than trying to teach every possible developer how to handle backwards compatible models - particularly when the clients application language of choice is something I’m not necessarily experienced in.
dijit 5y ago
You said "read" but there is at least this talk by the Datadog team: https://www.youtube.com/watch?v=2dsCwp_j0yQ https://www.youtube.com/watch?v=2dsCwp_j0yQ

same here. Really interested to read about the popular ways of screwing a k8s cluster

by bsaul 5y ago

same here. Really interested to read about the popular ways of screwing a k8s cluster

I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.

robertlagrant 5y ago
Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
- stephenr 5y ago
  Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
  - robertlagrant 5y ago
    We do it by having services auto-upgrade data migrations, migrations only take a few seconds at most, services maintain old endpoints, newer versions of which will work on new or old database structures (removed in future versions) and doing them at a time that will disrupt basically no one. That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
    - stephenr 5y ago
      My take on that is that you’re creating work in one place to avoid it in another. And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
      - robertlagrant 5y ago
        > I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
        stephenr 5y ago
        No there are definitely alternatives, but IME having a maintenance page is a good idea/necessity anyway, and the engineering involved to activate it during deployments is essentially set and forget, whereas the engineering to make your application backwards compatible with any possible schema change is entirely an ongoing issue, and a lot more complex. If your development team is sufficiently experienced to do allow for the no-downtime approach that’s a big win, but my experience so far makes me pessimistic about how widespread that kind of experience is. Edit: I should clarify somewhat: I’ve had to introduce at least one of the following concepts at basically every client: continuous/automatic deployments; automatic migrations; automatic maintenance pages during outages (planned or otherwise); and hard as it is to believe: version control. My clients are typically smallish, so anything I can do to solve problems/provide solutions at the infra layer is a no-brainer because setting up an automated {enable maintenance mode; apply migrations; disable maintenance mode} is a lot more reliable long term than trying to teach every possible developer how to handle backwards compatible models - particularly when the clients application language of choice is something I’m not necessarily experienced in.

dijit 5y ago

You said "read" but there is at least this talk by the Datadog team: https://www.youtube.com/watch?v=2dsCwp_j0yQ https://www.youtube.com/watch?v=2dsCwp_j0yQ