4 ms·

TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who kno

TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.

robertlagrant 5y ago
Be cool to hear some examples/war stories. Do you have a blog?
- bsaul 5y ago
  same here. Really interested to read about the popular ways of screwing a k8s cluster
  - stephenr 5y ago
    I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
    - robertlagrant 5y ago
      Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
      - stephenr 5y ago
        Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
        robertlagrant 5y ago
        We do it by having services auto-upgrade data migrations, migrations only take a few seconds at most, services maintain old endpoints, newer versions of which will work on new or old database structures (removed in future versions) and doing them at a time that will disrupt basically no one. That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
        stephenr 5y ago
        My take on that is that you’re creating work in one place to avoid it in another. And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
        robertlagrant 5y ago
        > I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
        stephenr 5y ago
        No there are definitely alternatives, but IME having a maintenance page is a good idea/necessity anyway, and the engineering involved to activate it during deployments is essentially set and forget, whereas the engineering to make your application backwards compatible with any possible schema change is entirely an ongoing issue, and a lot more complex. If your development team is sufficiently experienced to do allow for the no-downtime approach that’s a big win, but my experience so far makes me pessimistic about how widespread that kind of experience is. Edit: I should clarify somewhat: I’ve had to introduce at least one of the following concepts at basically every client: continuous/automatic deployments; automatic migrations; automatic maintenance pages during outages (planned or otherwise); and hard as it is to believe: version control. My clients are typically smallish, so anything I can do to solve problems/provide solutions at the infra layer is a no-brainer because setting up an automated {enable maintenance mode; apply migrations; disable maintenance mode} is a lot more reliable long term than trying to teach every possible developer how to handle backwards compatible models - particularly when the clients application language of choice is something I’m not necessarily experienced in.
  - dijit 5y ago
    You said "read" but there is at least this talk by the Datadog team: https://www.youtube.com/watch?v=2dsCwp_j0yQ https://www.youtube.com/watch?v=2dsCwp_j0yQ
- stephenr 5y ago
  > Do you have a blog No, I have a job. Seriously though, no. As a one-man band, anything besides "client X needs Y" or "[internal/forked/contributed to] tool A needs enhancement/bugfix B" tends to be very low priority.
KronisLV 5y ago
> TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. While that's a fair position to take, you also might be in the minority here. I've seen plenty of in-house solutions in corporate environments and one of the things that they've all had in common was how bad they were, due to them being a way to just get things done and no more (e.g. no proper testing, no proper documentation, bad/wrong abstractions). It's not like your (or anyone else's) collection of Bash scripts can compete with open source solutions that have had thousands of collective man years put into making them better and addressing the never ending churn of keeping things working across new distro releases etc. I'm not advocating to go straight to Kubernetes - Docker Swarm or Hashicorp Nomad, or even just Docker Compose can all be decent alternatives that are simpler to wrap one's head around and manage, in most smaller deployments. As for no one knowing what containers do, i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat, with who knows what sort of Cthulhu living in the shared libraries folder and everything being so brittle that you cannot even look at it the wrong way, or it'll break and you'll spend days digging through obscure error logs. Worse yet, have you ever needed to manage an environment that has applications that need 4 different JDK versions and many different launch methods (some use systemd services with Tomcat, some use Tomcat launch scripts, some have just a .jar that needs to be launched etc.), yet still count upon some shared environment configuration being present? Give me individual containers over that mess any day. I think that there should be a clear separation between what a server needs to keep operating and where the "business software" would run - basically an /app directory with a subdirectory per application with whatever it needs, be it bind mounted data storage, configuration files, log output (in lieu of log shipping) etc., as opposed to having some stuff be strewn across /var, /etc, /opt, /www and who knows how many other directories. Containers give me just that.
- stephenr 5y ago
  > you also might be in the minority here I'd be astounded if I'm not. The tech industry is ridiculously cargo-culty. I never said I expect the majority of people to agree with me, I said what I prefer. The majority of people probably think McDonalds is just fine, and that Google is "not so bad" too, and it doesn't bother me that other people disagree with my opinion. If you ask me to fix your fucked infra because someone threw together a Rube Goldberg like Kubernetes environment just to achieve rolling deployments of their application, I'm going to tell you it's a shit solution, just as if you asked me to critique a McDonalds hamburger I'm going to tell you it's a shit hamburger. Me saying it's shit doesn't mean you can't use the thing, it means my opinion is that there are better ways to solve the problem/achieve the desired outcome, and you can either accept that opinion and make a change, or disagree and I'll go provide my services to another client. It's really that simple. > thousands of collective man years put into making them better They've had a ridiculous large number of man years working on them, no question. The whole point is that we disagree about whether they're "better" or not. Adding complexity is not necessarily making something better. > i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat In... a decade and a half of doing this I think I've been asked to support one JVM based thing, and that was quite a while ago. I'm sure it is a shit show, but I'm also sure that there are better alternatives than 8 more layers of abstraction. Lightweight VM-like containers are literally a thing that exists, and can give essentially the same provisioning as a traditional VM, with the "lightweight" / "shared resources" aspect of a container solution.

robertlagrant 5y ago

Be cool to hear some examples/war stories. Do you have a blog?

bsaul 5y ago
same here. Really interested to read about the popular ways of screwing a k8s cluster
- stephenr 5y ago
  I won't (and can't) go into details for obvious reasons, but here's a tip from a recent WTF: Something is not right if at the start of a planned outage, it takes your staff who "built" the system in place an amount of time measured in hours to: - block most public traffic; while - allowing traffic from a restricted set of IPs; and - showing a 'nice' maintenance page to the blocked visitors I don't honestly recall the exact time it took them to make the above changes, but it was somewhere in the order of 3 hours.
  - robertlagrant 5y ago
    Agree that maintenance mode is often not a capability prioritised. Unless you use a cloud provider's product, like AFD.
    - stephenr 5y ago
      Which is a fucking ridiculous situation when you think about it. The whole point of "DevOps" is to promote things like continuous delivery. Smooth, automated deployments. You can't tell me that "we need to deploy a change, that requires a migration be applied to the stored data, and must prevent user access while this occurs" is not somewhat common.
      - robertlagrant 5y ago
        We do it by having services auto-upgrade data migrations, migrations only take a few seconds at most, services maintain old endpoints, newer versions of which will work on new or old database structures (removed in future versions) and doing them at a time that will disrupt basically no one. That's not to avoid the work of creating maintenance pages of course, it's to avoid needing them.
        stephenr 5y ago
        My take on that is that you’re creating work in one place to avoid it in another. And it sounds like it works for you, but I don’t think I’d choose that route, and I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations.
        robertlagrant 5y ago
        > I definitely wouldn’t say you can avoid needing maintenance pages just because you don’t have planned downtime to do migrations For sure. I'm just making the point that to do continuous delivery you wouldn't necessarily need maintenance mode pages, which predate continuous delivery.
        stephenr 5y ago
        No there are definitely alternatives, but IME having a maintenance page is a good idea/necessity anyway, and the engineering involved to activate it during deployments is essentially set and forget, whereas the engineering to make your application backwards compatible with any possible schema change is entirely an ongoing issue, and a lot more complex. If your development team is sufficiently experienced to do allow for the no-downtime approach that’s a big win, but my experience so far makes me pessimistic about how widespread that kind of experience is. Edit: I should clarify somewhat: I’ve had to introduce at least one of the following concepts at basically every client: continuous/automatic deployments; automatic migrations; automatic maintenance pages during outages (planned or otherwise); and hard as it is to believe: version control. My clients are typically smallish, so anything I can do to solve problems/provide solutions at the infra layer is a no-brainer because setting up an automated {enable maintenance mode; apply migrations; disable maintenance mode} is a lot more reliable long term than trying to teach every possible developer how to handle backwards compatible models - particularly when the clients application language of choice is something I’m not necessarily experienced in.
- dijit 5y ago
  You said "read" but there is at least this talk by the Datadog team: https://www.youtube.com/watch?v=2dsCwp_j0yQ https://www.youtube.com/watch?v=2dsCwp_j0yQ
stephenr 5y ago
> Do you have a blog No, I have a job. Seriously though, no. As a one-man band, anything besides "client X needs Y" or "[internal/forked/contributed to] tool A needs enhancement/bugfix B" tends to be very low priority.

KronisLV 5y ago

> TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. While that's a fair position to take, you also might be in the minority here. I've seen plenty of in-house solutions in corporate environments and one of the things that they've all had in common was how bad they were, due to them being a way to just get things done and no more (e.g. no proper testing, no proper documentation, bad/wrong abstractions). It's not like your (or anyone else's) collection of Bash scripts can compete with open source solutions that have had thousands of collective man years put into making them better and addressing the never ending churn of keeping things working across new distro releases etc. I'm not advocating to go straight to Kubernetes - Docker Swarm or Hashicorp Nomad, or even just Docker Compose can all be decent alternatives that are simpler to wrap one's head around and manage, in most smaller deployments. As for no one knowing what containers do, i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat, with who knows what sort of Cthulhu living in the shared libraries folder and everything being so brittle that you cannot even look at it the wrong way, or it'll break and you'll spend days digging through obscure error logs. Worse yet, have you ever needed to manage an environment that has applications that need 4 different JDK versions and many different launch methods (some use systemd services with Tomcat, some use Tomcat launch scripts, some have just a .jar that needs to be launched etc.), yet still count upon some shared environment configuration being present? Give me individual containers over that mess any day. I think that there should be a clear separation between what a server needs to keep operating and where the "business software" would run - basically an /app directory with a subdirectory per application with whatever it needs, be it bind mounted data storage, configuration files, log output (in lieu of log shipping) etc., as opposed to having some stuff be strewn across /var, /etc, /opt, /www and who knows how many other directories. Containers give me just that.

stephenr 5y ago
> you also might be in the minority here I'd be astounded if I'm not. The tech industry is ridiculously cargo-culty. I never said I expect the majority of people to agree with me, I said what I prefer. The majority of people probably think McDonalds is just fine, and that Google is "not so bad" too, and it doesn't bother me that other people disagree with my opinion. If you ask me to fix your fucked infra because someone threw together a Rube Goldberg like Kubernetes environment just to achieve rolling deployments of their application, I'm going to tell you it's a shit solution, just as if you asked me to critique a McDonalds hamburger I'm going to tell you it's a shit hamburger. Me saying it's shit doesn't mean you can't use the thing, it means my opinion is that there are better ways to solve the problem/achieve the desired outcome, and you can either accept that opinion and make a change, or disagree and I'll go provide my services to another client. It's really that simple. > thousands of collective man years put into making them better They've had a ridiculous large number of man years working on them, no question. The whole point is that we disagree about whether they're "better" or not. Adding complexity is not necessarily making something better. > i would still take a collection of containers over the nightmare of 10 different webapps all being deployed in an instance of GlassFish/Tomcat In... a decade and a half of doing this I think I've been asked to support one JVM based thing, and that was quite a while ago. I'm sure it is a shit show, but I'm also sure that there are better alternatives than 8 more layers of abstraction. Lightweight VM-like containers are literally a thing that exists, and can give essentially the same provisioning as a traditional VM, with the "lightweight" / "shared resources" aspect of a container solution.