4 ms·

Oh geez, having a team of 24 engineers across multiple countries trying to maintain a common set of bash scripts that work for 12+ unique services sounds like a

Oh geez, having a team of 24 engineers across multiple countries trying to maintain a common set of bash scripts that work for 12+ unique services sounds like a disaster. Instead we have a single workflow for docker, whether it's going out to AWS or Kubernetes and all the services use an identical flow. Integration testing is a cinch since it's running actual, version matched postgres/redis/etc on the CI nodes which get spun up on demand and against the same docker image that'll get pushed out to production. We haven't had a case of "Well it works on my machine why doesn't it work on yours!?" in a really long time since if it works in docker.. well it works in docker.

leecarraher 7y ago
Exactly, docker standardizes something that was an otherwise nightmare to maintain across platforms and organizations. Next we need to standardize the orchestrator, my money is on kubernetes, but I wouldn't say it has won over everyone just yet.
- SaasDeveloper12 7y ago
  The industry has standardized on Kubernetes for the orchestrator. If anything we are starting to see other container runtimes other than Docker as K8s added the CRI (Container Runtime Interface).
  - vorpalhex 7y ago
    Well everyone at least wants to use Kubernetes. I'm not sure any organization really knows how to best use it yet, and nobody has a set of standards that have seen adoption. Both the pro and the con of K8s is that it's exactly what you make of it and it's extremely flexible.
- user5994461 7y ago
  Kubernetes has won already and Docker went bankrupt. It might take a bit of time for everybody to realize but the dice have rolled some time ago. the company was sold for scrap shortly after this article https://thehftguy.com/2019/10/22/the-demise-of-docker-and-the-rise-of-kubernetes/ https://thehftguy.com/2019/10/22/the-demise-of-docker-and-th...
matteuan 7y ago
The author addressed exactly this type of comment. I don't see the real difference in complexity of maintaining 12+ bash scripts instead of the same number of Docker files. And for the deployment, the alternative given by the author is using fat binary.
- shadowgovt 7y ago
  > I don't see the real difference in complexity of maintaining 12+ bash scripts instead of the same number of Docker files. With respect, I have to assume your experience is either in an ecosystem with really strongly-established best practices for bash (and someone with the job of enforcing said practices) or your organization has no more than two or three people maintaining those dozen-plus bash scripts (and is on its way to a headache when two of those people retire at the same time and your org tries to on-board someone new to read those scripts).
  - matteuan 7y ago
    You judge "with respect" my experience instead of answering to the argument. So why is it easier with Docker?
    - shadowgovt 7y ago
      Because a Docker config makes it hard to accidentally wipe your filesystem instead of building a Docker image. Shell scripts make that less difficult (and any solution using shell to do something similar to Docker will have a bespoke layer of in-house code that should be correct but will require maintenance by your team to make sure nobody makes a change that turns it into a filesystem-eater).
    - cbushko 7y ago
      It is easier because you can actually build that functionality locally with a docker container to test it out. This is coming from someone that spent 10+ years maintaining a build system that was a mix of makefiles and shell scripts. It can be done but I don't know if it should.
  - 7y ago
jasonlotito 7y ago
Fewer than half the number of engineers with hundreds of unique services. We aren't "just" using a common set of bash scripts, but we do have few. It's not a disaster, and it just works. What we deploy to a testing environment is what we deploy to production. Not a disaster. Deployments are a single button push. Building the framework for a new services is filling out a form, and everything needed gets generated, including automated documentation, life cycle jobs, repos in github, properly configured security groups, etc. We are exploring Docker, and while Docker offers some nice advantages, the added complexity and additional points of failure needs to bring some major benefits, and need to solve real problems we have. Not imagined problems.
alxlaz 7y ago
That hasn't been my experience with docker at all. Except for pretty simple projects, I can't remember a single project I've worked on where I could just follow the README/"Here's how to set it up" email/whatever and have it work the first time. Every time a new developer came in, we'd find out that some implicit assumption was baked in somewhere, and finding out where took hours or days because debugging the damn thing is almost impossible. Everyone eventually syncs their environment but that just conveniently provides the illusion of standardization and repeatability. It's the same kind of "standardization" and "repeatability" we used to get twenty years ago -- all standard and repeatable, as long as no one touches the Solaris box that hosts the production deployments and no one updates their workstations. And that's just the "legitimate" stuff. The amount of hours that we've wasted on things like "X runs docker 1.9 but Y runs docker 1.10", or overlay bugs, or image pruning failures, is insane. Breaking things is an inevitable side effect of progress but maybe infrastructure software shouldn't break this much... Far from me to say that 20+ engineers across the world maintaining bash scripts for a dozen services is better. I used to do that and it was extremely terrible. Even when everyone involved had years of experience and the mundane task of updating deployment scripts wasn't delegated to interns, the way it is today (for reasons that I am entirely unable to comprehend). Definitely far more terrible than Docker. Docker is not as terrible, but still pretty terrible. I'm open to the possibility that I've been doing it wrong, and that every team I've worked in/every customer I've worked for has been doing it wrong. But this has been going on for like four years now and enlightenment doesn't seem to be any closer to hitting me...
- pm90 7y ago
  I don’t wanna gaslight you but this doesn’t match my experience at all. I’ve literally never run into issues with something compiling info one version of docker but not another. Image pulling has been a problem sometimes on Kubernetes but you just need to configure the creds correctly and boom it works.
  - alxlaz 7y ago
    Hey, don't worry :-). Like I said, I'm open to the possibility that I've been doing it wrong this whole time, or that I worked on projects that kept hitting cornercases and their associated bugs. I'm glad that you didn't have to deal with any of the bad parts.
mumblemumble 7y ago
> having a team of 24 engineers across multiple countries trying to maintain a common set of bash scripts that work for 12+ unique services At the last place I worked that had a truly tight devops operation, it was 3 DevOps people maintaining infrastructure for 200+ unique services. And most that infrastructure was building really useful junk that I've never seen available elsewhere, like daily email reports telling me about every error that was logged on any services that my team maintained, and a cool little web UI that monitored our message bus and generated a visualization of which services (and instances) were listening to what. The actual deployment and process management infrastructure wasn't very complicated, either, based on what I saw from poking around in their source control. They were able to shave a whole lot of unnecessary complexity by enforcing rigid standards about how each process would be packaged and configured and whatnot, regardless of implementation language. It was up to the developers who wanted to take on $hot_new_langauge to figure out how make it behave right. And that gets to my real worry about Docker. It's a way of dealing with messes, not by cleaning them up, but by sweepi^H^H^H^H^H^H hiding it behind an abstraction. It's like that meme image that floated around a while back - "But it works on my machine." "Then we'll ship your machine." And that's how Docker was born. Which isn't to say that this is useless - I use Docker all the time, chiefly to do things like making sure I'll never have to manually futz around with managing multiple concurrent PostgreSQL installs ever again - but, at least in production, it still feels to me like high tech duct tape and chewing gum.
- dariusj18 7y ago
  I think docker/containers are the opposite of abstraction. It's a reduction of variables and creates true predictability across very disparate systems.
  - mumblemumble 7y ago
    It's, at best, semi-consistent. In the Dockerized stack I'm working on right now, individual services are configured using a mishmash of environment variables, files you volume in (each one with a different format), shell scripts you volume in that will modify configuration files on startup, Dockerfiles that create derivative images that replace configuration files with instance-specific configuration, Dockerfiles with a bunch of RUN commands to handle more surgical modification of the configuration that was baked into some standardized base image. . . it's just crazy how variable it is. The obvious reply is, "Well, don't do that." And my general reaction to that idea is is, if you have the discipline to not do that with containers, you have the discipline to not do that without containers. Containerization is the stone in the soup. With a extra little side barb of, Docker doesn't really leave you any choice but to get messy. If you go troll through Docker issues, there are plenty of long-standing threads with community members requesting features to help them simplify and standardize their Docker situation, and Docker maintainers habitually rejecting the ideas on the grounds that they can already achieve what they want using some version of the abovementioned horrible mess of different ways to do things.

leecarraher 7y ago

Exactly, docker standardizes something that was an otherwise nightmare to maintain across platforms and organizations. Next we need to standardize the orchestrator, my money is on kubernetes, but I wouldn't say it has won over everyone just yet.

SaasDeveloper12 7y ago
The industry has standardized on Kubernetes for the orchestrator. If anything we are starting to see other container runtimes other than Docker as K8s added the CRI (Container Runtime Interface).
- vorpalhex 7y ago
  Well everyone at least wants to use Kubernetes. I'm not sure any organization really knows how to best use it yet, and nobody has a set of standards that have seen adoption. Both the pro and the con of K8s is that it's exactly what you make of it and it's extremely flexible.
user5994461 7y ago
Kubernetes has won already and Docker went bankrupt. It might take a bit of time for everybody to realize but the dice have rolled some time ago. the company was sold for scrap shortly after this article https://thehftguy.com/2019/10/22/the-demise-of-docker-and-the-rise-of-kubernetes/ https://thehftguy.com/2019/10/22/the-demise-of-docker-and-th...

matteuan 7y ago

The author addressed exactly this type of comment. I don't see the real difference in complexity of maintaining 12+ bash scripts instead of the same number of Docker files. And for the deployment, the alternative given by the author is using fat binary.

shadowgovt 7y ago
> I don't see the real difference in complexity of maintaining 12+ bash scripts instead of the same number of Docker files. With respect, I have to assume your experience is either in an ecosystem with really strongly-established best practices for bash (and someone with the job of enforcing said practices) or your organization has no more than two or three people maintaining those dozen-plus bash scripts (and is on its way to a headache when two of those people retire at the same time and your org tries to on-board someone new to read those scripts).
- matteuan 7y ago
  You judge "with respect" my experience instead of answering to the argument. So why is it easier with Docker?
  - shadowgovt 7y ago
    Because a Docker config makes it hard to accidentally wipe your filesystem instead of building a Docker image. Shell scripts make that less difficult (and any solution using shell to do something similar to Docker will have a bespoke layer of in-house code that should be correct but will require maintenance by your team to make sure nobody makes a change that turns it into a filesystem-eater).
  - cbushko 7y ago
    It is easier because you can actually build that functionality locally with a docker container to test it out. This is coming from someone that spent 10+ years maintaining a build system that was a mix of makefiles and shell scripts. It can be done but I don't know if it should.
- 7y ago

jasonlotito 7y ago

Fewer than half the number of engineers with hundreds of unique services. We aren't "just" using a common set of bash scripts, but we do have few. It's not a disaster, and it just works. What we deploy to a testing environment is what we deploy to production. Not a disaster. Deployments are a single button push. Building the framework for a new services is filling out a form, and everything needed gets generated, including automated documentation, life cycle jobs, repos in github, properly configured security groups, etc. We are exploring Docker, and while Docker offers some nice advantages, the added complexity and additional points of failure needs to bring some major benefits, and need to solve real problems we have. Not imagined problems.

alxlaz 7y ago

That hasn't been my experience with docker at all. Except for pretty simple projects, I can't remember a single project I've worked on where I could just follow the README/"Here's how to set it up" email/whatever and have it work the first time. Every time a new developer came in, we'd find out that some implicit assumption was baked in somewhere, and finding out where took hours or days because debugging the damn thing is almost impossible. Everyone eventually syncs their environment but that just conveniently provides the illusion of standardization and repeatability. It's the same kind of "standardization" and "repeatability" we used to get twenty years ago -- all standard and repeatable, as long as no one touches the Solaris box that hosts the production deployments and no one updates their workstations. And that's just the "legitimate" stuff. The amount of hours that we've wasted on things like "X runs docker 1.9 but Y runs docker 1.10", or overlay bugs, or image pruning failures, is insane. Breaking things is an inevitable side effect of progress but maybe infrastructure software shouldn't break this much... Far from me to say that 20+ engineers across the world maintaining bash scripts for a dozen services is better. I used to do that and it was extremely terrible. Even when everyone involved had years of experience and the mundane task of updating deployment scripts wasn't delegated to interns, the way it is today (for reasons that I am entirely unable to comprehend). Definitely far more terrible than Docker. Docker is not as terrible, but still pretty terrible. I'm open to the possibility that I've been doing it wrong, and that every team I've worked in/every customer I've worked for has been doing it wrong. But this has been going on for like four years now and enlightenment doesn't seem to be any closer to hitting me...

pm90 7y ago
I don’t wanna gaslight you but this doesn’t match my experience at all. I’ve literally never run into issues with something compiling info one version of docker but not another. Image pulling has been a problem sometimes on Kubernetes but you just need to configure the creds correctly and boom it works.
- alxlaz 7y ago
  Hey, don't worry :-). Like I said, I'm open to the possibility that I've been doing it wrong this whole time, or that I worked on projects that kept hitting cornercases and their associated bugs. I'm glad that you didn't have to deal with any of the bad parts.

mumblemumble 7y ago

> having a team of 24 engineers across multiple countries trying to maintain a common set of bash scripts that work for 12+ unique services At the last place I worked that had a truly tight devops operation, it was 3 DevOps people maintaining infrastructure for 200+ unique services. And most that infrastructure was building really useful junk that I've never seen available elsewhere, like daily email reports telling me about every error that was logged on any services that my team maintained, and a cool little web UI that monitored our message bus and generated a visualization of which services (and instances) were listening to what. The actual deployment and process management infrastructure wasn't very complicated, either, based on what I saw from poking around in their source control. They were able to shave a whole lot of unnecessary complexity by enforcing rigid standards about how each process would be packaged and configured and whatnot, regardless of implementation language. It was up to the developers who wanted to take on $hot_new_langauge to figure out how make it behave right. And that gets to my real worry about Docker. It's a way of dealing with messes, not by cleaning them up, but by sweepi^H^H^H^H^H^H hiding it behind an abstraction. It's like that meme image that floated around a while back - "But it works on my machine." "Then we'll ship your machine." And that's how Docker was born. Which isn't to say that this is useless - I use Docker all the time, chiefly to do things like making sure I'll never have to manually futz around with managing multiple concurrent PostgreSQL installs ever again - but, at least in production, it still feels to me like high tech duct tape and chewing gum.

dariusj18 7y ago
I think docker/containers are the opposite of abstraction. It's a reduction of variables and creates true predictability across very disparate systems.
- mumblemumble 7y ago
  It's, at best, semi-consistent. In the Dockerized stack I'm working on right now, individual services are configured using a mishmash of environment variables, files you volume in (each one with a different format), shell scripts you volume in that will modify configuration files on startup, Dockerfiles that create derivative images that replace configuration files with instance-specific configuration, Dockerfiles with a bunch of RUN commands to handle more surgical modification of the configuration that was baked into some standardized base image. . . it's just crazy how variable it is. The obvious reply is, "Well, don't do that." And my general reaction to that idea is is, if you have the discipline to not do that with containers, you have the discipline to not do that without containers. Containerization is the stone in the soup. With a extra little side barb of, Docker doesn't really leave you any choice but to get messy. If you go troll through Docker issues, there are plenty of long-standing threads with community members requesting features to help them simplify and standardize their Docker situation, and Docker maintainers habitually rejecting the ideas on the grounds that they can already achieve what they want using some version of the abovementioned horrible mess of different ways to do things.