6 ms·

You’re comparing cowboy sysadmin (mutable servers, ssh in and live-edit stuff) to process-heavy devops with CI. These are orthogonal to containers/not-container

You’re comparing cowboy sysadmin (mutable servers, ssh in and live-edit stuff) to process-heavy devops with CI. These are orthogonal to containers/not-containers. If you don’t use CI, it’s easy to get fast deploys with containers. Just build the images on your dev box, tag with the branch and commit hash, and push directly to a docker registry (docker push is smart enough to only push the layers the registry doesn’t already have). Code is running 30 seconds after compiling finishes. (Don’t want to pay for a registry? It’s trivial to run one yourself) These aren’t foolproof, fully reproducible builds, but practically they’re pretty close if your tools require your repo to be clean and pushed before building the image, and if your build system is sane. Besides, if you’re used to editing code as it’s running on servers, you don’t care about reproducible builds. Also, if you’re starting containers manually on the command line, you’re doing it wrong. At least use compose so your setup is declarative and lifetime-managed. (Edit: s/swarm/compose/)

murkt 6y ago
> Also, if you’re starting containers manually on the command line, you’re doing it wrong. At least use compose so your setup is declarative and lifetime-managed. As I wrote in a nearby comment, I'm not starting containers manually - we have compose, swarm, it's declarative and lifetime-managed. However, we often need to do some bespoke data analysis, so we often ssh into a server, type `make shell` to launch a REPL and type/paste some stuff into it.
- akvadrako 6y ago
  You can do all that with containers quickly. I develop my stuff with k8s with all it's lifetime management and have 10 second deploys from my dev box.
  - murkt 6y ago
    10 seconds deploys? How is this possible :) Do you have any links that explain your workflow?
    - akvadrako 6y ago
      It's nothing fancy - I just use a typical k8s Service/Deployment object on GKE. A deploy is: 1. docker build (most layers cached) - 2s 2. docker push - 2s 3. update deploy.yaml 4. kubectl apply -f deploy.yaml 5. kubectl rollout status deployments {name} - 6s
      - physicles 6y ago
        Yeah this is exactly what I do too, works just fine. You probably already have something like this, but I hacked a bash+yq script that automatically updates all relevant yaml files with the latest image tag. So getting new code running is two lines: make image push deploy kubectl -f somewhere/deploy.yaml
      - lovehashbrowns 6y ago
        Lucky!! With our infrastructure, deploys can take an hour or so. 10 minutes for the build, 10 minutes for the image to get built, plus the rest of the time for terraform to apply infrastructure changes across dev, staging, and prod. Only thing we do have is automated testing after every deploy so issues tend to get caught. But that's still so long for a deploy! I don't know of a good way to get it down faster.
        jonfw 6y ago
        Why is terraform deploying infrastructure for every container deployment? Can't you just rollout onto the existing infrastructure? Also sounds like there is some lay hanging fruit available by adding some caching/layering in the build process
        lovehashbrowns 6y ago
        Ah, I should have explained more. That's an hour for our ASG + EC2 deployments. The only benefit we get is easy roll backs because it always deploys a new ASG. We're switching over to Spinnaker and started with our EC2 infrastructure. I think container deployments will be faster but still, an hour for EC2 deployments!
  - tluyben2 6y ago
    Still 10 seconds. I have 0 seconds deploy for some ‘cowboy development’ I do; it is now a competitive advantage :) 10 sec (and for almost all setups I have seen it is vastly more that than) is a lot while devving and deploying for test. Each their own, but just fixing bugs live with my client (Zoom + me live on the test server fixing many issues in an afternoon) is vastly more efficient for me. Obviously committing from the test(now dev) server to github will result in ci/cd to the staging server, but workflows where I work on my local, commit, ci/cd to test and then the client tests is vastly slower and I do not like it; it feels like a waste of my time. In my main business I am forced (regulatory) to do it all by the book; vastly less enjoyable for me.
jrockway 6y ago
I think people kind of boiled themselves alive with Docker and don't step back to think if they're where they want to be often enough. Docker first started getting traction when people were building their software with make. Make never quite got caching right (not really its fault), so nobody was really sure that their changes were going to show up in their release unless they ran "make clean" first. And, you had to have 700 helper utilities installed on your workstation for a build to work after "make clean". Docker to the rescue! It gives you a fresh Linux distribution, apt-get installs all the utilities needed for your build, and then builds your thing. It works 100% of the time! Celebrate! At the same time, programming languages started moving towards being build systems. They want "mylang run foo.ml" to be fast, so they implemented their own caching. But this time they did it right; the "mylang" compiler knows about the effect every input file has on every output file, so it's guaranteed to give you the right answer with or without the cache. Some languages are so confident these days that you can't even disable the cache! They know it works perfectly every time. The result is extremely fast incremental builds that are just as reliable as clean builds, if you have access to that cache. This, unfortunately, is not something that Docker supports -- layers have one input filesystem and one output filesystem, but now languages are producing two outputs -- one binary for the next layer, one cache directory that should be used as input (opportunistically) to the next build. The result is, to work around people writing makefiles like "foo.o: foo.c" when they actually meant "foo.o: foo.c foo.h helper.h /usr/include/third-party-library.h /usr/lib/libthird-party.so.42", EVERYONE suffers. "mylang run foo.mt" takes a few milliseconds on your workstation, but 5 minutes in Docker. Your critical fix is hamstrung by your CI infrastructure. There are a number of solutions to this problem. You could have language-specific builders that integrate with your CI system, take the source code and a recent cache as inputs, and produce a Docker container as output. (Systems like Bazel even have a protocol for sharing cache between machines, so you don't have to copy gigabytes of data around that you probably don't need.) But instead of doing that, people are writing articles about how their CI takes 3 hours when a build on their workstation takes 3 seconds, and it's because containers suck! But they don't actually suck -- the underlying problem is not saying "in production, we will only run .tar.gz files that contain everything the application needs". That is actually wonderful. The underlying problem is "the first build step is 'rm -rf /' and the second is 'make world'".

murkt 6y ago

> Also, if you’re starting containers manually on the command line, you’re doing it wrong. At least use compose so your setup is declarative and lifetime-managed. As I wrote in a nearby comment, I'm not starting containers manually - we have compose, swarm, it's declarative and lifetime-managed. However, we often need to do some bespoke data analysis, so we often ssh into a server, type `make shell` to launch a REPL and type/paste some stuff into it.

akvadrako 6y ago
You can do all that with containers quickly. I develop my stuff with k8s with all it's lifetime management and have 10 second deploys from my dev box.
- murkt 6y ago
  10 seconds deploys? How is this possible :) Do you have any links that explain your workflow?
  - akvadrako 6y ago
    It's nothing fancy - I just use a typical k8s Service/Deployment object on GKE. A deploy is: 1. docker build (most layers cached) - 2s 2. docker push - 2s 3. update deploy.yaml 4. kubectl apply -f deploy.yaml 5. kubectl rollout status deployments {name} - 6s
    - physicles 6y ago
      Yeah this is exactly what I do too, works just fine. You probably already have something like this, but I hacked a bash+yq script that automatically updates all relevant yaml files with the latest image tag. So getting new code running is two lines: make image push deploy kubectl -f somewhere/deploy.yaml
    - lovehashbrowns 6y ago
      Lucky!! With our infrastructure, deploys can take an hour or so. 10 minutes for the build, 10 minutes for the image to get built, plus the rest of the time for terraform to apply infrastructure changes across dev, staging, and prod. Only thing we do have is automated testing after every deploy so issues tend to get caught. But that's still so long for a deploy! I don't know of a good way to get it down faster.
      - jonfw 6y ago
        Why is terraform deploying infrastructure for every container deployment? Can't you just rollout onto the existing infrastructure? Also sounds like there is some lay hanging fruit available by adding some caching/layering in the build process
        lovehashbrowns 6y ago
        Ah, I should have explained more. That's an hour for our ASG + EC2 deployments. The only benefit we get is easy roll backs because it always deploys a new ASG. We're switching over to Spinnaker and started with our EC2 infrastructure. I think container deployments will be faster but still, an hour for EC2 deployments!
- tluyben2 6y ago
  Still 10 seconds. I have 0 seconds deploy for some ‘cowboy development’ I do; it is now a competitive advantage :) 10 sec (and for almost all setups I have seen it is vastly more that than) is a lot while devving and deploying for test. Each their own, but just fixing bugs live with my client (Zoom + me live on the test server fixing many issues in an afternoon) is vastly more efficient for me. Obviously committing from the test(now dev) server to github will result in ci/cd to the staging server, but workflows where I work on my local, commit, ci/cd to test and then the client tests is vastly slower and I do not like it; it feels like a waste of my time. In my main business I am forced (regulatory) to do it all by the book; vastly less enjoyable for me.

jrockway 6y ago

I think people kind of boiled themselves alive with Docker and don't step back to think if they're where they want to be often enough. Docker first started getting traction when people were building their software with make. Make never quite got caching right (not really its fault), so nobody was really sure that their changes were going to show up in their release unless they ran "make clean" first. And, you had to have 700 helper utilities installed on your workstation for a build to work after "make clean". Docker to the rescue! It gives you a fresh Linux distribution, apt-get installs all the utilities needed for your build, and then builds your thing. It works 100% of the time! Celebrate! At the same time, programming languages started moving towards being build systems. They want "mylang run foo.ml" to be fast, so they implemented their own caching. But this time they did it right; the "mylang" compiler knows about the effect every input file has on every output file, so it's guaranteed to give you the right answer with or without the cache. Some languages are so confident these days that you can't even disable the cache! They know it works perfectly every time. The result is extremely fast incremental builds that are just as reliable as clean builds, if you have access to that cache. This, unfortunately, is not something that Docker supports -- layers have one input filesystem and one output filesystem, but now languages are producing two outputs -- one binary for the next layer, one cache directory that should be used as input (opportunistically) to the next build. The result is, to work around people writing makefiles like "foo.o: foo.c" when they actually meant "foo.o: foo.c foo.h helper.h /usr/include/third-party-library.h /usr/lib/libthird-party.so.42", EVERYONE suffers. "mylang run foo.mt" takes a few milliseconds on your workstation, but 5 minutes in Docker. Your critical fix is hamstrung by your CI infrastructure. There are a number of solutions to this problem. You could have language-specific builders that integrate with your CI system, take the source code and a recent cache as inputs, and produce a Docker container as output. (Systems like Bazel even have a protocol for sharing cache between machines, so you don't have to copy gigabytes of data around that you probably don't need.) But instead of doing that, people are writing articles about how their CI takes 3 hours when a build on their workstation takes 3 seconds, and it's because containers suck! But they don't actually suck -- the underlying problem is not saying "in production, we will only run .tar.gz files that contain everything the application needs". That is actually wonderful. The underlying problem is "the first build step is 'rm -rf /' and the second is 'make world'".