4 ms·

Am I the only one who seen the pattern of using one Docker on one VM? i.e whole VM does nothing except run one Docker, or multiple Dockers but all part of the s

Am I the only one who seen the pattern of using one Docker on one VM? i.e whole VM does nothing except run one Docker, or multiple Dockers but all part of the same app and running as a single unit, not distinct ones. I talked to a few friends and at least one of them told me they've seen the same pattern used in their company too.

throwaway4good 5y ago
There are many many silly uses of docker and k8s out there. But people seem to like it ...
- sn_master 5y ago
  Yup, I feel they try to justify themselves and feel they've done something complicated that takes time to understand by newcomers, when in reality the whole thing could be a single Go or C# binary and the entire utilized cluster power is less than a single mid-sized VM that could just run said single binary, without any external dependencies that usually get used to justify Docker. If people paying the money knew the amount of waste being done...
nrmitchi 5y ago
I've seen it, and I've also seen it in Kubernetes. It's obviously not the intended use case, but it can have the advantage of plugging in much nicer with your existing tooling. As an example, if I have a Kubernetes cluster, and a new app that needs a full VM. My options are to containerize it (it likely already comes this way), and let Kubernetes scheduled it on it's own VM, or use a different tool in order to manage the unique VM for this app. Unless there are reasons to avoid the containerize approach, I'm going to stick with the existing tooling rather than add additional complexity.
- sn_master 5y ago
  > and a new app that needs a full VM. Most cases I've seen the apps don't need more than a two dozen megabytes out of the gigabytes in the VM. It was crazy seeing how much resources were being wasted for no good reason.
nerdponx 5y ago
Why?
Spivak 5y ago
My company does it. Our atomic unit of software is a VM. All of our isolation, resource provisioning, regulatory compliance, lifecycle management, access management are based on VMs. Another way of saying this is “vms are typed.” So docker is basically just a package manager and supervisord replacement for us because both ops and devs find it easier to build do deploys with containers than software on the host. We use no actual features of docker, basically everything is disabled. We could use podman but that would be more effort for the same thing.
- sn_master 5y ago
  Do you use a VPC too for all the VMs?
deleted 5y ago
[deleted]
bob1029 5y ago
I still don't understand why every software shop on earth feels the need to start with docker/k8s/et. al. Build the damn app one time using SQLite on a single box. Then, figure out if the market even gives a shit about it. Building out a multi-cloud application architecture for your first prototype is a massive mistake. You don't need containerization tech if you can clone your application, hit F5 in visual studio, and the whole stack starts up automagically. You will probably find the performance of a single, powerful machine to be more than enough for your expected user base. SQLite can handle a fuckload of 'concurrent' writers if you know what you are doing. There are now framework features available in some areas that help to obviate the need for containerization. If you are a .NET shop, look into self-contained deployments: https://docs.microsoft.com/en-us/dotnet/core/deploying/#publish-self-contained https://docs.microsoft.com/en-us/dotnet/core/deploying/#publ... We have been using this deployment approach for ~3 years now and we have yet to have a single situation where a required dependency was missing or broken in production. And, we deploy to a very wide range of customer environments with all sorts of horrible IT practices applied to them. Imagine being able to unzip a build of your software to a blank windows/linux server and expect that it work flawlessly 100% of the time, regardless of any prior/lack of configuration or other supporting dependencies on that machine.
- sn_master 5y ago
  > I still don't understand why every software shop on earth feels the need to start with docker/k8s/et. al. YES YES YES I am certain at this point they're doing it only to show off and to have something complicated enough to talk about the train new hires on. One big argument for Docker is no dependencies, but Go and C# already can create fat native binaries that have no dependency on anything else (no .net framework or even VM, all native, same thing in Go). I believe Rust too offers the same thing. There is no excuse with all those different languages all supporting that. There absolutely are complicated apps that justify Docker and k8s, but the vast majority in the real world do not fall into that, and most certainly that includes small ad-hoc internal services.
  - gindely 5y ago
    > One big argument for Docker is no dependencies, but Go and C# already can create fat native binaries that have no dependency on anything else (no .net framework or even VM, all native, same thing in Go). I believe Rust too offers the same thing. There is no excuse with all those different languages all supporting that. Using Nix, you can build a self-contained deployment for just about any language/rutime you can imagine, and the target machine doesn't need to run Nix.
nyrikki 5y ago
To add to the other replies, remember that containers do not provide isolation for security, they are simply namespaces. Rootless mode helps provide some isolation and individual machines also provide some isolation. Anyone who can launch a container with the `--privileged` flag has the ability to read the host VM's disks, or on a physical machine even update the bios etc... Launch a container with that flag and play with mknod and dd for the hosts root drive if you don't understand that attack vector. The question you should ask is why you aren't using user namespaces or isolated machines if you are attempting to follow the least privileges and zero trust models.
- sn_master 5y ago
  But then why not just use the VM itself and lose the container? Whatever restrictions you want to apply to the Docker, can be applied to the binary as well when invoking it.
  - nyrikki 5y ago
    While the choice of VM vs container is not one of good or bad in the general sense there are several reasons to choose containers. 1) Containers reduce the number of shared dependencies between the VM and the application. A container based on alpine will even be using musl libc vs glibc. And a VM running on a private cloud or different cloud providers will require different dependencies. As dependency hell is NP-complete, increasing the number of dependencies dramatically increases the effort of testing, trouble shooting, and maintaining a system. The simplified contract between a container and it's host vs a binary on a VM reduces that cost. 2) Instantiation time is much longer for a VM than a container. 3) Container management systems like k8 etc... tend to have more robust health checking, service registration, and recovery tools. If one is using containers already the cost of maintaining a parallel infrastructure for VMs is often high. 4) Developers can often run a container on their desktops/laptops and iterate faster as they avoid the challenges and costs in maintaining dependencies called out in #1 or the expense and time of spinning up development instances. There are companies and groups that use containers and container management systems because they are the hot new thing, but even before the public cloud or even high density VMs were a reality I was on a team that leverage Linux VServers for all core services. It is still the only company I have ever been at where we could actually switch over to a cold DR site with everything from DNS to Oracle EBS being up in less than half an hour. All with only using rsync and tar for a publicly traded company. You can get around package version selection being NP-complete by putting on other constraints like enforcing semantic versioning and auto updates. But IMHO using namespaces(containers) is one solution one should consider. Here is a link to a paper that will describe the package dependency problem generalizes to the NP-complete SAT problem. For me, containers are a way to limit that problem to being one of the container host only needing satisfy a contract for a far more limited set of needs (kernel ABI, network, storage). https://hal.inria.fr/hal-00697463 https://hal.inria.fr/hal-00697463 Your needs and challenges may be different, but containers tend to be a good option for many of the needs I have run across.

throwaway4good 5y ago

There are many many silly uses of docker and k8s out there. But people seem to like it ...

sn_master 5y ago
Yup, I feel they try to justify themselves and feel they've done something complicated that takes time to understand by newcomers, when in reality the whole thing could be a single Go or C# binary and the entire utilized cluster power is less than a single mid-sized VM that could just run said single binary, without any external dependencies that usually get used to justify Docker. If people paying the money knew the amount of waste being done...

nrmitchi 5y ago

I've seen it, and I've also seen it in Kubernetes. It's obviously not the intended use case, but it can have the advantage of plugging in much nicer with your existing tooling. As an example, if I have a Kubernetes cluster, and a new app that needs a full VM. My options are to containerize it (it likely already comes this way), and let Kubernetes scheduled it on it's own VM, or use a different tool in order to manage the unique VM for this app. Unless there are reasons to avoid the containerize approach, I'm going to stick with the existing tooling rather than add additional complexity.

sn_master 5y ago
> and a new app that needs a full VM. Most cases I've seen the apps don't need more than a two dozen megabytes out of the gigabytes in the VM. It was crazy seeing how much resources were being wasted for no good reason.

nerdponx 5y ago

Why?

Spivak 5y ago

My company does it. Our atomic unit of software is a VM. All of our isolation, resource provisioning, regulatory compliance, lifecycle management, access management are based on VMs. Another way of saying this is “vms are typed.” So docker is basically just a package manager and supervisord replacement for us because both ops and devs find it easier to build do deploys with containers than software on the host. We use no actual features of docker, basically everything is disabled. We could use podman but that would be more effort for the same thing.

sn_master 5y ago
Do you use a VPC too for all the VMs?

deleted 5y ago

[deleted]

bob1029 5y ago

I still don't understand why every software shop on earth feels the need to start with docker/k8s/et. al. Build the damn app one time using SQLite on a single box. Then, figure out if the market even gives a shit about it. Building out a multi-cloud application architecture for your first prototype is a massive mistake. You don't need containerization tech if you can clone your application, hit F5 in visual studio, and the whole stack starts up automagically. You will probably find the performance of a single, powerful machine to be more than enough for your expected user base. SQLite can handle a fuckload of 'concurrent' writers if you know what you are doing. There are now framework features available in some areas that help to obviate the need for containerization. If you are a .NET shop, look into self-contained deployments: https://docs.microsoft.com/en-us/dotnet/core/deploying/#publish-self-contained https://docs.microsoft.com/en-us/dotnet/core/deploying/#publ... We have been using this deployment approach for ~3 years now and we have yet to have a single situation where a required dependency was missing or broken in production. And, we deploy to a very wide range of customer environments with all sorts of horrible IT practices applied to them. Imagine being able to unzip a build of your software to a blank windows/linux server and expect that it work flawlessly 100% of the time, regardless of any prior/lack of configuration or other supporting dependencies on that machine.

sn_master 5y ago
> I still don't understand why every software shop on earth feels the need to start with docker/k8s/et. al. YES YES YES I am certain at this point they're doing it only to show off and to have something complicated enough to talk about the train new hires on. One big argument for Docker is no dependencies, but Go and C# already can create fat native binaries that have no dependency on anything else (no .net framework or even VM, all native, same thing in Go). I believe Rust too offers the same thing. There is no excuse with all those different languages all supporting that. There absolutely are complicated apps that justify Docker and k8s, but the vast majority in the real world do not fall into that, and most certainly that includes small ad-hoc internal services.
- gindely 5y ago
  > One big argument for Docker is no dependencies, but Go and C# already can create fat native binaries that have no dependency on anything else (no .net framework or even VM, all native, same thing in Go). I believe Rust too offers the same thing. There is no excuse with all those different languages all supporting that. Using Nix, you can build a self-contained deployment for just about any language/rutime you can imagine, and the target machine doesn't need to run Nix.

nyrikki 5y ago

To add to the other replies, remember that containers do not provide isolation for security, they are simply namespaces. Rootless mode helps provide some isolation and individual machines also provide some isolation. Anyone who can launch a container with the `--privileged` flag has the ability to read the host VM's disks, or on a physical machine even update the bios etc... Launch a container with that flag and play with mknod and dd for the hosts root drive if you don't understand that attack vector. The question you should ask is why you aren't using user namespaces or isolated machines if you are attempting to follow the least privileges and zero trust models.

sn_master 5y ago
But then why not just use the VM itself and lose the container? Whatever restrictions you want to apply to the Docker, can be applied to the binary as well when invoking it.
- nyrikki 5y ago
  While the choice of VM vs container is not one of good or bad in the general sense there are several reasons to choose containers. 1) Containers reduce the number of shared dependencies between the VM and the application. A container based on alpine will even be using musl libc vs glibc. And a VM running on a private cloud or different cloud providers will require different dependencies. As dependency hell is NP-complete, increasing the number of dependencies dramatically increases the effort of testing, trouble shooting, and maintaining a system. The simplified contract between a container and it's host vs a binary on a VM reduces that cost. 2) Instantiation time is much longer for a VM than a container. 3) Container management systems like k8 etc... tend to have more robust health checking, service registration, and recovery tools. If one is using containers already the cost of maintaining a parallel infrastructure for VMs is often high. 4) Developers can often run a container on their desktops/laptops and iterate faster as they avoid the challenges and costs in maintaining dependencies called out in #1 or the expense and time of spinning up development instances. There are companies and groups that use containers and container management systems because they are the hot new thing, but even before the public cloud or even high density VMs were a reality I was on a team that leverage Linux VServers for all core services. It is still the only company I have ever been at where we could actually switch over to a cold DR site with everything from DNS to Oracle EBS being up in less than half an hour. All with only using rsync and tar for a publicly traded company. You can get around package version selection being NP-complete by putting on other constraints like enforcing semantic versioning and auto updates. But IMHO using namespaces(containers) is one solution one should consider. Here is a link to a paper that will describe the package dependency problem generalizes to the NP-complete SAT problem. For me, containers are a way to limit that problem to being one of the container host only needing satisfy a contract for a far more limited set of needs (kernel ABI, network, storage). https://hal.inria.fr/hal-00697463 https://hal.inria.fr/hal-00697463 Your needs and challenges may be different, but containers tend to be a good option for many of the needs I have run across.