8 ms·

"When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up act

• gavinray 6y ago
  Containerization and container orchestration platforms are only partly about scalability. The primary appeal for me is ease of deployment and reproducibility. This is why I develop everything in Docker Compose locally. Maybe the equivalent here would be something like Guix or Nix for declaratively writing the entire state of all the desired system packages and services + versions but honestly (without personal experience using these) they seem harder than containers.
  • tomberek 6y ago
    Nix/NixOS for this purpose is very nice.
  • ralmeida 6y ago
    Exactly! Other non-scalability concerns they address (specifically talking about Kubernetes here) is a primitive amount of monitoring/observability; no-downtime updates (rolling updates); liveness/readiness probes; primitive service discovery and load balancing; resiliency to any one single host failing (even if the total compute power could easily fit into a single bigger server).
    • dsr_ 6y ago
      Which of these are things you want on your house server? That's what the article author is writing about, and what I am writing about. I do not need an octopus conducting a herd of elephants.
      • p_l 6y ago
        I recently ended up setting up "classic" server again after a significant time keeping mostly containerized infrastructure on k8s. Never again, the amount of things that are simply harder in comparison is staggering.
        • Scramblejams 6y ago
          Sorry to sound pedantic, but what was harder? Containerized infra or a classic server? I assume the former but wanted to be sure.
          • arcticfox 6y ago
            To me it sounds like the latter, a classic server, which I agree... After getting comfortable with containerized deployment, "classic" servers are a huge pain
            • cecja 6y ago
              it honestly sounds just like you don't know what you are doing and docker was made for you
          • p_l 6y ago
            "Classic" approach turned out to be maddeningly harder. Everything, even inside single "application", going slightly off the reservation. Services that would die in stupid ways. Painful configuration that would have been abstracted out were I running containers on k8s (some benefits might be realized with Docker compose, but docker on its own is much more brittle than k8s). So much SSH-ing to a node to tweak things. apt-get fscking the server. Etc. Oh, and logging being shitshow.
  • dsr_ 6y ago
    I'm not deploying; this is the server. I do backups, and I keep config in git. Reproducibility? This is the server. I will restore from backups. There is no point in scaling. If you want to argue that containerization and VMs are portable and deployable and all that, I agree. This is not a reasonable place to do that extra work.
    • awill 6y ago
      Not sure about reproducibility. If the HD fails, sure, restore from backup. But what if the motherboard fails, and you buy/build a completely new machine. Does a backup work then, even if all the hardware is different? That's where a container makes restoring easier.
      • aflag 6y ago
        Why wouldn't it? Unless you are changing architecture after a component of your system dies, there's no reason your old binaries would not work.
        • OJFord 6y ago
          Drivers, config you missed/didn't realise was relevant/wasn't needed before, IDs (e.g. disks), etc. Nix or aconfmgr (for Arch) help. I still like containers for this though. Scalability doesn't mean I'm fooling myself into thinking hundreds of thousands of people are reading my blog, it means my personal use can outgrow the old old PC 'server' it's on and spill into the new old one, for example. Or that, for simplicity of configuration, each disk will be (the sole disk) mounted by a Pi.
          • aflag 6y ago
            There's more than one way to skin a cat. If you're running something as simple and low profile as OP suggested, all you need to backup from the system are the packages you installed and a handful of configurations you changed in /etc. That could be in ansible, but it could be just a .sh file, really. You'll also need a backup of the actual data, not the entire /. Although, even if all you did was backup the entire / there's a good chance it would work even if you try to recover it in new hardware. The services metioned by OP don't need to talk to each other, they are all things that work out of the box by just running apt-get install or equivalent. You don't need anything really fancy and you can set up a new box with part of the services if they are ever taking too much resources (which, for a small setup, will likely never really happen. At least in my experience)
  • ansible 6y ago
    > This is why I develop everything in Docker Compose locally. For a small setup like this, just having a docker compose file in version control is more than sufficient. You can easily leverage services someone else has set up, and the final config is easy to get going again if you need to rebuild the machine due to hardware failure.
  • GordonS 6y ago
    Reproducibility is the key reason I use Docker. Some stuff is really tricky to setup too - like postfix with working TLS, DKIM etc. Before Docker I'd eventually get stuff like this working, then a couple of years something would break and I'd have no clue how to fix it because I hadn't touched it for so long. With Docker (and Compose and Swarm), everything is codified in scripts and config files, all ready to be deployed anywhere.
• Narkov 6y ago
  Sounds like a dream to compromise! Rather than one attack surface, there's like 25. Crack one and you take the lot.
  • toyg 6y ago
    It's a home server, not an enterprise network. Security is a trade-off.
    • Narkov 6y ago
      I would assume the administrator cares enough about their own data to want to do it properly.
  • dsr_ 6y ago
    It provides 25 services. If I were going to deploy 25 machines to provide 25 services, that would not reduce the attack surface... of my house server.
    • Narkov 6y ago
      With no segregation, you crack one service you get access to all 25 services and their data. With segregation, you crack one service you get access to just that container/physical service.
      • kortilla 6y ago
        Linux has had support for multiple users for quite some time now. Popping a standard service process doesn’t get you any more than the privileges of the running user, which is usually scoped to that service alone. Install Ubuntu, Postgres+apache2, then su to www-data and try read some data from the postgres data directory to see what I mean.
  • ex_amazon_sde 6y ago
    Plus the additional huge attack surface of docker, docker hub and k8s
• rektide 6y ago
  > Containerization is not a win here. Where's the second machine to fail over to? It would probably take an hour or less to add another node to this set up. The author made some choices that block scaling, & changing that, installing service-lb, local-provisioner, would take a bunch of that time. Finding a resilient replacement for local-provisioner is something I wish we were better at, but the folk at Rook.io have a pretty good start on this. To me, the real hope is that we move more and more of our configuration into Kubernetes state. Building a container with a bunch of baked in configuration is one thing, but I hope we are headed towards a more "cloud native" system, where the email server is run not as containers, but as an operator, where configuration is kept in Kubernetes, and the operator goes out & configures the containers to run based on that. I agree that running a bunch of service on a Debian box with a couple different releases (testing/unstable) pinned into apt is not really that hard. But I am very excited to stop managing these pets. And I am very hopeful that we can start moving more and more of our configuration from /etc/whateverd/foo.conf files into something centrally & consistently managed. The services themselves all require special unique management today, & the hope, the dream, is that we get something more like big cloud style dashboards, where each of these services can be managed via common Kubernetes tools, that apply across all our services.
  • dsr_ 6y ago
    "But I am very excited to stop managing these pets." When you have a herd of cattle which is of size 1, it's a pet. You don't get any efficiencies from branding them all with laser-scannable barcodes, an 8-place milking machine, or an automatic silage manager. You still need to call the vet, and the vet needs to know what they are doing.
    • rektide 6y ago
      Having a consistent, managed experience with good top down controls is, in my world, far more efficient than tackling each service like a brand new problem to manage & operate independently.
      • dsr_ 6y ago
• shamallow 6y ago
  It is because at the time I was doing a lot of python development, and I was (and still) using my server as a dev workstation. Isolation with virtualenv was not great and many projects were needing conflicting versions of system package, or newer version than what Debian stable had. Lot of the issue was me messing around \o/ "just having the Kubernetes server components running add a 10% CPU on my Intel(R) Atom(TM) CPU C2338 @ 1.74GHz." Containerization is not a win here. Where's the second machine to fail over to? I think it is worth it in order to get a centralized control plane for everything and automatic build and deployment for eveything. But I agree with you, some apps (postfix, dovecot) doesn't feel great inside a container (Sharing data with UID issue is mewh, postfix with multiprocess design also...) I just wanted to have everything manage into containers, as they were the last ones, so I moved them into.
  • pdonis 6y ago
    > I was (and still) using my server as a dev workstation This seems like a very bad idea, and I'm not at all surprised you had problems. But it doesn't look like the problems were with the server part; if your machine had only been a server you could have avoided all the stuff about needing to pull from unstable. So I don't think "don't put all the server stuff on one machine" is the real takeaway from your experience; I think the real takeaway is "don't use the same machine as both a server and a dev workstation".
    • daniellarusso 6y ago
      It is a bad idea, but my third web development job, my workstation was also the company’s secondary public DNS server, so I’ve seen worse things.
      • bityard 6y ago
        Wow! More stories about _this_ place, please...
        • daniellarusso 6y ago
• deleted 6y ago
  [deleted]
• znpy 6y ago
  > "When you try to CRAM everything (mail, webserver, gitlab, pop3, imap, torrent, owncloud, munin, ...) into a single machine on Debian, you ultimately end-up activating unstable repository to get the latest version of packages and end-up with conflicting versions between softwares to the point that doing an apt-get update && apt-get upgrade is now your nemesis." I use Proxmox to avoid that. Some things I run in VMs (often with Docker containers), some other things I run them in LXC containers (persistant containers that behave like VMs). I can then automation (mostly Proxmox templates and Ansible) to make deployments repeatable. I'm interested in k3s, though, I'll give it a better look :) The next addition will be some form of NAS, either a qnap/synology or a custom build using FreeNAS or Unraid (probably FreeNAS).
• polote 6y ago
  I don't believe it, if you have postgresql package update can't be a breeze. Every major version you need to manually convert the database
  • notsureaboutpg 6y ago
    I _hate_ this about postgresql. Why is it so? It's extremely annoying, imo, and it holds me back from upgrading
  • _joel 6y ago
    You'd need to do that with a container too, if the volume mount for /var/lib/posgresql/data still has the older version of posgres data then if you update the container then that needs to be done too. Alternatively a dump and reimport.
• kenniskrag 6y ago
  What do you do as mitigation if one application has a remote execution? I use containers as a security layer.
  • lmm 6y ago
    Containers have a pretty poor security record. Frankly I'd feel more safe with a non-containerised service running under a non-root user than with a service running as root in a container, not that I'd feel particularly safe with either.
    • cfcosta 6y ago
      Containers != Docker. Podman supports rootless containers for quite a while, without a daemon, and it is compatible with Kubernetes.
• heavyset_go 6y ago
  If you're concerned about resource usage, at least with Swarm or Docker Compose, you get things like health checks, restart policies and replication for free with minimal overhead. Scaling horizontally is easy, too. It's really nice to have your infrastructure described as code versus in some configuration management tool or shell scripts, and for that reason alone I'll use it even on a single machine.
• manigandham 6y ago
  > "The only time I waste the overhead of a virtual machine is when I'm testing out new configurations and don't want to break what I have." So what happens when it does break? What do you do to fix it or do you just skip the update? This is one of the things that containerization solves, amongst the benefits described by others here. Installing Docker, creating volumes, and running/updating images are all single line commands, but with much better isolation and portability.
  • dsr_ 6y ago
    What do I do to fix it? I earn my pay as a sysadmin, of course. "This is one of the things that containerization solves" No. Containerization does not fix a broken system. Only fixing the broken system does that. Containerization lets you apply your fix to all the hosts that you want fixed, and as we have thoroughly established, the number of those hosts in this scenario is one. So far I have been told that containerization fixes configuration problems and allows multiple services to be configured the same way. No container will fix a typo in /etc/dovecot/config.d/20-imap.conf, and no container management system will make nginx.conf look like sendmail.cf. "running/updating images are all single line commands" There is some kind of elven glamour being cast over Kubernetes, Docker, and other container/VM/serverless systems that confuses people about the difference between configuring a service to be useful and managing the lifecycle of that service over a scalable number of machines. Docker cannot update an image that you have not already fixed. This reminds me of the MBA illusion, that claims that all management can be performed most efficiently by a management specialist with no particular knowledge or skills in the actual production process; all of that is irrelevant detail for somebody else to do. I assure you that detailed understanding is the sine qua non of getting things done.
    • manigandham 6y ago
      Docker solves the problem of breaking what you have because you can't. The isolation is built-in. Everything is contained. Of course if you're editing files locally or fixing a Dockerfile it's all the same, but it's not the commonality that's in discussion here. And what's all this about understanding? Who said we don't understand the software? That's your assumption but complete untrue. Managing everything directly isn't a sign of deep understanding, it's simply your preference, one that I and many others don't share. Abstractions exist for a reason. While this particular article might be over-leveraged for their scenario, it says absolutely nothing about the quality and need of those abstractions elsewhere.