20 ms·

Rebuilding My Personal Infrastructure With Alpine Linux and Docker

377 points by kristianp 8y ago

jaimex2 8y ago
I applaud the dedication if this really is just for personal infrastructure.
ruduhudi 8y ago
Gitlab offers free private docker registry with your non-commercial projects and it's fairly easy to build and deploy containers using their CI when hosting the Dockerfiles there.
- dsumenkovic 8y ago
  Thanks for mentioning that. Here's some more info about it, an official doc https://docs.gitlab.com/ee/user/project/container_registry.html https://docs.gitlab.com/ee/user/project/container_registry.h... and a blog post https://about.gitlab.com/2016/05/23/gitlab-container-registry/ https://about.gitlab.com/2016/05/23/gitlab-container-registr....
ratling 8y ago
I blacklist Vultr/choopa in every environment I manage. They make zero effort to deal with botnets and bad actors. They are the first org where I went to the trouble of blacklisting the ASN so new ranges get blocked as soon as maxmind updates. Everything else matches patterns we use. WebPageTest is probably the most hilariously janky application that’s the best at what it does that we use. Standing it up locally so you can test internal stuff is a revolting experience I’ve had several times. TBH I found kubernetes easier to use than docker compose. Mainly because I saw little reason to learn the syntax when I was already using kubernetes yamls and kubeadm makes it so easy to stand up. What you have looks pretty simple though so I may take another look at it. You can actually ship images as tarballs and reimport them. That’s what I do on personal stuff instead of standing up an ECR/registry. As long as you version tag your containers it should be fine. HTTP/2 is pretty much a nonissue for everyone except some janky java clients at this point. We turned it on a couple of years ago with only minimal issues.
- barbecue_sauce 8y ago
  Out of curiosity, how does one go about blacklisting an ASN?
  - steventhedev 8y ago
    Maxmind (and others probably) regularly publish lists of IP blocks mapped to the owning ASN. Basically geoip but instead of mapping to geographical regions, maps it to ASN. If you want to roll your own, you can probably start from a public route reflector and go from there.
    - broknbottle 8y ago
      whois -h whois.radb.net -- -i origin AS20473 | /usr/bin/egrep '^route' | awk '{print $2}'
    - ratling 8y ago
      This, fastly makes this easy to do as well (I don't work for them, just use the product). Maxmind dbs are... awkward to get access to. They are expensive as hell if you buy them directly but usually built into edge service products. F5 had them as well I think.
ggm 8y ago
I am in remarkably similar state to this 'prior' state being FreeBSD 11 hosted, with elements of other distributed service. I also looked to docker and gave up. I like bhyve, and have considered a low-pain migration to bhyve instances to package things into Linux, and then (re)migrate into Docker. A way to avoid pain, and cost of a duplicate site, to build out and integrate. I wish something as logistically simple as docker compose was in a BSD compatible model, to build packages to. I'd like the functional isolation of moving parts, and the redirection stuff. Nice write up. I wonder how many other people are in this 'BSD isn't working for me as well' model?
- frio 8y ago
  I use FreeBSD for my NAS/utility server at home, and am considering switching over to Linux now that ZFS seems pretty stable there. NixOS is my happy place these days.
  - ratling 8y ago
    I had not great experiences with ZFS on Linux. Like broken for a month on a standard distros kernel type issues. For personal it’s probably fine but I wouldn’t use it in prod again.
    - ggm 8y ago
      I'm half-yes half-no on this. I have successfully carried non-root ZFS partitions into Debian. But, I just lost 15TB to an unexpected multipath/iscsid zpool import so now.. I am unsure how I think this story goes. Debian ZFS is not easy to install as root FS which is .. disappointing. It would be nice if it was integrated into the net install .iso as a legit disk install option.
    - qalmakka 8y ago
      I've been using ZoL git on Arch Linux (several systems and configurations) since 2017 without any significant issues. It's probably more stable than mainline Btrfs.
    - lostapathy 8y ago
gravypod 8y ago
I'd suggest people looking to do something similar to this to check out Caddy as a reverse proxy for your services. It'll manage grabbing SSL certs for you and some people have already wrapped it into a nice docker container for you [0]. [0] - https://github.com/wemake-services/caddy-gen https://github.com/wemake-services/caddy-gen
- 0xCMP 8y ago
  Caddy is really nice. I use it on my Synology NAS which can use Docker containers and it works very well.
- lytedev 8y ago
  I recently had the pleasure of using Traefik[0] as my reverse proxy, which similarly handles SSL automatically via LetsEncrypt. Lovely piece of software! [0]: https://traefik.io https://traefik.io
  - gravypod 8y ago
    Yea, I've hears good things about it. My friend runs it as their load balancer + ingress for their k8s cluster.
  - windexh8er 8y ago
    I recently upgraded my simple web facing CV site over to a couple containers and front-ended with Traefik. What's really nice about it is if you run the host as a single-node Swarm you get a lot of freebies with regard to service discovery. I also use Traefik in my internal network to front-end Heimdall [0] in a container. This affords me a very elegant internal network dashboard, bastion host (proxy) and presents all my internal services with LetsEncrypt valid certificates (no more internal self-signed cert warnings). I've been meaning to start blogging again and use this as a first topic. [0]: https://heimdall.site/ https://heimdall.site/
lalos 8y ago
What about performance hit now that all of the services are running on the same server? Might be interesting to monitor and put resource limits on specific services.
- wezm 8y ago
  This is the CPU load on the server right now: https://imgur.com/a/nuM3CqX https://imgur.com/a/nuM3CqX Being on HN front page has pushed it up from a baseline of 7.5% utilisation to about 12.5%.
  - fineline 8y ago
    What kind of instance is that on? Thanks for the article, very informative.
    - wezm 8y ago
      Thanks for reading. It’s a 2 CPU 4Gb RAM instance. It’s probably over specced...
adtac 8y ago
>Note that Alpine Linux doesn’t use systemd, it uses OpenRC. This didn’t factor into my decision at all. systemd has worked well for me on my Arch Linux systems. [...] How does systemd, or any init for that matter, come into the picture if you're running everything inside docker? Containers don't use any init, right? They just execute the binary in the host environment (but containerised). Or am I missing something? Edit: nevermind, OP is using Alpine as the host OS as well.
- unk 8y ago
  The host environment you're running Docker in still needs a services to be executed at boot.
  - adtac 8y ago
    I just ran `docker run -it alpine /bin/sh` and when inside, I ran top. It looks like `/bin/sh` ran as PID 1. No init, just sh (busybox in this case). Edit: Ah you meant the host OS. I can't reply to wezm down below for whatever reason (there's no "reply" button), so I'll just edit this to say I didn't realise that he was using alpine as your host OS as well. I haven't seen many people running it outside Docker, so it's quite interesting.
    - Scarbutt 8y ago
      He means the host OS.
      - adtac 8y ago
        Host operating systems need a init, of course. I was just startled by the irrelevant discussion on alpine's init system when it isn't involved anywhere.
        wezm 8y ago
        Yeah my host is running Alpine as well the containers so it's init needs to start docker, cron, etc.
gcb0 8y ago
In sum, zero benefits from using containers. Still have to decide on a single OS to reduce maintenance problems. Could just have installed all the services (which are all available as packages) and handled the configuration files instead of configuration files+docker file+s3 costs of docker image with nothing but the base os + one package and a configuration file.
- adtac 8y ago
  >configuration files+docker file Dockerfiles are a part of the configuration >s3 costs of docker image Just like how most programs are available as packages in your favourite distribution's package manager, most programs are available as Docker images on Docker hub, which is as gratis as packages built for distros. And for the ones that don't have an image on Docker hub, you can include the Dockerfile to build it on the spot in the production environment. The real benefit comes from having the _ability_ to use a different base distribution if required. While alpine is suitable for most applications, sometimes you might be required to run a different distro; perhaps you're packaging a proprietary application compiled for Glibc. Also, networking isolation is straightforward in Docker. Let's say there's a serious security bug in Postgres that allows you to log in without any password. If someone could perform a RCE in a public-facing website, this would be a catastrophe as they'd be able to exploit the database server as well. With Docker, you can easily put each service in its own network and only give access to the database network to those services that need it. Or you can be even more paranoid and have a separate database and a separate network for each service. One more feature I'm fan of is the ability to run multiple versions of a software. Maybe some applications require Postgres 9.x and some require Postgres 10.x. No problem, run both in separate containers. You can't do that with regular distributions and package managers (at least in any I know of).
  - paulddraper 8y ago
    > You can't do that with regular distributions and package managers (at least in any I know of). Nix would be your best shot at that outside Docker.
    - 8y ago
sureaboutthis 8y ago
And after all that, RedHat dumps Docker and Docker adopts Kubernetes. He should have stuck with the stability of FreeBSD--which he prefers--and used jails.
- icebraining 8y ago
  Jails are just the runtime mechanism, equivalent to cgroups/namespaces/etc in Linux. What Docker provides is the tooling for repeatable container setups, so you can have a single configuration file that you can deploy locally for testing and then on the server. What would you use for doing the same with jails?
- sgt 8y ago
  But Kubernetes still uses Docker, don't they?
  - 1337shadow 8y ago
    Kubernetes relies on containerd which works with docker, rkt or anything that works with containerd. The prefered alternative to docker is rkt.
    - sascha_sl 8y ago
      No they don't. Containerd is the code Moby and Docker are built from, developed at Docker Inc. Kubernetes will take anything that implements CRI. https://cri-o.io/ https://cri-o.io/ https://kubernetes.io/docs/setup/cri/ https://kubernetes.io/docs/setup/cri/
      - 1337shadow 8y ago
        All right ! Thanks for the heads up, deeply appreciated. However, I still stand with rkt rather than docker...
        sascha_sl 8y ago
timClicks 8y ago
Question for those who have migrated to Kubernetes - at what point did you look for something bigger/better than what Docker Compose (or Mesos) can offer?
- gravypod 8y ago
  Once you need more than one machine, more than one engineer, and a desire to use existing tooling. K8S is a platform to build things. Because of this most of the amazing features you have access to are built by the community (service mesh for example). Docker Compose is a mess once you have 10+ services with each having different containers backing it.
gumby 8y ago
He finds ansible annoying — anybody suggest a good alternative? I need to re-spin a set of standard utilities on local hardware from time to time so am looking for the best way to manage the config files (bind, Apache, and the like)
- mongol 8y ago
  I have lately been thinking an Ansible-like approach but using compiled code, maybe using Go, could be a way to go. So I went looking for that, and found Sup. It was not what I had in mind but worth a closer look. https://news.ycombinator.com/item?id=12183370 https://news.ycombinator.com/item?id=12183370 Edit: I have not tried it
  - StreamBright 8y ago
    Same here. We should create an efficient compiled version of Ansible, same feature set much faster execution, single configuration file flavor (only yaml)
    - mongol 8y ago
      My idea is different. Instead of playbook yaml, playbook go source, that compiles to a fat binary that is transferred over ssh. That decreases the dependencies to ssh only on the target hosts (no python). The framework would include an idempotent API that matches all the tasks that Ansible provides.
      - StreamBright 8y ago
        Yes, you are thinking about the same. I would just use YAML for the things that can be configured.
    - GordonS 8y ago
      Is execution speed really an issue? Unless you're parsing millions of lines of YAML, it's difficult to believe that's a bottleneck. I reckon a good benefit would be static typing and intellisense.
nathankunicki 8y ago
I don't necessarilly agree with his decision to build all images by hand as opposed to using those available on Dockerhub, however it's a personal choice and I respect it. Given that he's using docker-compose, I wonder why he's chosen to host his images in a repository at all, instead of just specifying the Dockerfile in the yaml and having them locally.
- poxrud 8y ago
  He mentions that this way he can more easily upload his docker-compose file to AWS ECR when it's ready for production.
  - nathankunicki 8y ago
    Ah, thanks, I missed that comment.
coleifer 8y ago
This is what existential terror looks like. In the face of something incomprehensible and impersonal, OP decides to spend a while rearranging the deck chairs. Nothing has changed, but perhaps OP felt a little better about himself for a while.
- dwaltrip 8y ago
  What is the "incomprehensible and impersonal" thing that the OP is facing? I skimmed the post and I have no idea what you are talking about. If I was feeling adventurous, I might claim you are projecting a tad :)
  - deleted 8y ago
    [deleted]
- manigandham 8y ago
  It's fine to spend time learning with personal projects that aren't measured by forward progress on features. Many people rebuild their blogs every year just to try out new things.
xvilka 8y ago
There is also another option - Consul/Nomad/Vault/Terraform. I only wasn't able[1] to figure how to setup private Docker registry with Terraform/Nomad. The rest, including GitHub organisation, DNS zones, etc - can be defined as code. [1] https://github.com/hashicorp/nomad-guides/issues/50 https://github.com/hashicorp/nomad-guides/issues/50
- Svenstaro 8y ago
  That's an orthogonal option in my opinion. It still allows you to run a full docker setup. It just influences how you run manage everything. That being said, I really like the hashicorp stack and more people should run it.
Ao7bei3s 8y ago
What's the best way to handle updates? I would like to switch to a dockerized setup, but running everything on Debian/stable has the advantage of unattended-upgrades (which has worked absolutely flawlessly for me for years across dozens of snowflake VMs). Not going back to manual upgrades. I tried a Registry/Gitea/Drone.io/Watchtower (all on same host, rebuilding themselves too) pipeline and it worked, but felt patched together. Doing it wrong?
- alias_neo 8y ago
  In-container upgrades seems to be an issue for many, we've had this issue at $work. From what I've seen, the is no consensus on the "right" way to do it. You could run the upgrades in the container and lose them when it's re-upped, or, you need to continuously deploy the containers. This alone is one major reason I'm in favour of statically linked binaries in from-scratch containers, where possible.
- bradrydzewski 8y ago
  If you are building everything on the same machine you could get rid of the registry and watchtower, mount the host machine docker socket into your pipeline, and build and start the image on the host machine from inside your pipeline. I use dockerhub/github/drone/watchtower to automatically publish images and update services. I use dockerhub and github to avoid self-hosting the registry and version control. I also have about a dozen servers (as opposed to single-server in your example). This works really well for me and does not feel patchy. Yes there are moving parts, but fewer moving parts than a full-blown orchestration system.
edoo 8y ago
The real sauce in my infrastructure is simply 'generational' storage. No more clean installs with a backup dir of the system. All my data volumes get mounted into my machine/s as needed. I can swap out all my hard drives with new ones without any downtime too which is nice. I could reinstall my desktop each morning and it would barely set me back 15 minutes. If I also properly stored my general system settings as code (cfengine, puppet, chef, etc) I could practically just launch ephemeral desktop instances and have basically a native experience.
- dman 8y ago
  Could you elaborate a bit what generational storage means? If you get a brand new machine - how does it get configured in 15 mins?
  - edoo 8y ago
    For me a long time ago an upgrade meant buy new hard drives, do a reinstall, and copy either the entire old drive or main data dirs onto the computer. Now my data volumes are separate from the system. I used raid for redundancy on the physical machine with another drive for differential filesystem level backups (snapshot every day for 60 days with only differential storage cost, see rdiff-backup). When I upgrade I just add new drives, assign them into the raid, wait for them to sync, and remove the old drives. By generational I mean my storage and how I do things aren't going to change when I reset things like they used to. All my data is outside the system I'm using and must be attached. I can for example, spin up an AWS machine, run the package install commands for the apps I normally use, VPN to my fileserver and mount my home directory and project volumes, and in quite literally about 15-30 minutes have the exact same environment as I do at my house. CFEngine et al would speed that up quite a bit.
    - dman 8y ago
      Thanks for the explanation, super helpful.
      - 8y ago
fulafel 8y ago
> An aspect of Docker that I don’t really like is that inside the container you are root by default PSA: everyone should turn on the userns option in docker daemon settings. It messes with volume mounts but you can turn it off on a per container basis (userns=host) or arrange a manual uid mapping for the mounts. https://docs.docker.com/v17.12/engine/security/userns-remap/ https://docs.docker.com/v17.12/engine/security/userns-remap/
- wezm 8y ago
  Thank you. I didn’t know about this.
- jxub 8y ago
  Just create a user in the Dockerfile, change to him and chroot the container application directory to the created user
  - fulafel 8y ago
    This only works if you trust all your docker images and audit them for root processes. Even then you still have setuid binaries and other privilege escalation risks, if you inherit from dockerhub images etc.
- barrystaes 8y ago
  I wondered if this is what Unraid already does: yes its mentioned at https://wiki.unraid.net/UnRAID_6/Overview#Containers https://wiki.unraid.net/UnRAID_6/Overview#Containers > The cornerstone of Docker is in its ability to use Linux control groups, namespace isolation, and images to create isolated execution environments in the form of Docker containers. If you want to run dockers at home, i suggest you give it a try. All you need is an old computer and a USB (it runs in RAM). Unraid basically is Linux with a (happy little) webinterface for NAS shares + apps.
  - fulafel 8y ago
    The web page doesn't read like it's enabling userns-remap, to me.
z3t4 8y ago
Virtual machines are supported on most platforms. And gives more freedom then docker.
- icebraining 8y ago
  Docker is just setting a few kernel parameters, VMs run a full OS over the host, there's a cost to that. If you don't need the extra flexibility, there's no point. Mind that you don't need to use Docker to use containers, there's always LXD and others.
  - z3t4 8y ago
    Many virtualization technologies use hardware acceleration supported on modern server grade CPUs. It's not like running on "bare metal" but pretty close. One use case for VM's is running obsolete software, like some old proprietary OS, it's a weird feeling when the software run 100x faster then the hardware it used to run on.
- yatsyk 8y ago
  VMs and containers have pros and cons. Not sure what do you mean by freedom. If I need secure isolation or non-linux guest I would choose VM, but now I use containers mostly because lightness, speed, registry and snapshots.
drej 8y ago
I would love to Alpine all the things, it's just so fast to work with, BUT musl makes things hard, especially if you're over in Pythonland - packages with C dependencies, if built, are only built for glibc, so when installing them on Alpine, one has to have a compiler, development headers etc. Makes things too slow and error prone. Otherwise it's a great distro and I use it for non-Pythonic stuff or for Python with no dependencies.
- Gigablah 8y ago
  With multi-stage Docker builds nowadays it’s easier to have an intermediate builder environment with all the build deps installed.
  - drej 8y ago
    True, but it's still a pain having to do that, all I want is to install a package in an interpreted language, I'm not building a project in a compiled language.
    - yjftsjthsd-h 8y ago
      > all I want is to install a package in an interpreted language, I'm not building a project in a compiled language. I sympathize, but rather sounds like the problem is that you're not using a purely interpreted language anymore.
- StreamBright 8y ago
  That is a great opportunity to creat a list of packages that are working with Musl.
  - majewsky 8y ago
    Not trying to be snarky, but is https://pkgs.alpinelinux.org/packages https://pkgs.alpinelinux.org/packages not what you're looking for?
drdaeman 8y ago
In my experience, "slim" Debian images (like `python:slim`) aren't significantly larger than Alpine-based ones, but save lots of time and headache when something assumes glibc and breaks (or, worse - subtly breaks) with musl (or doesn't have a binary distribution for musl so every time image's built you have to build from source). Also, I'm not sure what are the benefits going `FROM alpine` and installing nginx, than just starting `FROM nginx:alpine`. The latter has benefit of a more straightforward update logic when a new nginx version is released - `docker build` will "just" detect this. It won't notice that Alpine repos have an upgrade, though, and will reuse cached layers for `RUN apk install nginx`. Just saying.
- TicklishTiger 8y ago
  I'm not sure what are the benefits going `FROM alpine` and installing nginx, than just starting `FROM nginx:alpine`. The former hands your balls over to Docker Inc and the "Alpine Linux Development Team". The latter hands your balls over to Docker Inc, the "Alpine Linux Development Team" and a guy or girl called "jkilbride". Edit: No, it does not! See toong's comment below.
  - toong 8y ago
    Where does "jkilbride" come from ? Aren't you confusing `nginx:alpine` with `jkilbride/nginx-alpine` ?
    - TicklishTiger 8y ago
      Yes, I did! This totally voids my comment!
- nickjj 8y ago
  > In my experience, "slim" Debian images (like `python:slim`) aren't significantly larger than Alpine-based ones, but save lots of time and headache. I came to pretty much the same conclusion too. For years I was using Alpine but as of about a year ago I've been going with Debian Slim and haven't looked back. I'd much rather have the confidence of using Debian inside of my images than save 100MB on a base image (which is about what it is in a real web project with lots of dependencies).
deleted 8y ago
[deleted]
chvid 8y ago
I haven't really used docker - so here is a dumb question; suppose one makes a setup like the author here, then what does a deployment of new version look like? Suppose the author updates one of his rails apps and there some database schema modifications. Is that handled by docker? How long does a deployment take? (Minutes, seconds ... basically is the tool able to figure out what is changed and only apply the changes or does it remove the old installation and build the new from scratch?)
- sbhat7 8y ago
  Deployment of a new version would depend upon your setup. Assuming a setup similar to the author, you can have a new Docker images with the new version of your code and run it in parallel. All you have to do after that is point the traffic from the old version to the new version (By just running `docker compose`). If you have a more complex setup, e.g. if by using Kubernetes, you can do things like run both the version at the same time, person A/B testing or have canary deployments to ensure the new version works . Time for deployment would be most likely in seconds unless the setup is complex/convoluted. Schema modifications are another beast. For small use cases, you could run a specialized one time container that performs the modifications, but once you need high availability, you'd have to consider a more complex approach. See https://queue.acm.org/detail.cfm?id=3300018 https://queue.acm.org/detail.cfm?id=3300018
- shaklee3 8y ago
  Docker works in layers. Depending on how your layers are set up, it may be just a single diff from the existing image you already have, which can be quite fast. However, if you change one of the base layers, like Alpine Linux it will rebuild all layers above that. Good container design should make that rare.
humbleMouse 8y ago
It's all fun and games using alpine linux until you run into weird networking issues that are caused by slim images. I was a big fan of slim images until unsolvable bugs started popping up. Like others have said, not much benefit shaving off a few hundred mbs in the age of fiber.
Mister_Snuggles 8y ago
This article was interesting to me because a lot of my personal infrastructure runs on FreeBSD. While I don't host anything publicly accessible, I do have some similar needs. The author mentions the Docker port for FreeBSD. According to the FreeBSD Wiki, it's meant to run Linux Docker images and relies on FreeBSD's Linux ABI layer to do so. To me, this is the wrong approach. FreeBSD already has good container technology, what it really needs are good tools around that. Since the author ended up building his own Docker images, I suspect that he'd be happy with a FreeBSD-equivalent way to declarative build and manage jails.
- waz0wski 8y ago
  iocage is a decent cli utility for managing jails, automating some common operations, and associated zfs bits for storage https://github.com/iocage/iocage https://github.com/iocage/iocage https://dan.langille.org/2015/03/07/getting-started-with-iocage-for-jails-on-freebsd/ https://dan.langille.org/2015/03/07/getting-started-with-ioc...
  - Mister_Snuggles 8y ago
    This does look to be better than ezjail - I'll have to give it a look! The vnet support is something that will be very useful.
vorpalhex 8y ago
I use a somewhat similar setup, although I kept FreeBSD ala FreeNAS. I use a FreeNAS server to manage storage pools and the bare metal box, run Ubuntu vms on top of it, and then manage top level applications in Docker in vms via Portainer. This is nice because the vms get their own ip leases, but can still be controlled and very locked down (or not) depending on their use. Docker volumes are mounted over NFS from the underlying NAS, and the docker level data is backed up with the rest of the NAS.
redsavagefiero 8y ago
This is pretty classic amateur hour. He basically says 'I now know docker and everything looks like a container! Look at how productive I can be by discarding this set piece environment with out of fashion, slow and deliberate provisioning for docker images that I can create once and run forever with auto provisioning and dutch oven magic + an editor!'
oarsinsync 8y ago
> My sites are already quite compact. Firefox tells me this page and all resources is 171KB / 54KB transferred 16323 bytes of text, 6933 bytes of vector graphics, 23,256 bytes. 22KB of content, 171KB total, 87% of the transfer total is potential bloat. It could be worse, but there's almost certainly room for improvement.
ElijahLynn 8y ago
Can use docker-compose up --build with build options in compose.yml to avoid the need for a container registry.

377 points by kristianp 8y ago

I applaud the dedication if this really is just for personal infrastructure.

Gitlab offers free private docker registry with your non-commercial projects and it's fairly easy to build and deploy containers using their CI when hosting the Dockerfiles there.

dsumenkovic 8y ago
Thanks for mentioning that. Here's some more info about it, an official doc https://docs.gitlab.com/ee/user/project/container_registry.html https://docs.gitlab.com/ee/user/project/container_registry.h... and a blog post https://about.gitlab.com/2016/05/23/gitlab-container-registry/ https://about.gitlab.com/2016/05/23/gitlab-container-registr....

ratling 8y ago

I blacklist Vultr/choopa in every environment I manage. They make zero effort to deal with botnets and bad actors. They are the first org where I went to the trouble of blacklisting the ASN so new ranges get blocked as soon as maxmind updates. Everything else matches patterns we use. WebPageTest is probably the most hilariously janky application that’s the best at what it does that we use. Standing it up locally so you can test internal stuff is a revolting experience I’ve had several times. TBH I found kubernetes easier to use than docker compose. Mainly because I saw little reason to learn the syntax when I was already using kubernetes yamls and kubeadm makes it so easy to stand up. What you have looks pretty simple though so I may take another look at it. You can actually ship images as tarballs and reimport them. That’s what I do on personal stuff instead of standing up an ECR/registry. As long as you version tag your containers it should be fine. HTTP/2 is pretty much a nonissue for everyone except some janky java clients at this point. We turned it on a couple of years ago with only minimal issues.

barbecue_sauce 8y ago
Out of curiosity, how does one go about blacklisting an ASN?
- steventhedev 8y ago
  Maxmind (and others probably) regularly publish lists of IP blocks mapped to the owning ASN. Basically geoip but instead of mapping to geographical regions, maps it to ASN. If you want to roll your own, you can probably start from a public route reflector and go from there.
  - broknbottle 8y ago
    whois -h whois.radb.net -- -i origin AS20473 | /usr/bin/egrep '^route' | awk '{print $2}'
  - ratling 8y ago
    This, fastly makes this easy to do as well (I don't work for them, just use the product). Maxmind dbs are... awkward to get access to. They are expensive as hell if you buy them directly but usually built into edge service products. F5 had them as well I think.

ggm 8y ago

I am in remarkably similar state to this 'prior' state being FreeBSD 11 hosted, with elements of other distributed service. I also looked to docker and gave up. I like bhyve, and have considered a low-pain migration to bhyve instances to package things into Linux, and then (re)migrate into Docker. A way to avoid pain, and cost of a duplicate site, to build out and integrate. I wish something as logistically simple as docker compose was in a BSD compatible model, to build packages to. I'd like the functional isolation of moving parts, and the redirection stuff. Nice write up. I wonder how many other people are in this 'BSD isn't working for me as well' model?

frio 8y ago
I use FreeBSD for my NAS/utility server at home, and am considering switching over to Linux now that ZFS seems pretty stable there. NixOS is my happy place these days.
- ratling 8y ago
  I had not great experiences with ZFS on Linux. Like broken for a month on a standard distros kernel type issues. For personal it’s probably fine but I wouldn’t use it in prod again.
  - ggm 8y ago
    I'm half-yes half-no on this. I have successfully carried non-root ZFS partitions into Debian. But, I just lost 15TB to an unexpected multipath/iscsid zpool import so now.. I am unsure how I think this story goes. Debian ZFS is not easy to install as root FS which is .. disappointing. It would be nice if it was integrated into the net install .iso as a legit disk install option.
  - qalmakka 8y ago
    I've been using ZoL git on Arch Linux (several systems and configurations) since 2017 without any significant issues. It's probably more stable than mainline Btrfs.
  - lostapathy 8y ago

gravypod 8y ago

I'd suggest people looking to do something similar to this to check out Caddy as a reverse proxy for your services. It'll manage grabbing SSL certs for you and some people have already wrapped it into a nice docker container for you [0]. [0] - https://github.com/wemake-services/caddy-gen https://github.com/wemake-services/caddy-gen

0xCMP 8y ago
Caddy is really nice. I use it on my Synology NAS which can use Docker containers and it works very well.
lytedev 8y ago
I recently had the pleasure of using Traefik[0] as my reverse proxy, which similarly handles SSL automatically via LetsEncrypt. Lovely piece of software! [0]: https://traefik.io https://traefik.io
- gravypod 8y ago
  Yea, I've hears good things about it. My friend runs it as their load balancer + ingress for their k8s cluster.
- windexh8er 8y ago
  I recently upgraded my simple web facing CV site over to a couple containers and front-ended with Traefik. What's really nice about it is if you run the host as a single-node Swarm you get a lot of freebies with regard to service discovery. I also use Traefik in my internal network to front-end Heimdall [0] in a container. This affords me a very elegant internal network dashboard, bastion host (proxy) and presents all my internal services with LetsEncrypt valid certificates (no more internal self-signed cert warnings). I've been meaning to start blogging again and use this as a first topic. [0]: https://heimdall.site/ https://heimdall.site/

lalos 8y ago

What about performance hit now that all of the services are running on the same server? Might be interesting to monitor and put resource limits on specific services.

wezm 8y ago
This is the CPU load on the server right now: https://imgur.com/a/nuM3CqX https://imgur.com/a/nuM3CqX Being on HN front page has pushed it up from a baseline of 7.5% utilisation to about 12.5%.
- fineline 8y ago
  What kind of instance is that on? Thanks for the article, very informative.
  - wezm 8y ago
    Thanks for reading. It’s a 2 CPU 4Gb RAM instance. It’s probably over specced...

adtac 8y ago

>Note that Alpine Linux doesn’t use systemd, it uses OpenRC. This didn’t factor into my decision at all. systemd has worked well for me on my Arch Linux systems. [...] How does systemd, or any init for that matter, come into the picture if you're running everything inside docker? Containers don't use any init, right? They just execute the binary in the host environment (but containerised). Or am I missing something? Edit: nevermind, OP is using Alpine as the host OS as well.

unk 8y ago
The host environment you're running Docker in still needs a services to be executed at boot.
- adtac 8y ago
  I just ran `docker run -it alpine /bin/sh` and when inside, I ran top. It looks like `/bin/sh` ran as PID 1. No init, just sh (busybox in this case). Edit: Ah you meant the host OS. I can't reply to wezm down below for whatever reason (there's no "reply" button), so I'll just edit this to say I didn't realise that he was using alpine as your host OS as well. I haven't seen many people running it outside Docker, so it's quite interesting.
  - Scarbutt 8y ago
    He means the host OS.
    - adtac 8y ago
      Host operating systems need a init, of course. I was just startled by the irrelevant discussion on alpine's init system when it isn't involved anywhere.
      - wezm 8y ago
        Yeah my host is running Alpine as well the containers so it's init needs to start docker, cron, etc.

gcb0 8y ago

In sum, zero benefits from using containers. Still have to decide on a single OS to reduce maintenance problems. Could just have installed all the services (which are all available as packages) and handled the configuration files instead of configuration files+docker file+s3 costs of docker image with nothing but the base os + one package and a configuration file.

adtac 8y ago
>configuration files+docker file Dockerfiles are a part of the configuration >s3 costs of docker image Just like how most programs are available as packages in your favourite distribution's package manager, most programs are available as Docker images on Docker hub, which is as gratis as packages built for distros. And for the ones that don't have an image on Docker hub, you can include the Dockerfile to build it on the spot in the production environment. The real benefit comes from having the _ability_ to use a different base distribution if required. While alpine is suitable for most applications, sometimes you might be required to run a different distro; perhaps you're packaging a proprietary application compiled for Glibc. Also, networking isolation is straightforward in Docker. Let's say there's a serious security bug in Postgres that allows you to log in without any password. If someone could perform a RCE in a public-facing website, this would be a catastrophe as they'd be able to exploit the database server as well. With Docker, you can easily put each service in its own network and only give access to the database network to those services that need it. Or you can be even more paranoid and have a separate database and a separate network for each service. One more feature I'm fan of is the ability to run multiple versions of a software. Maybe some applications require Postgres 9.x and some require Postgres 10.x. No problem, run both in separate containers. You can't do that with regular distributions and package managers (at least in any I know of).
- paulddraper 8y ago
  > You can't do that with regular distributions and package managers (at least in any I know of). Nix would be your best shot at that outside Docker.
  - 8y ago

sureaboutthis 8y ago

And after all that, RedHat dumps Docker and Docker adopts Kubernetes. He should have stuck with the stability of FreeBSD--which he prefers--and used jails.

icebraining 8y ago
Jails are just the runtime mechanism, equivalent to cgroups/namespaces/etc in Linux. What Docker provides is the tooling for repeatable container setups, so you can have a single configuration file that you can deploy locally for testing and then on the server. What would you use for doing the same with jails?
sgt 8y ago
But Kubernetes still uses Docker, don't they?
- 1337shadow 8y ago
  Kubernetes relies on containerd which works with docker, rkt or anything that works with containerd. The prefered alternative to docker is rkt.
  - sascha_sl 8y ago
    No they don't. Containerd is the code Moby and Docker are built from, developed at Docker Inc. Kubernetes will take anything that implements CRI. https://cri-o.io/ https://cri-o.io/ https://kubernetes.io/docs/setup/cri/ https://kubernetes.io/docs/setup/cri/
    - 1337shadow 8y ago
      All right ! Thanks for the heads up, deeply appreciated. However, I still stand with rkt rather than docker...
      - sascha_sl 8y ago

timClicks 8y ago

Question for those who have migrated to Kubernetes - at what point did you look for something bigger/better than what Docker Compose (or Mesos) can offer?

gravypod 8y ago
Once you need more than one machine, more than one engineer, and a desire to use existing tooling. K8S is a platform to build things. Because of this most of the amazing features you have access to are built by the community (service mesh for example). Docker Compose is a mess once you have 10+ services with each having different containers backing it.

gumby 8y ago

He finds ansible annoying — anybody suggest a good alternative? I need to re-spin a set of standard utilities on local hardware from time to time so am looking for the best way to manage the config files (bind, Apache, and the like)

mongol 8y ago
I have lately been thinking an Ansible-like approach but using compiled code, maybe using Go, could be a way to go. So I went looking for that, and found Sup. It was not what I had in mind but worth a closer look. https://news.ycombinator.com/item?id=12183370 https://news.ycombinator.com/item?id=12183370 Edit: I have not tried it
- StreamBright 8y ago
  Same here. We should create an efficient compiled version of Ansible, same feature set much faster execution, single configuration file flavor (only yaml)
  - mongol 8y ago
    My idea is different. Instead of playbook yaml, playbook go source, that compiles to a fat binary that is transferred over ssh. That decreases the dependencies to ssh only on the target hosts (no python). The framework would include an idempotent API that matches all the tasks that Ansible provides.
    - StreamBright 8y ago
      Yes, you are thinking about the same. I would just use YAML for the things that can be configured.
  - GordonS 8y ago
    Is execution speed really an issue? Unless you're parsing millions of lines of YAML, it's difficult to believe that's a bottleneck. I reckon a good benefit would be static typing and intellisense.

nathankunicki 8y ago

I don't necessarilly agree with his decision to build all images by hand as opposed to using those available on Dockerhub, however it's a personal choice and I respect it. Given that he's using docker-compose, I wonder why he's chosen to host his images in a repository at all, instead of just specifying the Dockerfile in the yaml and having them locally.

poxrud 8y ago
He mentions that this way he can more easily upload his docker-compose file to AWS ECR when it's ready for production.
- nathankunicki 8y ago
  Ah, thanks, I missed that comment.

coleifer 8y ago

This is what existential terror looks like. In the face of something incomprehensible and impersonal, OP decides to spend a while rearranging the deck chairs. Nothing has changed, but perhaps OP felt a little better about himself for a while.

dwaltrip 8y ago
What is the "incomprehensible and impersonal" thing that the OP is facing? I skimmed the post and I have no idea what you are talking about. If I was feeling adventurous, I might claim you are projecting a tad :)
- deleted 8y ago
  [deleted]
manigandham 8y ago
It's fine to spend time learning with personal projects that aren't measured by forward progress on features. Many people rebuild their blogs every year just to try out new things.

xvilka 8y ago

There is also another option - Consul/Nomad/Vault/Terraform. I only wasn't able[1] to figure how to setup private Docker registry with Terraform/Nomad. The rest, including GitHub organisation, DNS zones, etc - can be defined as code. [1] https://github.com/hashicorp/nomad-guides/issues/50 https://github.com/hashicorp/nomad-guides/issues/50

Svenstaro 8y ago
That's an orthogonal option in my opinion. It still allows you to run a full docker setup. It just influences how you run manage everything. That being said, I really like the hashicorp stack and more people should run it.

Ao7bei3s 8y ago

What's the best way to handle updates? I would like to switch to a dockerized setup, but running everything on Debian/stable has the advantage of unattended-upgrades (which has worked absolutely flawlessly for me for years across dozens of snowflake VMs). Not going back to manual upgrades. I tried a Registry/Gitea/Drone.io/Watchtower (all on same host, rebuilding themselves too) pipeline and it worked, but felt patched together. Doing it wrong?

alias_neo 8y ago
In-container upgrades seems to be an issue for many, we've had this issue at $work. From what I've seen, the is no consensus on the "right" way to do it. You could run the upgrades in the container and lose them when it's re-upped, or, you need to continuously deploy the containers. This alone is one major reason I'm in favour of statically linked binaries in from-scratch containers, where possible.
bradrydzewski 8y ago
If you are building everything on the same machine you could get rid of the registry and watchtower, mount the host machine docker socket into your pipeline, and build and start the image on the host machine from inside your pipeline. I use dockerhub/github/drone/watchtower to automatically publish images and update services. I use dockerhub and github to avoid self-hosting the registry and version control. I also have about a dozen servers (as opposed to single-server in your example). This works really well for me and does not feel patchy. Yes there are moving parts, but fewer moving parts than a full-blown orchestration system.

edoo 8y ago

The real sauce in my infrastructure is simply 'generational' storage. No more clean installs with a backup dir of the system. All my data volumes get mounted into my machine/s as needed. I can swap out all my hard drives with new ones without any downtime too which is nice. I could reinstall my desktop each morning and it would barely set me back 15 minutes. If I also properly stored my general system settings as code (cfengine, puppet, chef, etc) I could practically just launch ephemeral desktop instances and have basically a native experience.

dman 8y ago
Could you elaborate a bit what generational storage means? If you get a brand new machine - how does it get configured in 15 mins?
- edoo 8y ago
  For me a long time ago an upgrade meant buy new hard drives, do a reinstall, and copy either the entire old drive or main data dirs onto the computer. Now my data volumes are separate from the system. I used raid for redundancy on the physical machine with another drive for differential filesystem level backups (snapshot every day for 60 days with only differential storage cost, see rdiff-backup). When I upgrade I just add new drives, assign them into the raid, wait for them to sync, and remove the old drives. By generational I mean my storage and how I do things aren't going to change when I reset things like they used to. All my data is outside the system I'm using and must be attached. I can for example, spin up an AWS machine, run the package install commands for the apps I normally use, VPN to my fileserver and mount my home directory and project volumes, and in quite literally about 15-30 minutes have the exact same environment as I do at my house. CFEngine et al would speed that up quite a bit.
  - dman 8y ago
    Thanks for the explanation, super helpful.
    - 8y ago

fulafel 8y ago

> An aspect of Docker that I don’t really like is that inside the container you are root by default PSA: everyone should turn on the userns option in docker daemon settings. It messes with volume mounts but you can turn it off on a per container basis (userns=host) or arrange a manual uid mapping for the mounts. https://docs.docker.com/v17.12/engine/security/userns-remap/ https://docs.docker.com/v17.12/engine/security/userns-remap/

wezm 8y ago
Thank you. I didn’t know about this.
jxub 8y ago
Just create a user in the Dockerfile, change to him and chroot the container application directory to the created user
- fulafel 8y ago
  This only works if you trust all your docker images and audit them for root processes. Even then you still have setuid binaries and other privilege escalation risks, if you inherit from dockerhub images etc.
barrystaes 8y ago
I wondered if this is what Unraid already does: yes its mentioned at https://wiki.unraid.net/UnRAID_6/Overview#Containers https://wiki.unraid.net/UnRAID_6/Overview#Containers > The cornerstone of Docker is in its ability to use Linux control groups, namespace isolation, and images to create isolated execution environments in the form of Docker containers. If you want to run dockers at home, i suggest you give it a try. All you need is an old computer and a USB (it runs in RAM). Unraid basically is Linux with a (happy little) webinterface for NAS shares + apps.
- fulafel 8y ago
  The web page doesn't read like it's enabling userns-remap, to me.

z3t4 8y ago

Virtual machines are supported on most platforms. And gives more freedom then docker.

icebraining 8y ago
Docker is just setting a few kernel parameters, VMs run a full OS over the host, there's a cost to that. If you don't need the extra flexibility, there's no point. Mind that you don't need to use Docker to use containers, there's always LXD and others.
- z3t4 8y ago
  Many virtualization technologies use hardware acceleration supported on modern server grade CPUs. It's not like running on "bare metal" but pretty close. One use case for VM's is running obsolete software, like some old proprietary OS, it's a weird feeling when the software run 100x faster then the hardware it used to run on.
yatsyk 8y ago
VMs and containers have pros and cons. Not sure what do you mean by freedom. If I need secure isolation or non-linux guest I would choose VM, but now I use containers mostly because lightness, speed, registry and snapshots.

drej 8y ago

I would love to Alpine all the things, it's just so fast to work with, BUT musl makes things hard, especially if you're over in Pythonland - packages with C dependencies, if built, are only built for glibc, so when installing them on Alpine, one has to have a compiler, development headers etc. Makes things too slow and error prone. Otherwise it's a great distro and I use it for non-Pythonic stuff or for Python with no dependencies.

Gigablah 8y ago
With multi-stage Docker builds nowadays it’s easier to have an intermediate builder environment with all the build deps installed.
- drej 8y ago
  True, but it's still a pain having to do that, all I want is to install a package in an interpreted language, I'm not building a project in a compiled language.
  - yjftsjthsd-h 8y ago
    > all I want is to install a package in an interpreted language, I'm not building a project in a compiled language. I sympathize, but rather sounds like the problem is that you're not using a purely interpreted language anymore.
StreamBright 8y ago
That is a great opportunity to creat a list of packages that are working with Musl.
- majewsky 8y ago
  Not trying to be snarky, but is https://pkgs.alpinelinux.org/packages https://pkgs.alpinelinux.org/packages not what you're looking for?

drdaeman 8y ago

In my experience, "slim" Debian images (like `python:slim`) aren't significantly larger than Alpine-based ones, but save lots of time and headache when something assumes glibc and breaks (or, worse - subtly breaks) with musl (or doesn't have a binary distribution for musl so every time image's built you have to build from source). Also, I'm not sure what are the benefits going `FROM alpine` and installing nginx, than just starting `FROM nginx:alpine`. The latter has benefit of a more straightforward update logic when a new nginx version is released - `docker build` will "just" detect this. It won't notice that Alpine repos have an upgrade, though, and will reuse cached layers for `RUN apk install nginx`. Just saying.

TicklishTiger 8y ago
I'm not sure what are the benefits going `FROM alpine` and installing nginx, than just starting `FROM nginx:alpine`. The former hands your balls over to Docker Inc and the "Alpine Linux Development Team". The latter hands your balls over to Docker Inc, the "Alpine Linux Development Team" and a guy or girl called "jkilbride". Edit: No, it does not! See toong's comment below.
- toong 8y ago
  Where does "jkilbride" come from ? Aren't you confusing `nginx:alpine` with `jkilbride/nginx-alpine` ?
  - TicklishTiger 8y ago
    Yes, I did! This totally voids my comment!
nickjj 8y ago
> In my experience, "slim" Debian images (like `python:slim`) aren't significantly larger than Alpine-based ones, but save lots of time and headache. I came to pretty much the same conclusion too. For years I was using Alpine but as of about a year ago I've been going with Debian Slim and haven't looked back. I'd much rather have the confidence of using Debian inside of my images than save 100MB on a base image (which is about what it is in a real web project with lots of dependencies).

deleted 8y ago

[deleted]

chvid 8y ago

I haven't really used docker - so here is a dumb question; suppose one makes a setup like the author here, then what does a deployment of new version look like? Suppose the author updates one of his rails apps and there some database schema modifications. Is that handled by docker? How long does a deployment take? (Minutes, seconds ... basically is the tool able to figure out what is changed and only apply the changes or does it remove the old installation and build the new from scratch?)

sbhat7 8y ago
Deployment of a new version would depend upon your setup. Assuming a setup similar to the author, you can have a new Docker images with the new version of your code and run it in parallel. All you have to do after that is point the traffic from the old version to the new version (By just running `docker compose`). If you have a more complex setup, e.g. if by using Kubernetes, you can do things like run both the version at the same time, person A/B testing or have canary deployments to ensure the new version works . Time for deployment would be most likely in seconds unless the setup is complex/convoluted. Schema modifications are another beast. For small use cases, you could run a specialized one time container that performs the modifications, but once you need high availability, you'd have to consider a more complex approach. See https://queue.acm.org/detail.cfm?id=3300018 https://queue.acm.org/detail.cfm?id=3300018
shaklee3 8y ago
Docker works in layers. Depending on how your layers are set up, it may be just a single diff from the existing image you already have, which can be quite fast. However, if you change one of the base layers, like Alpine Linux it will rebuild all layers above that. Good container design should make that rare.

humbleMouse 8y ago

It's all fun and games using alpine linux until you run into weird networking issues that are caused by slim images. I was a big fan of slim images until unsolvable bugs started popping up. Like others have said, not much benefit shaving off a few hundred mbs in the age of fiber.

Mister_Snuggles 8y ago

This article was interesting to me because a lot of my personal infrastructure runs on FreeBSD. While I don't host anything publicly accessible, I do have some similar needs. The author mentions the Docker port for FreeBSD. According to the FreeBSD Wiki, it's meant to run Linux Docker images and relies on FreeBSD's Linux ABI layer to do so. To me, this is the wrong approach. FreeBSD already has good container technology, what it really needs are good tools around that. Since the author ended up building his own Docker images, I suspect that he'd be happy with a FreeBSD-equivalent way to declarative build and manage jails.

waz0wski 8y ago
iocage is a decent cli utility for managing jails, automating some common operations, and associated zfs bits for storage https://github.com/iocage/iocage https://github.com/iocage/iocage https://dan.langille.org/2015/03/07/getting-started-with-iocage-for-jails-on-freebsd/ https://dan.langille.org/2015/03/07/getting-started-with-ioc...
- Mister_Snuggles 8y ago
  This does look to be better than ezjail - I'll have to give it a look! The vnet support is something that will be very useful.

vorpalhex 8y ago

I use a somewhat similar setup, although I kept FreeBSD ala FreeNAS. I use a FreeNAS server to manage storage pools and the bare metal box, run Ubuntu vms on top of it, and then manage top level applications in Docker in vms via Portainer. This is nice because the vms get their own ip leases, but can still be controlled and very locked down (or not) depending on their use. Docker volumes are mounted over NFS from the underlying NAS, and the docker level data is backed up with the rest of the NAS.

redsavagefiero 8y ago

This is pretty classic amateur hour. He basically says 'I now know docker and everything looks like a container! Look at how productive I can be by discarding this set piece environment with out of fashion, slow and deliberate provisioning for docker images that I can create once and run forever with auto provisioning and dutch oven magic + an editor!'

oarsinsync 8y ago

> My sites are already quite compact. Firefox tells me this page and all resources is 171KB / 54KB transferred 16323 bytes of text, 6933 bytes of vector graphics, 23,256 bytes. 22KB of content, 171KB total, 87% of the transfer total is potential bloat. It could be worse, but there's almost certainly room for improvement.

ElijahLynn 8y ago

Can use docker-compose up --build with build options in compose.yml to avoid the need for a container registry.