24 ms·

Self-Hosting Dozens of Web Applications and Services on a Single Server

511 points by mattrighetti 5y ago

js4ever 5y ago
I went the opposite direction, 1 vm per service. It's much easier to secure, backup, restore thanks to full isolation.
- sdze 5y ago
  I hope you don't use the VM snapshotting feature as backup method? I had problems when running a database service on it for obvious reasons.
  - js4ever 5y ago
    Nop indeed, I use the native backup tool to create a dump then use borg to save the backup in a remote location
- fersarr 5y ago
  Me too. Yesterday I was trying to add a second app (3 docker-compose services) to one of my 5$/mo droplets and it started impacting and slowing down the other app. Debugging was harder too and could affect both... Perhaps at some point I'll try it again.
  - southerntofu 5y ago
    Sounds like your server may have been swapping? Disk IO is notoriously bad on most VPS so make sure to disable swap and setup oomkiller. In the case of a small VPS, containerization overhead is also not negligible: you may consider to setup everything natively (to your VM) to counteract that.
    - fersarr 5y ago
      Yea, that makes sense! Ty
- candiddevmike 5y ago
  You should look into systemd-nspawn, VM experience without the overhead.
- k8sToGo 5y ago
  But that bloats your backups like crazy. Much better to use something like Ansible to setup the VMs and then just backup the data.
  - rubatuga 5y ago
    True about the backups. I'm thinking once I upgrade to 22.04, I'll be able to use virtio-fs for better efficiency (+ actual dedup!).
rudasn 5y ago
Great post with lots of details. I'm sure I'll try some of tools mentioned. We do something similar, multiple django sites and static sites on a single digital ocean droplet. We use docker for everything though, and have a single nginx listening on 80/443 that routes traffic to the sites by domain. Each site has a separate pg db and everything is glued together / configured / deployed via ansible.
- rajasimon 5y ago
  Do you think I can make this without Docker? I afraid to use docker I don't get it. Why not just systemctrl restart webserivce1 or 2 ?
  - gmfawcett 5y ago
    Of course you can. That approach makes different tradeoffs, of course, but it's entirely possible. I have a "legacy" server with ~20 small/medium, non-containerized applications running on it (Django, older Python frameworks, some Lisp & Scheme stuff, etc.), all running as system services and sitting behind an Nginx front-end. But I can see the merits of containers, and might containerize these apps & services some day.
throwaway744678 5y ago
I (my company) use a similar approach: a single dedicated server with docker containers, by using dokku [0], for an heroku-like self-hosted PaaS. Most of our applications are either: - app developped in-house (django/flask): Procfile + deploy with git push - standard app with a docker image available: deploy directly Dokku comes with useful "service" plugins for databases, auto https (letsencrypt), virtual hosts... Overall, a good experience. [0] https://dokku.com/ https://dokku.com/
- password4321 5y ago
  Pardon my curiosity, but do you pay for Dokku Pro?
  - throwaway744678 5y ago
    I do not.
- throwaway77384 5y ago
  +1 for Dokku here. Been running 10 containers on the second-cheapest Hetzner instance available for years now. Never had any issues. My only, tiny gripe would be excessive space consumption on the somewhat small 20gb SSDs you get with Hetzner VPSs.
  - jakswa 5y ago
    My only, tiny gripe would be excessive space consumption on the somewhat small 20gb SSDs you get with Hetzner VPSs. I was trying to imagine the reason for this. Is it that dokku is similar to heroku, and does a build step upon git-push-to-deploy? So assets/compilation/etc have artifacts and a cache that sticks around to help speed up future pushes/builds?
- conradfr 5y ago
  Same here (not a company but some paying users) but with CapRover. Multiple webapps (Symfony, Phoenix, Magento) on a 20€/m OneProvider server. Databases backups are done by a cron bash script which uploads to a ftp. It works fine, only real downside for my use case is the small downtime after a deploy. I probably would use something else for a frequently deployed pro webapp.
dnzm 5y ago
Nice writeup, and at the core not too different from what I'm doing myself (albeit with the server at home, not colo'd, and the specs are far more modest). The only thing I'd change in your workflow, perhaps, is switching from docker CLI commands to docker-compose. That'd make things a lot more reproducible and easy to read, and if you group relevant containers into a compose file, they're also automagically networked together, which is handy. Your "trick" of grouping TCP ports by the hundreds is something I might steal, I've been using $lastport+1 for a while and now there's a bit of guesswork involved whenever I need to do things that don't go through the proxy. Then again, that's not often, so I might leave it.
rapnie 5y ago
There's projects like Yunohost [0] and Freedombox [1] that aim to make all this easier for the average person. Interested to hear about more such software projects. [0] https://yunohost.org https://yunohost.org [1] https://freedombox.org https://freedombox.org
- rakoo 5y ago
  I've been using yunohost for a few months and nothing could be easier. I wish more guides would spread the word about it instead of promoting manual installation of multiple packages involving the command line and debugging.
- southerntofu 5y ago
  Another one is libreserver.org (formerly freedombone), whose blog is a treasure trove. To my knowledge, it's the only solution that supports Tor/I2P out of the box, and even has experimental branches (looking for testers) to support a local "mesh" setup. These projects are amazing but if you're gonna use them, don't forget to do backups, especially if your system is hosted on a SD card (for example on a raspi). I'm just a little sad there's no standard packaging format across those distros, although i've spoken with yunohost and libreserver maintainers in the past and they seemed somewhat interested in the topic!
YPPH 5y ago
>I have all my HTTPS certificates issued + managed via letsencrypt. It's a terrific service; all I have to do is run snap run certonly --nginx every 3 months and everything is taken care of automatically. Hopefully through cron and not manual invokation! Certbot can safely be executed daily by cron, as it will only attempt renewal if it is required. Automating certificate renewal is an very important step in ensuring availability. I feel like part of the on-call sysadmin initiation process (pre-ACME days) was getting a frantic midnight phone call because someone forgot to renew a certificate... I suspect they are using cron, but this has been omitted unintentionally.
- padde 5y ago
  Except when restarting the services that rely on the certificate doesn't work reliably... but yeah, needs to be automated.
- wiether 5y ago
  A few months ago I tried Nginx Proxy Manager[1] and never looked back. It provides a nice looking UI to manage reverse proxy with Let's Encrypt certs, auto renewal and a few other nice features. [1]: https://nginxproxymanager.com/ https://nginxproxymanager.com/
padde 5y ago
Nice write-up :) I'll probably adopt some of these fancy new tools like nginx or munin (that you call old) some day... still running good old Apache + php, postfix, dovecot, irssi, ... I think my box (in its various reincarnations, first as dedicated server, then as VM on a bigger dedicated server that I shared with friends, and now as cloud VM) has been running since ~2005. Now I feel old ;)
- exitheone 5y ago
  Same here. It's amazing how many services you can easily host if they are on PHP and low traffic. The beauty of PHP is that more services consume no additional resources until they are used. This lets you run many many services without worrying about resource usage. In stark contrast to running everything in containers where you have at least a process per idle service.
  - tluyben2 5y ago
    I ran servers with 100000s of sites/applications per server and indeed php made that possible. I had a complex setup with jails/chroots and selinux and some customizations to the mysql, apache and php source to make sure users couldn't abuse things. With nodejs or whatever, we would've ran a huge loss, now we ran a nice profit. When people go on about environmental issues and such, I cannot help to think all the useless cycles spent by 'modern software'. FaaS helps I guess. But that is not really unlike CGI; with my old setup it is actually practically the same but mine was less flexible (php only).
  - diordiderot 5y ago
    Why does php not need to idle?
vaxman 5y ago
Is your 64MB memory ECC? Why do you RAID-1 NVMe? They are likely to fail simultaneously so maybe make them independent and schedule regular backups from one volume to the other, or to a remote disk drive somewhere.
- TacticalCoder 5y ago
  I don't think OVH offers any dedicated servers without ECC. Their cheaper offspring, Kimsufi (now a separate company since quite a few years) may, although I'm not sure. Even the cheapest OVH dedicated server, starting at about 50 EUR / month, has 16 GB of RAM and it's ECC.
  - dspillett 5y ago
    Many Kimsufi offerings are not fitted with ECC, I don't think some of the lower-end CPUs (the Atoms) even support it at all. Their middle-budget brand SoYouStart seens to only offer machines with ECC, I started using one recently, so I assume you are right and the main brand operates likewise.
- tpetry 5y ago
  Raid-1 is doing exactly what you recommend without any effort. A perfect replica of the disk. And if the other one dies, who cares, the beauty of raid-1 is you don‘t need the other one to have a full copy.
  - vaxman 5y ago
    Again, both NVMe modules are likely to fail simultaneously when used in a RAID-1 mirror on the same chassis, controller and PSU, under the same workload, especially if they are the same model and age.
    - dspillett 5y ago
      I'm not sure this issue is significantly worse for SSDs compared to other drive types, except once they get really old and are close to EOL as defined by the amount written, though I'm happy to be proven wrong if you have some references where the theory has been tested. If you are really worried about that, perhaps artificially stress one of the drives for some days before building the array so it is more likely to go first by enough time to replace and bring the array back from a degraded state?
progx 5y ago
Easy setup with: - traefik (nginx proxy with auto letscencrypt) - portainer (docker container management) - fail2ban (basic security) - logwatch (server / security stats by mail) - munin (server stats) - restic (cloud backup) - unattended-upgrades (auto install security updates) - apticron (weekly info) - n8n (automatisation for e.g. quick info via telegram, if something not work) Run every app that you want in your container.
- kilroy123 5y ago
  This is _very_ close to my exact setup. Though, I never used portainer (just manage a big docker-compose file) is it worth it?
  - razemio 5y ago
    If you are comfortable with the cli you won't need it at all. It over complicates things in my opinion.
  - justin_oaks 5y ago
    Portainer does make certain tasks much easier. I use Portainer to clean up any Docker volumes, containers, and images I no longer need. Doing that for a lot of Docker resources using the command line gets old really fast. I say this as someone who uses docker and docker-compose extensively and I'm very comfortable with a CLI.
    - smallnamespace 5y ago
      Can't you: docker system prune with perhaps a filtering flag attached?
  - Jnr 5y ago
    I also put everything in docker containers, and docker-compose for each thing is a must. It usually pains me to see when popular projects have no decent docker-compose files available and I have to make them. For backups restic is a blessing. And so is Syncthing for getting things to the backup machine quickly (from mobile and non-unix machines).
fersarr 5y ago
Great post! Thanks for sharing all the details.
jareware 5y ago
Echoing the sentiment here, this is a great way to host smaller projects on the cheap, without adding the complexity/price of k8s, Nomad et al! I do the same, and have spent some time automating the backup of such a set of standalone containers [0], in case others also find it useful. [0] https://github.com/jareware/docker-volume-backup https://github.com/jareware/docker-volume-backup
timwis 5y ago
Nice! I do something similar for a few production applications, but I use docker compose to bring the services up, so I don’t have to create the networks manually or remember all the command arguments. Works great.
nodlija 5y ago
Instead of recording the “docker run“ commands, you might want to have a look at docker-compose
- mattbee 5y ago
  Yes I thought that was odd too! OP mentions keeping manual records of "docker run" commands and that it might need some improvement... Well apart from that detail that's exactly what I do for my stack of personal sites. "docker-compose" is really hard to beat for all kinds for setups. And I get 64GB to play with for 50EUR/month from Hetzner, so I don't spend too much time worrying about RAM.
- jlkuester7 5y ago
  Came here to say this! Honestly, my into to docker-compose came when I realised I was starting to do the same thing (saving 'run' commands). Small learning curve for the yaml structure, but well worth it (and honestly I think it is easier to visualize the deployment when it is all nicely laid out in the compose file). Pretty much the only way I use docker now. (Also has a bonus of not really adding a bunch of overhead or complex. Just does what you need for a basic multi-container deploymnet and no more.)
- ytjohn 5y ago
  I had started out the same way, especially if it was a new app and I wasn't familiar with how I really wanted to run it. Some containers expect a fair number of environment variables and multiple mounts. Once I got everything working, I would create a script /svcs with the corresponding docker run command. There's even a cool tool called "runlike" which can create a well formatted command for any running container. https://github.com/lavie/runlike/ https://github.com/lavie/runlike/ But I've got those migrated to docker-compose files these days and I try to start with the docker-compose file instead of going directly into testing out docker run commands.
tweetle_beetle 5y ago
Really good write up and always nice to see the honesty about past mistakes. For those without time to get hands quite as dirty, like me, I've found that CapRover[1] gives just enough UI to fiddle while having sensible defaults. [1] https://caprover.com/ https://caprover.com/
- Aeolun 5y ago
  The only negative about caprover I’ve found is the distinction between containers with persistent storage and not. I don’t want to recreate my whole app just to add persistent storage. Otherwise I’ve been using it without issue for months now.
  - kasra85 5y ago
    Kasra from CapRover. Feel free to open a Feature Request on Github. There is no architectural design limitation that prevents this. It can certainly be added.
lucb1e 5y ago
I wonder what the environmental impact of a 24/7 running system with such high-end specs is. Desktops are worse with a graphics card (just yesterday I noticed my system's power consumption doubles when the GPU turns on: 20W with 100% CPU on all cores and WiFi stressed; 37W with additionally primusrun glxgears running), but desktops only run on demand. Dedicated non-mobile hardware doesn't scale to demand that well and has fairly high idle draw. Don't get me wrong, my server also runs a ton of crap, like git/email/backups/irc/mysql/etc. plus a couple websites and some python scripts (3x doing continuous data logging, 2x providing some service), but it draws less than a traditional light bulb and also runs everything in linux containers. Unless you're doing large data processing tasks like machine learning on a weekly/daily basis, odds are you don't need more than two consumer-grade cores (I've got more since I repurposed an old laptop, but they're very rarely used). Large RAM is nice, though, I do have to say that. My 8GB is stretching it a little, considering that my favorite email server software is a Windows GUI thing and so I run virtualbox to have this guilty pleasure, which consumes about a quarter of the system's RAM just for one service. Aside from Owncast, no idea what requirements that has, I think all of OP's services would be fine with a Raspberry Pi as well, just RAM would be tight with so many individual containers. Especially given the static site hosting, and doubly especially with that offloaded to a CDN, the only dynamic things (screenshot hoster, pastebin) are own use and won't see a dozen concurrent users continuously or anything. Edit: I read over the -track API websites part. I don't know whether those might fit on a Pi as well, but that sounds like various people make use of it and this server is not just single user.
- amaccuish 5y ago
  Out of interest, what is the email server? Your server's EHLO wasn't very revealing.
  - lucb1e 5y ago
    It's an old thing that I should probably not mention for opsec reasons since, honestly, it probably can get pwned (I would be surprised if no intelligence agency has a 0-day for it), and I keep thinking I should switch to linux-based solutions, but it just never floats to the top of my todo list. It's also a fairly big operation since I'd have to coordinate the switchover with some people and I have no idea how to go about IMAP message copying other than telling them to move it over in Thunderbird which seems error-prone. If anyone has a good suggestion for an open source system where I can: - Give people access to control their domains via a web interface (create accounts, set a catch-all, such things) - Have aliases and server-side rules (e.g. blackhole email coming into spamaddr2021december@lucb1e.com; move From "bank" Subject "balance change*" Into inbox/notifications), preferably via the web interface - Optionally: have IMAP and SMTP without a lot of moving parts (I've looked at typical Linux setups before and they're usually quite finicky) Then please let me know!
PascalW 5y ago
I'm doing something similar though I've opted specifically to _do_ use Kubernetes via k0s [0]. It works wonderfully well and allows me to use most things that are available in the k8s ecosystem like cert-manager [1] and external-dns [2]. All configuration is stored in Git and the server itself is basically disposable as it only runs k0s and some very basic stuff like iptables rules. I see this sentiment quite a lot that k8s is too complex for small scale setups, but in my experience it scales down very well to small setups. Granted, you do have to know how to work with k8s, but once you learn that you can apply that knowledge to many different kinds of setups like k0s, bare metal, EKS/AKS etc. [0] https://k0sproject.io/ https://k0sproject.io/ [1] https://cert-manager.io/ https://cert-manager.io/ [2] https://github.com/kubernetes-sigs/external-dns https://github.com/kubernetes-sigs/external-dns
- znpy 5y ago
  I'm playing with k0s and it seems it doesn't play nice with firewalld. With firewalld active containers cannot do networking, not even with hosts jn the same lan. Everything else works beautifully though.
  - PascalW 5y ago
    I'm using iptables myself and it works fine, though you have to make sure that traffic on the kube-bridge interface is allowed. With iptables: -A INPUT -i kube-bridge -j ACCEPT -A OUTPUT -o kube-bridge -j ACCEPT Other than that I've configured iptables to drop all incoming traffic except a few whitelisted ports.
- Already__Taken 5y ago
  Just done a very quick search but it seems like k3s is the better choice so what did you like about k0s?
  - PascalW 5y ago
    Curious what makes you think k3s is the better choice? The only reason I ended up going with k0s was that I had problems getting k3s working well behind a locked down firewall. With k0s that was pretty easy.
jarv 5y ago
Curious what service management are you using for all of these docker run commands, systemd? I see you mention that you might consider Prometheus at some point, is there any process monitoring if one of your services crash?
bottled_poe 5y ago
It’s a good setup, but the question is why run this many services on one machine? The risk profile is too great in my opinion. The software/uptime maintenance effort to keep all users happy would far outweigh the cost of using essentially the same setup with a Kubernetes layer over a few servers. This setup only makes sense for experiments, portfolio, etc.
- southerntofu 5y ago
  What's the "risk profile" and why is it too great? Kubernetes requires dedicated knowledge and can fail in surprising ways. Using a simple server with some established technology can be surprisingly/boringly reliable. Some hosts can reach incredible scale with a just few beefy dedicated machines, given the right software architecture.
  - throwaway984393 5y ago
    The fact that it's a dedicated physical host means that any of a dozen components could fail and bring the box down permanently. The hosting company would have to reassign or build a new server for the customer, which could take from an hour to a week (or more in a pandemic with scarce parts). It depends on their excess capacity and on the machine's specifications. If it was one virtual machine I'd say, sure, just use one box, you can bring up a new one automated in 10 minutes. But if it's physical, it could either last 10 years without going down, or 10 weeks, you never know. Physical machines is precisely the time to use a distributed system as a hedge against expensive failures. Another consideration is cost; at $1200/year you could just keep buying servers and Colo them. It's more work but you get more (old) hardware to play with.
    - southerntofu 5y ago
      > any of a dozen components could fail and bring the box down permanently The same is true with VPS and "cloud" setups. As the OVH fire last year showed, offsite backups are not a luxury. The chances of that, though, are ridiculously low. Higher-end dedicated hosts have very low failure rates due to using higher-grade hardware and rotating the hardware every few years; they also usually have very competent and available technicians on site which are paid by your monthly bill. > If it was one virtual machine I'd say, sure, just use one box, you can bring up a new one automated in 10 minutes. The same is true with dedicated servers. If you house them yourself then sure bringing up hardware replacement is going to take longer. But if you use a managed dedicated hosting solution (as was mentioned in the article) the provisioning times are sensibly similar to that of a VPS (a few minutes).
entropyie 5y ago
I also have a tonne of stuff running on a single, fanless server via docker compose. It is amazing, barely breaks 20% CPU, better uptime than aws ;-) ... One overlooked aspect is that if you use something like my Streacom db4 case, beautiful, fanless, noiseless, you can put it in a living area. Depending on where you live, the power usage can be fully offset against heating costs for your home, making it very efficient.
- greggyb 5y ago
  Just a note on power consumption with regard to heat output. Electric resistive heating is 100% efficient with regard to electricity from the socket. This is the upper limit on efficiency for a computer. This ignores efficiency losses from transmission and from electricity generation. If your electricity is generated by fossil fuels, you are getting much less than 100% efficiency from the energy stored in those fuels. Heat pumps can achieve greater than 100% efficiency relative to electric resistive heating, because they move heat, rather than do work to create heat. These can achieve equivalent heating at 50% of the electricity utilization compared to resistive heating.[0] Local fossil fuel heating (e.g. natural gas furnace) is much more efficient than electric resistive heating. The efficiency of local fossil fuel combustion is close to 100% of the energy content of the fuel, as it is turned directly into heat and that heat is dissipated into the home. More than 60% of the energy content used to generate electricity is lost in the generation process in the US.[1] So, the heat from a computer does offset heating costs, but at less than 100% efficiency. For a small fanless machine, this is not likely to be a large expenditure, but it is valuable to keep in mind that heat from electronics is not "free". [0]: https://www.energy.gov/energysaver/heat-pump-systems https://www.energy.gov/energysaver/heat-pump-systems [1]: https://www.eia.gov/todayinenergy/detail.php?id=44436 https://www.eia.gov/todayinenergy/detail.php?id=44436
southerntofu 5y ago
Thanks for this article, it's great to see people caring for their server (does it have a name?) and not defaulting to the serverless craze. Here's a few thoughts :) > there is some small downtime when I deploy new versions of things since I don't have any load balancing or rolling deployments It's entirely possible to achieve, depending on your stack. `nginx -s reload` will reload the entire config without killing existing connections or inducing undue downtime. So if you can start a second instance of your "webapp" on a separate port/socket (or folder for PHP) and point nginx to it there shouldn't be any downtime involved. > for users that are geographically far away latency can be high That's true, but counter-intuitively, i found unless you're serving huge content (think video or multiple MB pages) it's not a problem. CDN can actually make it worse on a bad connection, because it takes additional roundtrips to resolve the CDN's domain and fetch stuff from there while i already have a connection established to your site. As someone who regularly uses really poor xDSL (from the other side of the atlantic ocean) i have a better experience with sites without a CDN that fit in under 1MB (or even better < 200KB) with as little requests as possible (for the clients that don't support HTTP2). > CloudFlare (...) That may become necessary if I ever have trouble with DDOS attacks I've personally found OVH to be more than capable and willing to deal with DDOS for their customers. OVH has been previously posted on HN for dealing with huge DDOS. That is of course if you have proper caching and you don't have an easy venue for a remote attacker to induce huge CPU/RAM load. For example, Plausible-like analytics can be such an attack vector because every request is logged in a database; something like GoAccess [0] is more resilient, and no logs is even lighter on resources. [0] https://goaccess.io/ https://goaccess.io/
- remram 5y ago
  > there is some small downtime when I deploy new versions of things Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors. I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this? [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]
rcarmo 5y ago
Nice. I've gone down a different path and built https://github.com/piku/piku https://github.com/piku/piku, which I use to run 12-15 web services (and batch workers) off a single-core VM in Azure, plus around the same amount of private services on a 4GB RAM Raspberry Pi. I still use docker-compose and have a k3s cluster running, but I really like the simplicity of deploying via git. Cloud-init for my core setups is here: https://github.com/piku/cloud-init https://github.com/piku/cloud-init
exhaze 5y ago
Three independent, but somewhat related thoughts on this topic: 1). On HOWTO articles about infra (1/2): I'd like to see more articles that lead with requirements, rather than setups that then justify the setup with requirements. Like, congrats, you managed to host a bunch of web applications via containers on a dedicated server. It's really nice for a super personal project and I'm sure it helped OP gain a lot of operational experience across several domains, but I just find this type of article to be "DYI porn" for a highly specific subset of DYI. 2). On HOWTO articles about infra (2/2): Is there any sort of comprehensive infra setup guide out there? Even something that just covers "core" AWS services (CloudFront, EC2, ELB, ECS, Lambda, CloudWatch, CloudTrail, and a few others...) feels like it would be incredibly useful for so many people. 3). "AWS on Rails" - I feel like we're getting close to a time when "AWS on Rails" emerges and we see the birth of a new opinionated tool that says "Most people just want to ship their product, not fiddle with cloud infra. Do it our way and you can focus on product instead of infra"
- Aeolun 5y ago
  > Is there any sort of comprehensive infra setup guide out there? This would be the AWS certification at their different levels.
  - pyrale 5y ago
    I would tend to disagree. First, because not everyone is on AWS. Second, because even for people on AWS, some of a company's infra isn't on AWS (e.g. gmail). Third because not every infra setup work has an AWS tool. To take a personal example, we use multiple infra providers. Some of our infra is gmail (currently working on automating it with terraform). Some of it is other infrastructure providers that aren't US-based. etc.
    - sokoloff 5y ago
      What’s the “automating gmail with terraform” angle? G-suite automation? Or GCP?
      - 5y ago
kalev 5y ago
Be careful with docker and ufw though! Any rules you setup in ufw will be ignored by docker, so exposing a container will always open it up to the public even if you specifically denied access through ufw.
- adamddev1 5y ago
  Very good point, I didn't know this almost got burned by this while learning Docker. What I did was use shared network (for private db connection etc) in a docker-compose file, and then expose the port I wanted to reverse proxy out on by ports: - 127.0.0.1:3000:3000 This way it only exposes it to the local host on the machine without exposing it on the firewall. Then I reverse proxied out port 3000 with NGINX to the outside world. I'm surprised this isn't talked about more in beginner tutorials etc.
- k8sToGo 5y ago
  Technically, Docker is adding iptables rules that are ignored by ufw rather than docker ignoring ufw. To fix, just turn off iptables for the docker daemon and add the rules manually to UFW
  - dawnerd 5y ago
    And if you don’t want to do that because there are some downsides, make sure you setup container networking correctly and don’t just expose ports just to expose ports. Learned that one the hard way when someone exposed redis.
maxicarlos08 5y ago
Hehe, have been doing that on my Raspberry Pi 4 for years now - Nextcloud - Gitlab - A bunch of websites hosted under different domains - Matrix homeserver And the load is still surprisingly low
- thatwasunusual 5y ago
  > And the load is still surprisingly low Which means that you have some extremely unpopular services running...? :) /s (Happy new year!)
kxjkckicllf 5y ago
kxjkckicllfhhh 5y ago
z3t4 5y ago
No need for CDN, if the connection is already setup and the server supports HTTP2 the files can be sent in parallel. TCP and SSL handshake back-n-forth will likely eat up the initial Atlantic latency. Also static content can be cached on the user device. And serving static content is well optimized in most web servers, so you do not need beefy hardware.
kristianpaul 5y ago
Security is not only firewall rules. A LIDS like OSSEC its a good option to install a automatically attacks as well.
- hestefisk 5y ago
  I use Snort. It works like a charm.
  - kristianpaul 5y ago
    Do you decrypt tls ?
sillycube 5y ago
Did you try to upgrade the server os? If you put everything in 1 server, isn't it painful to migrate all services at once?
janvdberg 5y ago
Great write up! Tip I learned this week when I migrated my VPS [1]: when dumping MySQL/MariaDB databases for disaster recovery, dump the grants (aka database user rights) with the pt-show-grants tool. You don't want to import the mysql table itself on a fresh MySQL/MariaDB installation, it's a headache. So dump all your tables to SQL (raw data) and dump your user grants/rights with pt-show-grants (which in itself creates a small SQL file) that you easily import in MySQL/MariaDB. [1] https://j11g.com/2021/12/28/migrating-a-lamp-vps/ https://j11g.com/2021/12/28/migrating-a-lamp-vps/
- daneel_w 5y ago
  If you don't have access to (or don't want to use) the Percona toolkit, you can get all of the grants directly from the `sql_grants` table by installing `common_schema`.
thecrumb 5y ago
I can remember back in the dotcom boom I had Windows small biz server, Exchange, our phone system and hosting about 3 dozen websites on a single server in our breakroom. We ran like that for about 2 years with really no issues other than the occasional bandwidth issue...
- siilats 5y ago
  The new windows 2022 vnext with azure arc is pretty good, RDP is so much better than VNC. Also wamp is so much faster on windows than on linux in my experience on tuning php/python/mysql/postgres stuff. Windows and its shared memory connection to DB on the same machine bypasses pipes, which is where most of the slowdown comes i think
justsomeuser 5y ago
I do something similar: - GCP VM with “Google Container OS”. - “Cloud init” config set on the VM metadata (can easily recreate the VM; no snowflake config via SSH mutations). - My service runs in a docker container, reads/writes to a SQLite file on the host disk. - GCP incrementally snapshots the disk every hour or so, and makes copies to different region. Any disk writes are copied instantly to another zone. - Lets encrypt cert is read from the host disk and the docker container serves HTTPS directly (no proxy). Certificate is renewed with the LE CLI. - The service logs to standard out, this is collected by the Google logs daemon which I can view with the web UI. - Google have their own HTTP uptime monitoring and alerting which sends you an SMS.
- hobo_mark 5y ago
  TIL about their uptime monitor, thanks! https://cloud.google.com/monitoring/uptime-checks https://cloud.google.com/monitoring/uptime-checks
dawnerd 5y ago
I’m using the same server type from OVH in Hillsboro and it’s great. They frequently do sales. Highly recommend anyone interested to go for the nvme upgrade. I rent three now total, one for production sites, one as a development machine strictly for vscode and another for development / staging / misc. waaaaaay overkill but it’s been a huge quality of life improvement. For containers I just use a couple docker compose and a script to run them all. Reverse proxy is through nginx proxy manager which takes care of LE renewals and has a nice web interface.
- Ocha 5y ago
  I’m running collocated servers with simple docker compose file too. Actually https://mailsnag.com https://mailsnag.com is hosted on one of the servers and is using single docker compose file for deployment. Not sure why the author had to record every single docker run command when docker compose file manages all of that for you.
- Axsuul 5y ago
  Where do you go to monitor these Hillsboro OVH sales?
  - dawnerd 5y ago
    They send out emails but also just around major holidays and just randomly. They use this page most of the time from what I can tell https://us.ovhcloud.com/deals https://us.ovhcloud.com/deals Sometimes though they'll send targeted email offers that are not on the site and are good for older hardware but at a really cheap price.
piyiotisk 5y ago
I have a similar setup and wrote how to setup everything here https://costapiy.com/deploy_django_project_linux_server/ https://costapiy.com/deploy_django_project_linux_server/ and here https://github.com/costapiy/server_setup https://github.com/costapiy/server_setup
nyolfen 5y ago
great writeup that overlaps with a lot of stuff i do for myself — a few related tool suggestions from my own experience are caddy for https (absurdly easy config — seriously, i would never go back to nginx if i could help it) and netdata for server monitoring (open source, optionally connectable to a web dashboard with email alerts) [0] https://caddyserver.com/ https://caddyserver.com/ [1] https://github.com/netdata/netdata https://github.com/netdata/netdata
_xnmw 5y ago
This is super useful, thanks for posting. I hate AWS complexity-as-a-service, just give me a damn baremetal machine and let me run and tune my own services. I love having full visibility into raw disk/cpu performance and even benching my hardware. So many useful comparisons here, Bunny.net vs. CloudFlare, and the fact that you got this far without even using fail2ban! Questions (not necessarily for OP, but for anyone) - Give us an idea of the maximum load this server is doing in terms of requests per second? - Anyone choosing AMD Epyc over Intel to avoid spectre/meltdown mitigation slowdowns? - Any use of postfix or other SMTP forwarding via Postmark or another email provider? - What is your Postgres configuration like for your heaviest loads? max_connections? Just wondering how Postgres scales across cores.
- Ameo 5y ago
  Hey, author here! - Request rate is pretty low on average and peaks at around 15 requests/second. That's nothing really and it would probably take 100x or more request rate before I saw any kind of CPU bottlenecking or similar; my upload would probably bottleneck first. The biggest resource usage on the server comes from some expensive database queries made by some of my applications. - I'd definitely be down to use that kind of CPU but it wasn't available at the price point I wanted. Most of my stuff isn't CPU-bound anyway. - I used to self-host my own mailserver and webmail, but to be honest it was a disaster. Even after spending tons of time configuring reverse DNS, DMARC, SKIN, SPF, and the other list of arcane and specialized stuff you need, I still had tons of issues with mail I sent going to spam. I gave up and pay Google to host my mail for my own domains now. - I really haven't done much tuning. MySQL/MariaDB is the DB that gets the most load and its settings are most default. I have put a lot of effort into profiling my apps' query patterns and making sure proper indexes and schemas are in place, though.
2bluesc 5y ago
Traefik Proxy[0] was a game changer for my self-hosted setup of Docker containers. Traefik can read labels applied to Docker containers (easily done with docker-compose) and setup the proxy for you as containers come and go. Even maintains the Lets Encrypt certificates seamlessly. [0] https://traefik.io/traefik/ https://traefik.io/traefik/
- dkryptr 5y ago
  Traefik is great, but their documentation is awful IMO. I moved to Caddy which I prefer currently.
  - rstat1 5y ago
    To be fair, Caddy's docs aren't great either. Last time I tried to deploy it to give SSL to something that didn't have it, took me 2 hrs to figure out the docs enough to get it working.
    - mholt 5y ago
      What could make the docs better? What solved your problem?
      - rstat1 5y ago
        Time and trial-and-error is what eventually solved it. I was trying to configure the auto-HTTPS functionality to use DNS challenges, because my setup forbids any of the others, and this apparently requires plugins that are only supported by community efforts, so that's probably why I had such an issue. The docs around configuring this were not immediately clear to me. Its like one part is over in this corner, the other over there. And the part about DNS provider plugins is off in a completely different direction. I definitely think things could be better organized. Also your one tutorial for getting HTTPS support only mentions the very basic cases. I think it'd be beneficial to see some more advanced tutorials using for instance some of the other LE challenge methods, something like this definitely would've saved me some time.
    - 5y ago
api 5y ago
Yet another modern "cloud native" era developer discovers how overpriced and underpowered most cloud offerings are. Here's what they did folks: they set cloud prices a while back, did not drop them as Moore's Law delivered more power but instead pocketed the profit, and meanwhile ran an entire "cloud native" development push to encourage development practices that maximize cloud lock-in (both through dependence on cloud services and encouraging complexity). Oh, and bandwidth is ludicrously overpriced in the cloud. You can transfer hundreds of terabytes a month outbound for <$500 if you know where to look. No I don't mean from static data either. Look at bare metal hosting.
gogs 5y ago
We kind of moved away from this monolithic setup tho. Oh you can scale-up your hardware like we used to, it will work, until some point. And the day you will need to split your services will come quickly. This is a perfect solution for early stage projects if you're in a startup or for a personel project, but monoliths are....well....monoliths.
- jazzyjackson 5y ago
  why constrain it to early stage? must everyone set their sights on 1 million DAU? monoliths are KISS, and kubernetes can be YAGNI
- Nextgrid 5y ago
  The only thing that is hard/impossible to scale (without tradeoffs) is stateful services like databases, and those won't go away even if you move to microservices. Appservers can be trivially scaled horizontally regardless of whether it's one process that does all (monolith) or a mess of microservices.
Terretta 5y ago
> Whenever I add a new service, I record the exact docker run command I used to launch it and refer to it when I need to re-create the containers for whatever reason. I know that this is an area that could use improvement; it's quite manual right now. I'd be interested to hear about some kind of light-weight solution for this that people have come up with. Later... > I host my own analytics ... They provide an open-source, self-hostable version which I have it deployed with docker-compose. Wouldn't docker-compose be the "already doing it" answer to the first question? It's pretty much your library of 'exact docker run commands', plus sugar to manage the running ones collectively.
- dhc02 5y ago
  > Wouldn't docker-compose be the "already doing it" answer to the first question? It's pretty much your library of 'exact docker run commands', plus sugar to manage the running ones collectively. I'm also confused why he's not using compose. Perhaps he is unaware that when you have multiple services in a compose file, re-running `docker compose up` only restarts containers whose parameters have changed in the compose file. He mentions further down that he's using compose for one service, so obviously he is aware of it. But I can't blame him. When getting up to speed on docker, I found that figuring out compose and how to use it effectively was frustratingly under-documented.
praveen9920 5y ago
I tried something like with a ec2 instance. It may be easy to maintain it as all your hosted services are in one place. Main problem I faced is that the server has become single point of failure which caused lot of issues with customers. It’s really hard to explain why all your services are down at once.
galaxyLogic 5y ago
This all seems quite complicated. Wouldn't it be easier to use Amazon Lambda or similar service. Spend your time creating and optimizing the applications instead of the infrastructure. I've known people who built their own boat and also built the engine for the boat. And it's a great hobby which gives you a great sense of accomplishment. But, if you just need to get some fishing done, buy the boat with the engine already installed.
- BackBlast 5y ago
  Lambda has it's own set of complexities and concerns. It's also not a good fit for his use cases. Now, a containerization service like Fargate would fit his use case better since he is using containers. However, cloud services come with their own complexities and the principle problem they solve (horizontal scaling) doesn't apply to his sites.
- walrus01 5y ago
  maybe the goal is to develop the skills and experience to be able to build more boats. or to work on marine diesel engines. or to do something like rehabilitate gigantic 70 year old wooden tugboats. you can't always rely on third party things to host your stuff if your intention is to become the person who runs the hosting services for other peoples' stuff. same way you can't rely on a boatyard's contractors to maintain your boat if your goal is to become an engineer for medium sized yachts.
- Yaina 5y ago
  I also self-host my own site and some smaller web-services. It's a pain and I don't like it, but the costs are low and predictable and I don't have to write for any proprietary API like Lambda
- chickenpotpie 5y ago
  Personally, I would never use Lambda for a passion project. The uncontrollable pricing model is not friendly for individual developers. What do you do if you wake up in the morning to a $1000 bill? Leave it up and just keep pouring money down the drain or just end up rewriting the whole site to custom infrastructure so you can control the spending anyways?
anderspitman 5y ago
If you want to do something like this from home behind CGNAT, blocked 80/443 ports, etc, you'll probably need to set up tunneling as well: https://github.com/anderspitman/awesome-tunneling https://github.com/anderspitman/awesome-tunneling
throwaway950q 5y ago
Using FreeBSD jails you can easily host hundreds if not thousands of web applications and services on a single server having the specs mentioned by the OP. This practice isn't even noteworthy in FreeBSD land as it is so common.
amiantos 5y ago
Is there a reason people don’t self-host on a small home server? $85/mo is a lot of money if you aren’t making any money off what you’re hosting. If you mostly run CRUD APIs and sites, is there any downside to a low bandwidth home connection? Multiple family members already stream Plex content from my measly 20mbit upstream. Why not run a whole site or multiples?
- laristine 5y ago
  I both maintain a server cabinet at home and rent VMs from providers. There are many reasons people don't self-host: noise, heat, dealing with hardware setup, etc. And if any of your service receives public traffic, you'll have to be very careful in configuring it because you don't want attackers to target your home network or the ISP/police to come at you for illegal traffic on your line. $85/month is a lot of money, but given the number of services he runs on his server, each service now costs roughly $3/month, lower than the price of the cheapest DigitalOcean droplet.
- rubatuga 5y ago
  We made a product called Hoppy Network (https://hoppy.network https://hoppy.network) that fits this use case perfectly. We use WireGuard tunnels to bypass all ISP restrictions, and provide a clean /32 IPv4 and /56 IPv6 using a native network interface. Some of our customers are on Starlink or cellular ISPs. We just launched our annual plans today, cheapest plan is $80/year. Plus, we're sending out a sticker-pack to our first 20 annual plan customers. For the purists out there, we don't filter ICMP packets and don't block any ports. TLDR: You can self-host without your ISP knowing!
- graton 5y ago
  You could. If I did it then I would most likely go with Cloudflare Tunnel( https://blog.cloudflare.com/tunnel-for-everyone/ https://blog.cloudflare.com/tunnel-for-everyone/ ) which is free to use. Since I would want to try to protect myself from a possible DDoS attack. But using that then I think it would be more than feasible to do it from your home network if you have a fast enough upload link.
- wiether 5y ago
awayto 5y ago
Lots of AWS frustration here, so I will mention a project I created to auto generate AWS web applications, covering what I believe are a majority of practical use cases (db, ui, api, users, roles, groups). https://awayto.dev https://awayto.dev https://github.com/keybittech/awayto https://github.com/keybittech/awayto It's not a product, just a project. Check it out if AWS is getting you down! :)
stavros 5y ago
I had the same problem and didn't want to manage things by hand, so I wrote Harbormaster: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster It basically pulls Compose apps from the git repositories you specify, builds the containers and makes sure they're running. Pretty simple and works really well for me.
figassis 5y ago
Just want to point out that docker bypasses ufw, so binding to 0.0.0.0 might not be as safe as you think. Best think to do is not expose any ports at all and have your proxy connect to the container’s internal port.
nirav72 5y ago
I got the same cpu , as part of a hp Z420 workstation I picked up for $250. Added 32 gb ecc memory and able to easily run about 25 containers with plenty of resources left over for VM or two.
jmatthews 5y ago
Thanks for the write up. I use roughly the same set up minus docker. My deploy is usually log in to ssh, git pull and restart the systemctl service.
BrandoElFollito 5y ago
I have been self hosting on and off since about 2000. When I got my fiber link (4 or so years ago) I also started to do home automation and the self hosting got more serious ("national security infrastructure" as my wife calls it, her to-go question is "what will happen with that shit if you die tomorrow?"). I run everything in docker containers, except Wireguard and the backup. I had an idea to "document everything" but the realized there would be no-one to maintain the services (neither my wife not my children are interested in IT). I then decided to document how to untangle themselves from the self-hosting (moving to well-established services, dismantle the home automation, ...). Thi sis going to make more sense in the long term. Since I do not plan to die tomorrow but I do not know for my server, I tried to do a disaster recovery from scratch. It was surprisingly easy - after starting a VM on my laptop (simulating the bare-metal rebuild of my server), I was up and running in about an hour. Docker is simply fantastic for that. I took some notes about the disaster recovery, which I promptly lost so I will have to run this again soon (maybe next week). Actually I will copy the encryption key for my remote backup right away as this is the only thing I won't be able to recover. EDIT: done :) TL;DR: - use docker containers for your services, it is a life-saver and makes everything easier - use a docker-compose script to have everything on file - document either your maintenance or exit strategy - do a quick DRP test on a VM
nokya 5y ago
Thank you for sharing your approach, I learned a few things to improve my own server and highly looking forward to deploying them!
XCSme 5y ago
I have a much simpler set-up, but I don't run everything on one server. I usually prefer having a separate VPS or dedicated server for each service, but this is starting to be a bit more costly than simply renting a powerful machine for everything. One advantage of having everything scattered around is that not everything is going to go down at once (eg: a DDOS or even a traffic spike on your machine will kill all your sites/services if you only have one machine). I use: - Percona PMM[0] - Database/server monitoring - Backups to BackBlaze B2 using cron jobs - Simple deployments, usually using SFTP to upload files (usually for LAMP applications, also running Node.js apps using pm2 that can listen to file changes and restart the app) I am actually working on creating some tools/scripts/guides to aid with self-hosting (for backing up stuff, uptime monitoring, alerting).

511 points by mattrighetti 5y ago

js4ever 5y ago

I went the opposite direction, 1 vm per service. It's much easier to secure, backup, restore thanks to full isolation.

sdze 5y ago
I hope you don't use the VM snapshotting feature as backup method? I had problems when running a database service on it for obvious reasons.
- js4ever 5y ago
  Nop indeed, I use the native backup tool to create a dump then use borg to save the backup in a remote location
fersarr 5y ago
Me too. Yesterday I was trying to add a second app (3 docker-compose services) to one of my 5$/mo droplets and it started impacting and slowing down the other app. Debugging was harder too and could affect both... Perhaps at some point I'll try it again.
- southerntofu 5y ago
  Sounds like your server may have been swapping? Disk IO is notoriously bad on most VPS so make sure to disable swap and setup oomkiller. In the case of a small VPS, containerization overhead is also not negligible: you may consider to setup everything natively (to your VM) to counteract that.
  - fersarr 5y ago
    Yea, that makes sense! Ty
candiddevmike 5y ago
You should look into systemd-nspawn, VM experience without the overhead.
k8sToGo 5y ago
But that bloats your backups like crazy. Much better to use something like Ansible to setup the VMs and then just backup the data.
- rubatuga 5y ago
  True about the backups. I'm thinking once I upgrade to 22.04, I'll be able to use virtio-fs for better efficiency (+ actual dedup!).

rudasn 5y ago

Great post with lots of details. I'm sure I'll try some of tools mentioned. We do something similar, multiple django sites and static sites on a single digital ocean droplet. We use docker for everything though, and have a single nginx listening on 80/443 that routes traffic to the sites by domain. Each site has a separate pg db and everything is glued together / configured / deployed via ansible.

rajasimon 5y ago
Do you think I can make this without Docker? I afraid to use docker I don't get it. Why not just systemctrl restart webserivce1 or 2 ?
- gmfawcett 5y ago
  Of course you can. That approach makes different tradeoffs, of course, but it's entirely possible. I have a "legacy" server with ~20 small/medium, non-containerized applications running on it (Django, older Python frameworks, some Lisp & Scheme stuff, etc.), all running as system services and sitting behind an Nginx front-end. But I can see the merits of containers, and might containerize these apps & services some day.

throwaway744678 5y ago

I (my company) use a similar approach: a single dedicated server with docker containers, by using dokku [0], for an heroku-like self-hosted PaaS. Most of our applications are either: - app developped in-house (django/flask): Procfile + deploy with git push - standard app with a docker image available: deploy directly Dokku comes with useful "service" plugins for databases, auto https (letsencrypt), virtual hosts... Overall, a good experience. [0] https://dokku.com/ https://dokku.com/

password4321 5y ago
Pardon my curiosity, but do you pay for Dokku Pro?
- throwaway744678 5y ago
  I do not.
throwaway77384 5y ago
+1 for Dokku here. Been running 10 containers on the second-cheapest Hetzner instance available for years now. Never had any issues. My only, tiny gripe would be excessive space consumption on the somewhat small 20gb SSDs you get with Hetzner VPSs.
- jakswa 5y ago
  My only, tiny gripe would be excessive space consumption on the somewhat small 20gb SSDs you get with Hetzner VPSs. I was trying to imagine the reason for this. Is it that dokku is similar to heroku, and does a build step upon git-push-to-deploy? So assets/compilation/etc have artifacts and a cache that sticks around to help speed up future pushes/builds?
conradfr 5y ago
Same here (not a company but some paying users) but with CapRover. Multiple webapps (Symfony, Phoenix, Magento) on a 20€/m OneProvider server. Databases backups are done by a cron bash script which uploads to a ftp. It works fine, only real downside for my use case is the small downtime after a deploy. I probably would use something else for a frequently deployed pro webapp.

dnzm 5y ago

Nice writeup, and at the core not too different from what I'm doing myself (albeit with the server at home, not colo'd, and the specs are far more modest). The only thing I'd change in your workflow, perhaps, is switching from docker CLI commands to docker-compose. That'd make things a lot more reproducible and easy to read, and if you group relevant containers into a compose file, they're also automagically networked together, which is handy. Your "trick" of grouping TCP ports by the hundreds is something I might steal, I've been using $lastport+1 for a while and now there's a bit of guesswork involved whenever I need to do things that don't go through the proxy. Then again, that's not often, so I might leave it.

rapnie 5y ago

There's projects like Yunohost [0] and Freedombox [1] that aim to make all this easier for the average person. Interested to hear about more such software projects. [0] https://yunohost.org https://yunohost.org [1] https://freedombox.org https://freedombox.org

rakoo 5y ago
I've been using yunohost for a few months and nothing could be easier. I wish more guides would spread the word about it instead of promoting manual installation of multiple packages involving the command line and debugging.
southerntofu 5y ago
Another one is libreserver.org (formerly freedombone), whose blog is a treasure trove. To my knowledge, it's the only solution that supports Tor/I2P out of the box, and even has experimental branches (looking for testers) to support a local "mesh" setup. These projects are amazing but if you're gonna use them, don't forget to do backups, especially if your system is hosted on a SD card (for example on a raspi). I'm just a little sad there's no standard packaging format across those distros, although i've spoken with yunohost and libreserver maintainers in the past and they seemed somewhat interested in the topic!

YPPH 5y ago

>I have all my HTTPS certificates issued + managed via letsencrypt. It's a terrific service; all I have to do is run snap run certonly --nginx every 3 months and everything is taken care of automatically. Hopefully through cron and not manual invokation! Certbot can safely be executed daily by cron, as it will only attempt renewal if it is required. Automating certificate renewal is an very important step in ensuring availability. I feel like part of the on-call sysadmin initiation process (pre-ACME days) was getting a frantic midnight phone call because someone forgot to renew a certificate... I suspect they are using cron, but this has been omitted unintentionally.

padde 5y ago
Except when restarting the services that rely on the certificate doesn't work reliably... but yeah, needs to be automated.
wiether 5y ago
A few months ago I tried Nginx Proxy Manager[1] and never looked back. It provides a nice looking UI to manage reverse proxy with Let's Encrypt certs, auto renewal and a few other nice features. [1]: https://nginxproxymanager.com/ https://nginxproxymanager.com/

padde 5y ago

Nice write-up :) I'll probably adopt some of these fancy new tools like nginx or munin (that you call old) some day... still running good old Apache + php, postfix, dovecot, irssi, ... I think my box (in its various reincarnations, first as dedicated server, then as VM on a bigger dedicated server that I shared with friends, and now as cloud VM) has been running since ~2005. Now I feel old ;)

exitheone 5y ago
Same here. It's amazing how many services you can easily host if they are on PHP and low traffic. The beauty of PHP is that more services consume no additional resources until they are used. This lets you run many many services without worrying about resource usage. In stark contrast to running everything in containers where you have at least a process per idle service.
- tluyben2 5y ago
  I ran servers with 100000s of sites/applications per server and indeed php made that possible. I had a complex setup with jails/chroots and selinux and some customizations to the mysql, apache and php source to make sure users couldn't abuse things. With nodejs or whatever, we would've ran a huge loss, now we ran a nice profit. When people go on about environmental issues and such, I cannot help to think all the useless cycles spent by 'modern software'. FaaS helps I guess. But that is not really unlike CGI; with my old setup it is actually practically the same but mine was less flexible (php only).
- diordiderot 5y ago
  Why does php not need to idle?

vaxman 5y ago

Is your 64MB memory ECC? Why do you RAID-1 NVMe? They are likely to fail simultaneously so maybe make them independent and schedule regular backups from one volume to the other, or to a remote disk drive somewhere.

TacticalCoder 5y ago
I don't think OVH offers any dedicated servers without ECC. Their cheaper offspring, Kimsufi (now a separate company since quite a few years) may, although I'm not sure. Even the cheapest OVH dedicated server, starting at about 50 EUR / month, has 16 GB of RAM and it's ECC.
- dspillett 5y ago
  Many Kimsufi offerings are not fitted with ECC, I don't think some of the lower-end CPUs (the Atoms) even support it at all. Their middle-budget brand SoYouStart seens to only offer machines with ECC, I started using one recently, so I assume you are right and the main brand operates likewise.
tpetry 5y ago
Raid-1 is doing exactly what you recommend without any effort. A perfect replica of the disk. And if the other one dies, who cares, the beauty of raid-1 is you don‘t need the other one to have a full copy.
- vaxman 5y ago
  Again, both NVMe modules are likely to fail simultaneously when used in a RAID-1 mirror on the same chassis, controller and PSU, under the same workload, especially if they are the same model and age.
  - dspillett 5y ago
    I'm not sure this issue is significantly worse for SSDs compared to other drive types, except once they get really old and are close to EOL as defined by the amount written, though I'm happy to be proven wrong if you have some references where the theory has been tested. If you are really worried about that, perhaps artificially stress one of the drives for some days before building the array so it is more likely to go first by enough time to replace and bring the array back from a degraded state?

progx 5y ago

Easy setup with: - traefik (nginx proxy with auto letscencrypt) - portainer (docker container management) - fail2ban (basic security) - logwatch (server / security stats by mail) - munin (server stats) - restic (cloud backup) - unattended-upgrades (auto install security updates) - apticron (weekly info) - n8n (automatisation for e.g. quick info via telegram, if something not work) Run every app that you want in your container.

kilroy123 5y ago
This is _very_ close to my exact setup. Though, I never used portainer (just manage a big docker-compose file) is it worth it?
- razemio 5y ago
  If you are comfortable with the cli you won't need it at all. It over complicates things in my opinion.
- justin_oaks 5y ago
  Portainer does make certain tasks much easier. I use Portainer to clean up any Docker volumes, containers, and images I no longer need. Doing that for a lot of Docker resources using the command line gets old really fast. I say this as someone who uses docker and docker-compose extensively and I'm very comfortable with a CLI.
  - smallnamespace 5y ago
    Can't you: docker system prune with perhaps a filtering flag attached?
- Jnr 5y ago
  I also put everything in docker containers, and docker-compose for each thing is a must. It usually pains me to see when popular projects have no decent docker-compose files available and I have to make them. For backups restic is a blessing. And so is Syncthing for getting things to the backup machine quickly (from mobile and non-unix machines).

fersarr 5y ago

Great post! Thanks for sharing all the details.

jareware 5y ago

Echoing the sentiment here, this is a great way to host smaller projects on the cheap, without adding the complexity/price of k8s, Nomad et al! I do the same, and have spent some time automating the backup of such a set of standalone containers [0], in case others also find it useful. [0] https://github.com/jareware/docker-volume-backup https://github.com/jareware/docker-volume-backup

timwis 5y ago

Nice! I do something similar for a few production applications, but I use docker compose to bring the services up, so I don’t have to create the networks manually or remember all the command arguments. Works great.

nodlija 5y ago

Instead of recording the “docker run“ commands, you might want to have a look at docker-compose

mattbee 5y ago
Yes I thought that was odd too! OP mentions keeping manual records of "docker run" commands and that it might need some improvement... Well apart from that detail that's exactly what I do for my stack of personal sites. "docker-compose" is really hard to beat for all kinds for setups. And I get 64GB to play with for 50EUR/month from Hetzner, so I don't spend too much time worrying about RAM.
jlkuester7 5y ago
Came here to say this! Honestly, my into to docker-compose came when I realised I was starting to do the same thing (saving 'run' commands). Small learning curve for the yaml structure, but well worth it (and honestly I think it is easier to visualize the deployment when it is all nicely laid out in the compose file). Pretty much the only way I use docker now. (Also has a bonus of not really adding a bunch of overhead or complex. Just does what you need for a basic multi-container deploymnet and no more.)
ytjohn 5y ago
I had started out the same way, especially if it was a new app and I wasn't familiar with how I really wanted to run it. Some containers expect a fair number of environment variables and multiple mounts. Once I got everything working, I would create a script /svcs with the corresponding docker run command. There's even a cool tool called "runlike" which can create a well formatted command for any running container. https://github.com/lavie/runlike/ https://github.com/lavie/runlike/ But I've got those migrated to docker-compose files these days and I try to start with the docker-compose file instead of going directly into testing out docker run commands.

tweetle_beetle 5y ago

Really good write up and always nice to see the honesty about past mistakes. For those without time to get hands quite as dirty, like me, I've found that CapRover[1] gives just enough UI to fiddle while having sensible defaults. [1] https://caprover.com/ https://caprover.com/

Aeolun 5y ago
The only negative about caprover I’ve found is the distinction between containers with persistent storage and not. I don’t want to recreate my whole app just to add persistent storage. Otherwise I’ve been using it without issue for months now.
- kasra85 5y ago
  Kasra from CapRover. Feel free to open a Feature Request on Github. There is no architectural design limitation that prevents this. It can certainly be added.

lucb1e 5y ago

I wonder what the environmental impact of a 24/7 running system with such high-end specs is. Desktops are worse with a graphics card (just yesterday I noticed my system's power consumption doubles when the GPU turns on: 20W with 100% CPU on all cores and WiFi stressed; 37W with additionally primusrun glxgears running), but desktops only run on demand. Dedicated non-mobile hardware doesn't scale to demand that well and has fairly high idle draw. Don't get me wrong, my server also runs a ton of crap, like git/email/backups/irc/mysql/etc. plus a couple websites and some python scripts (3x doing continuous data logging, 2x providing some service), but it draws less than a traditional light bulb and also runs everything in linux containers. Unless you're doing large data processing tasks like machine learning on a weekly/daily basis, odds are you don't need more than two consumer-grade cores (I've got more since I repurposed an old laptop, but they're very rarely used). Large RAM is nice, though, I do have to say that. My 8GB is stretching it a little, considering that my favorite email server software is a Windows GUI thing and so I run virtualbox to have this guilty pleasure, which consumes about a quarter of the system's RAM just for one service. Aside from Owncast, no idea what requirements that has, I think all of OP's services would be fine with a Raspberry Pi as well, just RAM would be tight with so many individual containers. Especially given the static site hosting, and doubly especially with that offloaded to a CDN, the only dynamic things (screenshot hoster, pastebin) are own use and won't see a dozen concurrent users continuously or anything. Edit: I read over the -track API websites part. I don't know whether those might fit on a Pi as well, but that sounds like various people make use of it and this server is not just single user.

amaccuish 5y ago
Out of interest, what is the email server? Your server's EHLO wasn't very revealing.
- lucb1e 5y ago
  It's an old thing that I should probably not mention for opsec reasons since, honestly, it probably can get pwned (I would be surprised if no intelligence agency has a 0-day for it), and I keep thinking I should switch to linux-based solutions, but it just never floats to the top of my todo list. It's also a fairly big operation since I'd have to coordinate the switchover with some people and I have no idea how to go about IMAP message copying other than telling them to move it over in Thunderbird which seems error-prone. If anyone has a good suggestion for an open source system where I can: - Give people access to control their domains via a web interface (create accounts, set a catch-all, such things) - Have aliases and server-side rules (e.g. blackhole email coming into spamaddr2021december@lucb1e.com; move From "bank" Subject "balance change*" Into inbox/notifications), preferably via the web interface - Optionally: have IMAP and SMTP without a lot of moving parts (I've looked at typical Linux setups before and they're usually quite finicky) Then please let me know!

PascalW 5y ago

I'm doing something similar though I've opted specifically to _do_ use Kubernetes via k0s [0]. It works wonderfully well and allows me to use most things that are available in the k8s ecosystem like cert-manager [1] and external-dns [2]. All configuration is stored in Git and the server itself is basically disposable as it only runs k0s and some very basic stuff like iptables rules. I see this sentiment quite a lot that k8s is too complex for small scale setups, but in my experience it scales down very well to small setups. Granted, you do have to know how to work with k8s, but once you learn that you can apply that knowledge to many different kinds of setups like k0s, bare metal, EKS/AKS etc. [0] https://k0sproject.io/ https://k0sproject.io/ [1] https://cert-manager.io/ https://cert-manager.io/ [2] https://github.com/kubernetes-sigs/external-dns https://github.com/kubernetes-sigs/external-dns

znpy 5y ago
I'm playing with k0s and it seems it doesn't play nice with firewalld. With firewalld active containers cannot do networking, not even with hosts jn the same lan. Everything else works beautifully though.
- PascalW 5y ago
  I'm using iptables myself and it works fine, though you have to make sure that traffic on the kube-bridge interface is allowed. With iptables: -A INPUT -i kube-bridge -j ACCEPT -A OUTPUT -o kube-bridge -j ACCEPT Other than that I've configured iptables to drop all incoming traffic except a few whitelisted ports.
Already__Taken 5y ago
Just done a very quick search but it seems like k3s is the better choice so what did you like about k0s?
- PascalW 5y ago
  Curious what makes you think k3s is the better choice? The only reason I ended up going with k0s was that I had problems getting k3s working well behind a locked down firewall. With k0s that was pretty easy.

jarv 5y ago

Curious what service management are you using for all of these docker run commands, systemd? I see you mention that you might consider Prometheus at some point, is there any process monitoring if one of your services crash?

bottled_poe 5y ago

It’s a good setup, but the question is why run this many services on one machine? The risk profile is too great in my opinion. The software/uptime maintenance effort to keep all users happy would far outweigh the cost of using essentially the same setup with a Kubernetes layer over a few servers. This setup only makes sense for experiments, portfolio, etc.

southerntofu 5y ago
What's the "risk profile" and why is it too great? Kubernetes requires dedicated knowledge and can fail in surprising ways. Using a simple server with some established technology can be surprisingly/boringly reliable. Some hosts can reach incredible scale with a just few beefy dedicated machines, given the right software architecture.
- throwaway984393 5y ago
  The fact that it's a dedicated physical host means that any of a dozen components could fail and bring the box down permanently. The hosting company would have to reassign or build a new server for the customer, which could take from an hour to a week (or more in a pandemic with scarce parts). It depends on their excess capacity and on the machine's specifications. If it was one virtual machine I'd say, sure, just use one box, you can bring up a new one automated in 10 minutes. But if it's physical, it could either last 10 years without going down, or 10 weeks, you never know. Physical machines is precisely the time to use a distributed system as a hedge against expensive failures. Another consideration is cost; at $1200/year you could just keep buying servers and Colo them. It's more work but you get more (old) hardware to play with.
  - southerntofu 5y ago
    > any of a dozen components could fail and bring the box down permanently The same is true with VPS and "cloud" setups. As the OVH fire last year showed, offsite backups are not a luxury. The chances of that, though, are ridiculously low. Higher-end dedicated hosts have very low failure rates due to using higher-grade hardware and rotating the hardware every few years; they also usually have very competent and available technicians on site which are paid by your monthly bill. > If it was one virtual machine I'd say, sure, just use one box, you can bring up a new one automated in 10 minutes. The same is true with dedicated servers. If you house them yourself then sure bringing up hardware replacement is going to take longer. But if you use a managed dedicated hosting solution (as was mentioned in the article) the provisioning times are sensibly similar to that of a VPS (a few minutes).

entropyie 5y ago

I also have a tonne of stuff running on a single, fanless server via docker compose. It is amazing, barely breaks 20% CPU, better uptime than aws ;-) ... One overlooked aspect is that if you use something like my Streacom db4 case, beautiful, fanless, noiseless, you can put it in a living area. Depending on where you live, the power usage can be fully offset against heating costs for your home, making it very efficient.

greggyb 5y ago
Just a note on power consumption with regard to heat output. Electric resistive heating is 100% efficient with regard to electricity from the socket. This is the upper limit on efficiency for a computer. This ignores efficiency losses from transmission and from electricity generation. If your electricity is generated by fossil fuels, you are getting much less than 100% efficiency from the energy stored in those fuels. Heat pumps can achieve greater than 100% efficiency relative to electric resistive heating, because they move heat, rather than do work to create heat. These can achieve equivalent heating at 50% of the electricity utilization compared to resistive heating.[0] Local fossil fuel heating (e.g. natural gas furnace) is much more efficient than electric resistive heating. The efficiency of local fossil fuel combustion is close to 100% of the energy content of the fuel, as it is turned directly into heat and that heat is dissipated into the home. More than 60% of the energy content used to generate electricity is lost in the generation process in the US.[1] So, the heat from a computer does offset heating costs, but at less than 100% efficiency. For a small fanless machine, this is not likely to be a large expenditure, but it is valuable to keep in mind that heat from electronics is not "free". [0]: https://www.energy.gov/energysaver/heat-pump-systems https://www.energy.gov/energysaver/heat-pump-systems [1]: https://www.eia.gov/todayinenergy/detail.php?id=44436 https://www.eia.gov/todayinenergy/detail.php?id=44436

southerntofu 5y ago

Thanks for this article, it's great to see people caring for their server (does it have a name?) and not defaulting to the serverless craze. Here's a few thoughts :) > there is some small downtime when I deploy new versions of things since I don't have any load balancing or rolling deployments It's entirely possible to achieve, depending on your stack. `nginx -s reload` will reload the entire config without killing existing connections or inducing undue downtime. So if you can start a second instance of your "webapp" on a separate port/socket (or folder for PHP) and point nginx to it there shouldn't be any downtime involved. > for users that are geographically far away latency can be high That's true, but counter-intuitively, i found unless you're serving huge content (think video or multiple MB pages) it's not a problem. CDN can actually make it worse on a bad connection, because it takes additional roundtrips to resolve the CDN's domain and fetch stuff from there while i already have a connection established to your site. As someone who regularly uses really poor xDSL (from the other side of the atlantic ocean) i have a better experience with sites without a CDN that fit in under 1MB (or even better < 200KB) with as little requests as possible (for the clients that don't support HTTP2). > CloudFlare (...) That may become necessary if I ever have trouble with DDOS attacks I've personally found OVH to be more than capable and willing to deal with DDOS for their customers. OVH has been previously posted on HN for dealing with huge DDOS. That is of course if you have proper caching and you don't have an easy venue for a remote attacker to induce huge CPU/RAM load. For example, Plausible-like analytics can be such an attack vector because every request is logged in a database; something like GoAccess [0] is more resilient, and no logs is even lighter on resources. [0] https://goaccess.io/ https://goaccess.io/

remram 5y ago
> there is some small downtime when I deploy new versions of things Some time ago I was looking for an easier way to fix this. It seemed to me that a good way would be to have the reverse proxy (e.g. nginx or similar) hold the requests into the app restarts. Fit the user, this would mean a ~10s hiccup rather than 504 errors. I didn't find an easy way to do it with nginx though and was sort of disappointed. Maybe other reverse proxies make this easier? Or maybe there is a stand-alone tool to do this? [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so]

rcarmo 5y ago

Nice. I've gone down a different path and built https://github.com/piku/piku https://github.com/piku/piku, which I use to run 12-15 web services (and batch workers) off a single-core VM in Azure, plus around the same amount of private services on a 4GB RAM Raspberry Pi. I still use docker-compose and have a k3s cluster running, but I really like the simplicity of deploying via git. Cloud-init for my core setups is here: https://github.com/piku/cloud-init https://github.com/piku/cloud-init

exhaze 5y ago

Three independent, but somewhat related thoughts on this topic: 1). On HOWTO articles about infra (1/2): I'd like to see more articles that lead with requirements, rather than setups that then justify the setup with requirements. Like, congrats, you managed to host a bunch of web applications via containers on a dedicated server. It's really nice for a super personal project and I'm sure it helped OP gain a lot of operational experience across several domains, but I just find this type of article to be "DYI porn" for a highly specific subset of DYI. 2). On HOWTO articles about infra (2/2): Is there any sort of comprehensive infra setup guide out there? Even something that just covers "core" AWS services (CloudFront, EC2, ELB, ECS, Lambda, CloudWatch, CloudTrail, and a few others...) feels like it would be incredibly useful for so many people. 3). "AWS on Rails" - I feel like we're getting close to a time when "AWS on Rails" emerges and we see the birth of a new opinionated tool that says "Most people just want to ship their product, not fiddle with cloud infra. Do it our way and you can focus on product instead of infra"

Aeolun 5y ago
> Is there any sort of comprehensive infra setup guide out there? This would be the AWS certification at their different levels.
- pyrale 5y ago
  I would tend to disagree. First, because not everyone is on AWS. Second, because even for people on AWS, some of a company's infra isn't on AWS (e.g. gmail). Third because not every infra setup work has an AWS tool. To take a personal example, we use multiple infra providers. Some of our infra is gmail (currently working on automating it with terraform). Some of it is other infrastructure providers that aren't US-based. etc.
  - sokoloff 5y ago
    What’s the “automating gmail with terraform” angle? G-suite automation? Or GCP?
    - 5y ago

kalev 5y ago

Be careful with docker and ufw though! Any rules you setup in ufw will be ignored by docker, so exposing a container will always open it up to the public even if you specifically denied access through ufw.

adamddev1 5y ago
Very good point, I didn't know this almost got burned by this while learning Docker. What I did was use shared network (for private db connection etc) in a docker-compose file, and then expose the port I wanted to reverse proxy out on by ports: - 127.0.0.1:3000:3000 This way it only exposes it to the local host on the machine without exposing it on the firewall. Then I reverse proxied out port 3000 with NGINX to the outside world. I'm surprised this isn't talked about more in beginner tutorials etc.
k8sToGo 5y ago
Technically, Docker is adding iptables rules that are ignored by ufw rather than docker ignoring ufw. To fix, just turn off iptables for the docker daemon and add the rules manually to UFW
- dawnerd 5y ago
  And if you don’t want to do that because there are some downsides, make sure you setup container networking correctly and don’t just expose ports just to expose ports. Learned that one the hard way when someone exposed redis.

maxicarlos08 5y ago

Hehe, have been doing that on my Raspberry Pi 4 for years now - Nextcloud - Gitlab - A bunch of websites hosted under different domains - Matrix homeserver And the load is still surprisingly low

thatwasunusual 5y ago
> And the load is still surprisingly low Which means that you have some extremely unpopular services running...? :) /s (Happy new year!)

kxjkckicllf 5y ago

kxjkckicllfhhh 5y ago

z3t4 5y ago

No need for CDN, if the connection is already setup and the server supports HTTP2 the files can be sent in parallel. TCP and SSL handshake back-n-forth will likely eat up the initial Atlantic latency. Also static content can be cached on the user device. And serving static content is well optimized in most web servers, so you do not need beefy hardware.

kristianpaul 5y ago

Security is not only firewall rules. A LIDS like OSSEC its a good option to install a automatically attacks as well.

hestefisk 5y ago
I use Snort. It works like a charm.
- kristianpaul 5y ago
  Do you decrypt tls ?

sillycube 5y ago

Did you try to upgrade the server os? If you put everything in 1 server, isn't it painful to migrate all services at once?

janvdberg 5y ago

Great write up! Tip I learned this week when I migrated my VPS [1]: when dumping MySQL/MariaDB databases for disaster recovery, dump the grants (aka database user rights) with the pt-show-grants tool. You don't want to import the mysql table itself on a fresh MySQL/MariaDB installation, it's a headache. So dump all your tables to SQL (raw data) and dump your user grants/rights with pt-show-grants (which in itself creates a small SQL file) that you easily import in MySQL/MariaDB. [1] https://j11g.com/2021/12/28/migrating-a-lamp-vps/ https://j11g.com/2021/12/28/migrating-a-lamp-vps/

daneel_w 5y ago
If you don't have access to (or don't want to use) the Percona toolkit, you can get all of the grants directly from the `sql_grants` table by installing `common_schema`.

thecrumb 5y ago

I can remember back in the dotcom boom I had Windows small biz server, Exchange, our phone system and hosting about 3 dozen websites on a single server in our breakroom. We ran like that for about 2 years with really no issues other than the occasional bandwidth issue...

siilats 5y ago
The new windows 2022 vnext with azure arc is pretty good, RDP is so much better than VNC. Also wamp is so much faster on windows than on linux in my experience on tuning php/python/mysql/postgres stuff. Windows and its shared memory connection to DB on the same machine bypasses pipes, which is where most of the slowdown comes i think

justsomeuser 5y ago

I do something similar: - GCP VM with “Google Container OS”. - “Cloud init” config set on the VM metadata (can easily recreate the VM; no snowflake config via SSH mutations). - My service runs in a docker container, reads/writes to a SQLite file on the host disk. - GCP incrementally snapshots the disk every hour or so, and makes copies to different region. Any disk writes are copied instantly to another zone. - Lets encrypt cert is read from the host disk and the docker container serves HTTPS directly (no proxy). Certificate is renewed with the LE CLI. - The service logs to standard out, this is collected by the Google logs daemon which I can view with the web UI. - Google have their own HTTP uptime monitoring and alerting which sends you an SMS.

hobo_mark 5y ago
TIL about their uptime monitor, thanks! https://cloud.google.com/monitoring/uptime-checks https://cloud.google.com/monitoring/uptime-checks

dawnerd 5y ago

I’m using the same server type from OVH in Hillsboro and it’s great. They frequently do sales. Highly recommend anyone interested to go for the nvme upgrade. I rent three now total, one for production sites, one as a development machine strictly for vscode and another for development / staging / misc. waaaaaay overkill but it’s been a huge quality of life improvement. For containers I just use a couple docker compose and a script to run them all. Reverse proxy is through nginx proxy manager which takes care of LE renewals and has a nice web interface.

Ocha 5y ago
I’m running collocated servers with simple docker compose file too. Actually https://mailsnag.com https://mailsnag.com is hosted on one of the servers and is using single docker compose file for deployment. Not sure why the author had to record every single docker run command when docker compose file manages all of that for you.
Axsuul 5y ago
Where do you go to monitor these Hillsboro OVH sales?
- dawnerd 5y ago
  They send out emails but also just around major holidays and just randomly. They use this page most of the time from what I can tell https://us.ovhcloud.com/deals https://us.ovhcloud.com/deals Sometimes though they'll send targeted email offers that are not on the site and are good for older hardware but at a really cheap price.

piyiotisk 5y ago

I have a similar setup and wrote how to setup everything here https://costapiy.com/deploy_django_project_linux_server/ https://costapiy.com/deploy_django_project_linux_server/ and here https://github.com/costapiy/server_setup https://github.com/costapiy/server_setup

nyolfen 5y ago

great writeup that overlaps with a lot of stuff i do for myself — a few related tool suggestions from my own experience are caddy for https (absurdly easy config — seriously, i would never go back to nginx if i could help it) and netdata for server monitoring (open source, optionally connectable to a web dashboard with email alerts) [0] https://caddyserver.com/ https://caddyserver.com/ [1] https://github.com/netdata/netdata https://github.com/netdata/netdata

_xnmw 5y ago

This is super useful, thanks for posting. I hate AWS complexity-as-a-service, just give me a damn baremetal machine and let me run and tune my own services. I love having full visibility into raw disk/cpu performance and even benching my hardware. So many useful comparisons here, Bunny.net vs. CloudFlare, and the fact that you got this far without even using fail2ban! Questions (not necessarily for OP, but for anyone) - Give us an idea of the maximum load this server is doing in terms of requests per second? - Anyone choosing AMD Epyc over Intel to avoid spectre/meltdown mitigation slowdowns? - Any use of postfix or other SMTP forwarding via Postmark or another email provider? - What is your Postgres configuration like for your heaviest loads? max_connections? Just wondering how Postgres scales across cores.

Ameo 5y ago
Hey, author here! - Request rate is pretty low on average and peaks at around 15 requests/second. That's nothing really and it would probably take 100x or more request rate before I saw any kind of CPU bottlenecking or similar; my upload would probably bottleneck first. The biggest resource usage on the server comes from some expensive database queries made by some of my applications. - I'd definitely be down to use that kind of CPU but it wasn't available at the price point I wanted. Most of my stuff isn't CPU-bound anyway. - I used to self-host my own mailserver and webmail, but to be honest it was a disaster. Even after spending tons of time configuring reverse DNS, DMARC, SKIN, SPF, and the other list of arcane and specialized stuff you need, I still had tons of issues with mail I sent going to spam. I gave up and pay Google to host my mail for my own domains now. - I really haven't done much tuning. MySQL/MariaDB is the DB that gets the most load and its settings are most default. I have put a lot of effort into profiling my apps' query patterns and making sure proper indexes and schemas are in place, though.

2bluesc 5y ago

Traefik Proxy[0] was a game changer for my self-hosted setup of Docker containers. Traefik can read labels applied to Docker containers (easily done with docker-compose) and setup the proxy for you as containers come and go. Even maintains the Lets Encrypt certificates seamlessly. [0] https://traefik.io/traefik/ https://traefik.io/traefik/

dkryptr 5y ago
Traefik is great, but their documentation is awful IMO. I moved to Caddy which I prefer currently.
- rstat1 5y ago
  To be fair, Caddy's docs aren't great either. Last time I tried to deploy it to give SSL to something that didn't have it, took me 2 hrs to figure out the docs enough to get it working.
  - mholt 5y ago
    What could make the docs better? What solved your problem?
    - rstat1 5y ago
      Time and trial-and-error is what eventually solved it. I was trying to configure the auto-HTTPS functionality to use DNS challenges, because my setup forbids any of the others, and this apparently requires plugins that are only supported by community efforts, so that's probably why I had such an issue. The docs around configuring this were not immediately clear to me. Its like one part is over in this corner, the other over there. And the part about DNS provider plugins is off in a completely different direction. I definitely think things could be better organized. Also your one tutorial for getting HTTPS support only mentions the very basic cases. I think it'd be beneficial to see some more advanced tutorials using for instance some of the other LE challenge methods, something like this definitely would've saved me some time.
  - 5y ago

api 5y ago

Yet another modern "cloud native" era developer discovers how overpriced and underpowered most cloud offerings are. Here's what they did folks: they set cloud prices a while back, did not drop them as Moore's Law delivered more power but instead pocketed the profit, and meanwhile ran an entire "cloud native" development push to encourage development practices that maximize cloud lock-in (both through dependence on cloud services and encouraging complexity). Oh, and bandwidth is ludicrously overpriced in the cloud. You can transfer hundreds of terabytes a month outbound for <$500 if you know where to look. No I don't mean from static data either. Look at bare metal hosting.

gogs 5y ago

We kind of moved away from this monolithic setup tho. Oh you can scale-up your hardware like we used to, it will work, until some point. And the day you will need to split your services will come quickly. This is a perfect solution for early stage projects if you're in a startup or for a personel project, but monoliths are....well....monoliths.

jazzyjackson 5y ago
why constrain it to early stage? must everyone set their sights on 1 million DAU? monoliths are KISS, and kubernetes can be YAGNI
Nextgrid 5y ago
The only thing that is hard/impossible to scale (without tradeoffs) is stateful services like databases, and those won't go away even if you move to microservices. Appservers can be trivially scaled horizontally regardless of whether it's one process that does all (monolith) or a mess of microservices.

Terretta 5y ago

> Whenever I add a new service, I record the exact docker run command I used to launch it and refer to it when I need to re-create the containers for whatever reason. I know that this is an area that could use improvement; it's quite manual right now. I'd be interested to hear about some kind of light-weight solution for this that people have come up with. Later... > I host my own analytics ... They provide an open-source, self-hostable version which I have it deployed with docker-compose. Wouldn't docker-compose be the "already doing it" answer to the first question? It's pretty much your library of 'exact docker run commands', plus sugar to manage the running ones collectively.

dhc02 5y ago
> Wouldn't docker-compose be the "already doing it" answer to the first question? It's pretty much your library of 'exact docker run commands', plus sugar to manage the running ones collectively. I'm also confused why he's not using compose. Perhaps he is unaware that when you have multiple services in a compose file, re-running `docker compose up` only restarts containers whose parameters have changed in the compose file. He mentions further down that he's using compose for one service, so obviously he is aware of it. But I can't blame him. When getting up to speed on docker, I found that figuring out compose and how to use it effectively was frustratingly under-documented.

praveen9920 5y ago

I tried something like with a ec2 instance. It may be easy to maintain it as all your hosted services are in one place. Main problem I faced is that the server has become single point of failure which caused lot of issues with customers. It’s really hard to explain why all your services are down at once.

galaxyLogic 5y ago

This all seems quite complicated. Wouldn't it be easier to use Amazon Lambda or similar service. Spend your time creating and optimizing the applications instead of the infrastructure. I've known people who built their own boat and also built the engine for the boat. And it's a great hobby which gives you a great sense of accomplishment. But, if you just need to get some fishing done, buy the boat with the engine already installed.

BackBlast 5y ago
Lambda has it's own set of complexities and concerns. It's also not a good fit for his use cases. Now, a containerization service like Fargate would fit his use case better since he is using containers. However, cloud services come with their own complexities and the principle problem they solve (horizontal scaling) doesn't apply to his sites.
walrus01 5y ago
maybe the goal is to develop the skills and experience to be able to build more boats. or to work on marine diesel engines. or to do something like rehabilitate gigantic 70 year old wooden tugboats. you can't always rely on third party things to host your stuff if your intention is to become the person who runs the hosting services for other peoples' stuff. same way you can't rely on a boatyard's contractors to maintain your boat if your goal is to become an engineer for medium sized yachts.
Yaina 5y ago
I also self-host my own site and some smaller web-services. It's a pain and I don't like it, but the costs are low and predictable and I don't have to write for any proprietary API like Lambda
chickenpotpie 5y ago
Personally, I would never use Lambda for a passion project. The uncontrollable pricing model is not friendly for individual developers. What do you do if you wake up in the morning to a $1000 bill? Leave it up and just keep pouring money down the drain or just end up rewriting the whole site to custom infrastructure so you can control the spending anyways?

anderspitman 5y ago

If you want to do something like this from home behind CGNAT, blocked 80/443 ports, etc, you'll probably need to set up tunneling as well: https://github.com/anderspitman/awesome-tunneling https://github.com/anderspitman/awesome-tunneling

throwaway950q 5y ago

Using FreeBSD jails you can easily host hundreds if not thousands of web applications and services on a single server having the specs mentioned by the OP. This practice isn't even noteworthy in FreeBSD land as it is so common.

amiantos 5y ago

Is there a reason people don’t self-host on a small home server? $85/mo is a lot of money if you aren’t making any money off what you’re hosting. If you mostly run CRUD APIs and sites, is there any downside to a low bandwidth home connection? Multiple family members already stream Plex content from my measly 20mbit upstream. Why not run a whole site or multiples?

laristine 5y ago
I both maintain a server cabinet at home and rent VMs from providers. There are many reasons people don't self-host: noise, heat, dealing with hardware setup, etc. And if any of your service receives public traffic, you'll have to be very careful in configuring it because you don't want attackers to target your home network or the ISP/police to come at you for illegal traffic on your line. $85/month is a lot of money, but given the number of services he runs on his server, each service now costs roughly $3/month, lower than the price of the cheapest DigitalOcean droplet.
rubatuga 5y ago
We made a product called Hoppy Network (https://hoppy.network https://hoppy.network) that fits this use case perfectly. We use WireGuard tunnels to bypass all ISP restrictions, and provide a clean /32 IPv4 and /56 IPv6 using a native network interface. Some of our customers are on Starlink or cellular ISPs. We just launched our annual plans today, cheapest plan is $80/year. Plus, we're sending out a sticker-pack to our first 20 annual plan customers. For the purists out there, we don't filter ICMP packets and don't block any ports. TLDR: You can self-host without your ISP knowing!
graton 5y ago
You could. If I did it then I would most likely go with Cloudflare Tunnel( https://blog.cloudflare.com/tunnel-for-everyone/ https://blog.cloudflare.com/tunnel-for-everyone/ ) which is free to use. Since I would want to try to protect myself from a possible DDoS attack. But using that then I think it would be more than feasible to do it from your home network if you have a fast enough upload link.
wiether 5y ago

awayto 5y ago

Lots of AWS frustration here, so I will mention a project I created to auto generate AWS web applications, covering what I believe are a majority of practical use cases (db, ui, api, users, roles, groups). https://awayto.dev https://awayto.dev https://github.com/keybittech/awayto https://github.com/keybittech/awayto It's not a product, just a project. Check it out if AWS is getting you down! :)

stavros 5y ago

I had the same problem and didn't want to manage things by hand, so I wrote Harbormaster: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster It basically pulls Compose apps from the git repositories you specify, builds the containers and makes sure they're running. Pretty simple and works really well for me.

figassis 5y ago

Just want to point out that docker bypasses ufw, so binding to 0.0.0.0 might not be as safe as you think. Best think to do is not expose any ports at all and have your proxy connect to the container’s internal port.

nirav72 5y ago

I got the same cpu , as part of a hp Z420 workstation I picked up for $250. Added 32 gb ecc memory and able to easily run about 25 containers with plenty of resources left over for VM or two.

jmatthews 5y ago

Thanks for the write up. I use roughly the same set up minus docker. My deploy is usually log in to ssh, git pull and restart the systemctl service.

BrandoElFollito 5y ago

I have been self hosting on and off since about 2000. When I got my fiber link (4 or so years ago) I also started to do home automation and the self hosting got more serious ("national security infrastructure" as my wife calls it, her to-go question is "what will happen with that shit if you die tomorrow?"). I run everything in docker containers, except Wireguard and the backup. I had an idea to "document everything" but the realized there would be no-one to maintain the services (neither my wife not my children are interested in IT). I then decided to document how to untangle themselves from the self-hosting (moving to well-established services, dismantle the home automation, ...). Thi sis going to make more sense in the long term. Since I do not plan to die tomorrow but I do not know for my server, I tried to do a disaster recovery from scratch. It was surprisingly easy - after starting a VM on my laptop (simulating the bare-metal rebuild of my server), I was up and running in about an hour. Docker is simply fantastic for that. I took some notes about the disaster recovery, which I promptly lost so I will have to run this again soon (maybe next week). Actually I will copy the encryption key for my remote backup right away as this is the only thing I won't be able to recover. EDIT: done :) TL;DR: - use docker containers for your services, it is a life-saver and makes everything easier - use a docker-compose script to have everything on file - document either your maintenance or exit strategy - do a quick DRP test on a VM

nokya 5y ago

Thank you for sharing your approach, I learned a few things to improve my own server and highly looking forward to deploying them!

XCSme 5y ago

I have a much simpler set-up, but I don't run everything on one server. I usually prefer having a separate VPS or dedicated server for each service, but this is starting to be a bit more costly than simply renting a powerful machine for everything. One advantage of having everything scattered around is that not everything is going to go down at once (eg: a DDOS or even a traffic spike on your machine will kill all your sites/services if you only have one machine). I use: - Percona PMM[0] - Database/server monitoring - Backups to BackBlaze B2 using cron jobs - Simple deployments, usually using SFTP to upload files (usually for LAMP applications, also running Node.js apps using pm2 that can listen to file changes and restart the app) I am actually working on creating some tools/scripts/guides to aid with self-hosting (for backing up stuff, uptime monitoring, alerting).