5 ms·

Scaling Mastodon with systemd template units

69 points by ghuntley 4y ago

nisa 4y ago
there is also a gist with some notes from Gargron himself: https://gist.github.com/Gargron/aa9341a49dc91d5a721019d9e0c9fd11 https://gist.github.com/Gargron/aa9341a49dc91d5a721019d9e0c9...
techhazard 4y ago
> Requires=network.target tells systemd not to start this mastodon.target if the network.target hasn’t been reached successfully. That’s not true, Wants=/Requires= do not specify any order between units, only that without them the unit cannot be marked as succesful. This way they can still be started in parallel. If you really want it to happen after network is up, you should add a dependency of After= under the Requires=. Also network-online.target might be a better choice but YMMV.
- ilyt 4y ago
  Blog says "reached" and not "finished" starting, the example here also includes After= for network target...
bkq 4y ago
systemd, in my opinion (one that doesn't seem to be widely shared), really does make administering a Linux server easier. No longer do I have to wrangle with shell scripts for the process control of daemons, or worry about the dependencies between them, instead I can organise that logic in a nice and neat .service file for systemd to worry about. The templating feature as this article demonstrates is also incredibly handy for DRYing up some .service files you may write. Another feature of systemd I really like are the timers [1][2] it introduces. It offers a much more sane approach to handling scheduled jobs, and allows for an easy overview of which jobs are running and when the next job will fire via systemctl --list-timers. [1] - https://www.freedesktop.org/software/systemd/man/systemd.timer.html https://www.freedesktop.org/software/systemd/man/systemd.tim... [2] - https://wiki.archlinux.org/title/systemd/Timers https://wiki.archlinux.org/title/systemd/Timers
- arnsholt 4y ago
  I think it's more widely shared than one might believe, because of the people who don't like systemd there's a contingent that really, REALLY hate it.
- ilyt 4y ago
  I despise the amount of random stuff that's tied to it and journalctl data format is still an abomination (and my bug with it trying to search thru entirety of log dir to show single app status is still open) ...but we did remove few thousands lines of fixed init scripts (anyone that tells you sysv scripts are easy and simple is lying to you, multiple projects fail there) *and* an bunch of Monit instances thanks to the features the base service management has. And simplified a bunch of other cases. So yeah, overall even with issues it is a huge benefit.
  - necrotic_comp 4y ago
    The process control in systemd I think is unambigously good, but it would be nice if it wasn't as tightly coupled with journalctl and all of the other random stuff it's gobbling up.
    - ilyt 4y ago
      Well, to be fair you can't exactly ask typical syslog demon "give me log files of an app" but the way journalctl implemented it is just fucking terrible. It takes 4 whole seconds [1] for systemctl/journald to tell me it has no logs for the daemon on my NAS and it opens 985 files while it is doing it If it just used sqlite as a backend it might've been useful for analysis (on top of way faster...) but Lennart wanted to have a go at implementing binary DB format badly so we're stuck with it. If it at least kept a pointer to last file the app's logs were written to the lack of proper indexing also wouldn't be a problem. But nope, it's just entirely worse than text format. And I do mean that in entirety, acking thru text files is faster than systemctl trying to find log files for the app... * [1] https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460
raziel2p 4y ago
It's odd to me that systemd didn't implement a "run X number of processes" feature. I've also found it difficult to find a well-written tiny application written in C/Rust/Go whose only job it is to start and supervise X number of processes. supervisord is the well-known workhorse here but it's really old and quite buggy.
- lantry 4y ago
  Have you seen overmind? https://github.com/DarthSim/overmind https://github.com/DarthSim/overmind
  - raziel2p 4y ago
    Unfortunately Procfile doesn't support declaratively setting the number of processes, Heroku designed it so you have to specify that with their CLI.
- ilyt 4y ago
  From my experience there are very little use cases where you want to start same app exactly multiple times, usually you always need to vary at least one parameter (usually config file or workdir), and that's pretty app-specific. ...that being said make template unit `myapp@` then just systemctl start myapp@1 systemctl start myapp@2 systemctl start myapp@3 can also tie them to start/stop together via dependencies or make wrapper script that starts all and use man systemd.kill modes to send signal to all of them.
  - raziel2p 4y ago
    Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a declarative configuration format. In the past I did actually use the templated solution, e.g. systemctl enable myapp-worker@{1..4} systemctl start/restart/reload/stop myapp-worker@* However, in order to scale the number of processes down, you have to know how many are running already. Not that difficult to write a wrapper script around of course, but why isn't it just built-in?
ilyt 4y ago
Environment="NODE_ENV=production" Environment="PORT=4000" Environment="BIND=::" Environment="STREAMING_CLUSTER_NUM=2" You probably want to use EnvironmentFile= and shove that file with the rest of app's configs And, uh, shove the queue size parameter there instead of sudo systemctl enable mastodon-sidekiq-default@25.service sudo systemctl enable mastodon-sidekiq-mailers@2.service sudo systemctl enable mastodon-sidekiq-scheduler@2.service using feature designed to run multiple instances as a poor man's way to configure stuff. Let's just say nothing will stop you from also doing systemctl enable mastodon-sidekiq-default@15.service and systemd will just try to start the sidekiq twice The article's proposed way of using templates is TERRIBLE. I had assumed from the title it will show off how to run multiple instances using same template units (say to vary a run dir or whatever), but no, it is using them as glorified config file for number of threads/db conns. Just shove that into env file, you can use env parameters in rest of the file too...
- arminiusreturns 4y ago
  You sound very knowledgable on systemd, could ypu share some suggestions for learning resources? I would like to push a better use of it in my org, but besides some unit file messing about and using all the cool anaylysis tools, I don't know as much as I'd like.
  - ilyt 4y ago
    Well, I've written a blogpost on dependencies [1] but really reading all of the man pages (the most useful ones [2][3][4]) will get you a good idea on what it can do, I had many "oh, we can use THAT" moments with it. They are written pretty well. The directives we got most use in org are probably Restart* family and RequiresMountsFor, althought the last one is more and more common in distro packages so it is not even needed that often. The usecase being "don't start DB if mount fails" usually. Overrides are probably the most used feature overall, we often find that the standard distro unit just needs a little bit of tweaking for our purpose (like adding restart), so we just create file /etc/systemd/system/servicename.service.d/01-restart.conf with [Service] Restart=always RestartSec=30 and voila, service now auto-restarts and that behaviour won't break when package updates, as we're not changing any package file. The few other modifications of existing ones included adding new read-write dirs or adding env variables for proxy. Technically you can even go as far as making services into little containers with all the options for limiting permissions but I haven't got that far. Most apps don't exactly need it either, have readonly/readwrite dir + some limits and it is pretty safe out of the box, I've also noticed many distro packages going that way. * [1] https://devrandom.eu/blog/post/2022-02-27_increasing_system_robustness_with_systemd_dependencies https://devrandom.eu/blog/post/2022-02-27_increasing_system_... * [2] https://www.freedesktop.org/software/systemd/man/systemd.exec.html https://www.freedesktop.org/software/systemd/man/systemd.exe... * [3] https://www.freedesktop.org/software/systemd/man/systemd.unit.html https://www.freedesktop.org/software/systemd/man/systemd.uni... * [4] https://www.freedesktop.org/software/systemd/man/systemd.service.html https://www.freedesktop.org/software/systemd/man/systemd.ser...
Karellen 4y ago
We have the following files in /lib/systemd/system/ to define our sidekiq unit templates, which are all just copies of the same content as above. [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] Wat
- ehPReth 4y ago
  Cloudflare weirdness I suppose. I see them all, but perhaps they think you're likely a bot or something and don't allow you to see it? (That or you have JS off?)
  - Karellen 4y ago
    Ah yeah, if I enable JS for the page then I get the right content. sigh

69 points by ghuntley 4y ago

nisa 4y ago

there is also a gist with some notes from Gargron himself: https://gist.github.com/Gargron/aa9341a49dc91d5a721019d9e0c9fd11 https://gist.github.com/Gargron/aa9341a49dc91d5a721019d9e0c9...

techhazard 4y ago

> Requires=network.target tells systemd not to start this mastodon.target if the network.target hasn’t been reached successfully. That’s not true, Wants=/Requires= do not specify any order between units, only that without them the unit cannot be marked as succesful. This way they can still be started in parallel. If you really want it to happen after network is up, you should add a dependency of After= under the Requires=. Also network-online.target might be a better choice but YMMV.

ilyt 4y ago
Blog says "reached" and not "finished" starting, the example here also includes After= for network target...

bkq 4y ago

systemd, in my opinion (one that doesn't seem to be widely shared), really does make administering a Linux server easier. No longer do I have to wrangle with shell scripts for the process control of daemons, or worry about the dependencies between them, instead I can organise that logic in a nice and neat .service file for systemd to worry about. The templating feature as this article demonstrates is also incredibly handy for DRYing up some .service files you may write. Another feature of systemd I really like are the timers [1][2] it introduces. It offers a much more sane approach to handling scheduled jobs, and allows for an easy overview of which jobs are running and when the next job will fire via systemctl --list-timers. [1] - https://www.freedesktop.org/software/systemd/man/systemd.timer.html https://www.freedesktop.org/software/systemd/man/systemd.tim... [2] - https://wiki.archlinux.org/title/systemd/Timers https://wiki.archlinux.org/title/systemd/Timers

arnsholt 4y ago
I think it's more widely shared than one might believe, because of the people who don't like systemd there's a contingent that really, REALLY hate it.
ilyt 4y ago
I despise the amount of random stuff that's tied to it and journalctl data format is still an abomination (and my bug with it trying to search thru entirety of log dir to show single app status is still open) ...but we did remove few thousands lines of fixed init scripts (anyone that tells you sysv scripts are easy and simple is lying to you, multiple projects fail there) *and* an bunch of Monit instances thanks to the features the base service management has. And simplified a bunch of other cases. So yeah, overall even with issues it is a huge benefit.
- necrotic_comp 4y ago
  The process control in systemd I think is unambigously good, but it would be nice if it wasn't as tightly coupled with journalctl and all of the other random stuff it's gobbling up.
  - ilyt 4y ago
    Well, to be fair you can't exactly ask typical syslog demon "give me log files of an app" but the way journalctl implemented it is just fucking terrible. It takes 4 whole seconds [1] for systemctl/journald to tell me it has no logs for the daemon on my NAS and it opens 985 files while it is doing it If it just used sqlite as a backend it might've been useful for analysis (on top of way faster...) but Lennart wanted to have a go at implementing binary DB format badly so we're stuck with it. If it at least kept a pointer to last file the app's logs were written to the lack of proper indexing also wouldn't be a problem. But nope, it's just entirely worse than text format. And I do mean that in entirety, acking thru text files is faster than systemctl trying to find log files for the app... * [1] https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460

raziel2p 4y ago

It's odd to me that systemd didn't implement a "run X number of processes" feature. I've also found it difficult to find a well-written tiny application written in C/Rust/Go whose only job it is to start and supervise X number of processes. supervisord is the well-known workhorse here but it's really old and quite buggy.

lantry 4y ago
Have you seen overmind? https://github.com/DarthSim/overmind https://github.com/DarthSim/overmind
- raziel2p 4y ago
  Unfortunately Procfile doesn't support declaratively setting the number of processes, Heroku designed it so you have to specify that with their CLI.
ilyt 4y ago
From my experience there are very little use cases where you want to start same app exactly multiple times, usually you always need to vary at least one parameter (usually config file or workdir), and that's pretty app-specific. ...that being said make template unit `myapp@` then just systemctl start myapp@1 systemctl start myapp@2 systemctl start myapp@3 can also tie them to start/stop together via dependencies or make wrapper script that starts all and use man systemd.kill modes to send signal to all of them.
- raziel2p 4y ago
  Queue workers is a classic example. It's not necessarily about wanting an exact multiple of processes, but more about wanting to scale them up and down with a declarative configuration format. In the past I did actually use the templated solution, e.g. systemctl enable myapp-worker@{1..4} systemctl start/restart/reload/stop myapp-worker@* However, in order to scale the number of processes down, you have to know how many are running already. Not that difficult to write a wrapper script around of course, but why isn't it just built-in?

ilyt 4y ago

Environment="NODE_ENV=production" Environment="PORT=4000" Environment="BIND=::" Environment="STREAMING_CLUSTER_NUM=2" You probably want to use EnvironmentFile= and shove that file with the rest of app's configs And, uh, shove the queue size parameter there instead of sudo systemctl enable mastodon-sidekiq-default@25.service sudo systemctl enable mastodon-sidekiq-mailers@2.service sudo systemctl enable mastodon-sidekiq-scheduler@2.service using feature designed to run multiple instances as a poor man's way to configure stuff. Let's just say nothing will stop you from also doing systemctl enable mastodon-sidekiq-default@15.service and systemd will just try to start the sidekiq twice The article's proposed way of using templates is TERRIBLE. I had assumed from the title it will show off how to run multiple instances using same template units (say to vary a run dir or whatever), but no, it is using them as glorified config file for number of threads/db conns. Just shove that into env file, you can use env parameters in rest of the file too...

arminiusreturns 4y ago
You sound very knowledgable on systemd, could ypu share some suggestions for learning resources? I would like to push a better use of it in my org, but besides some unit file messing about and using all the cool anaylysis tools, I don't know as much as I'd like.
- ilyt 4y ago
  Well, I've written a blogpost on dependencies [1] but really reading all of the man pages (the most useful ones [2][3][4]) will get you a good idea on what it can do, I had many "oh, we can use THAT" moments with it. They are written pretty well. The directives we got most use in org are probably Restart* family and RequiresMountsFor, althought the last one is more and more common in distro packages so it is not even needed that often. The usecase being "don't start DB if mount fails" usually. Overrides are probably the most used feature overall, we often find that the standard distro unit just needs a little bit of tweaking for our purpose (like adding restart), so we just create file /etc/systemd/system/servicename.service.d/01-restart.conf with [Service] Restart=always RestartSec=30 and voila, service now auto-restarts and that behaviour won't break when package updates, as we're not changing any package file. The few other modifications of existing ones included adding new read-write dirs or adding env variables for proxy. Technically you can even go as far as making services into little containers with all the options for limiting permissions but I haven't got that far. Most apps don't exactly need it either, have readonly/readwrite dir + some limits and it is pretty safe out of the box, I've also noticed many distro packages going that way. * [1] https://devrandom.eu/blog/post/2022-02-27_increasing_system_robustness_with_systemd_dependencies https://devrandom.eu/blog/post/2022-02-27_increasing_system_... * [2] https://www.freedesktop.org/software/systemd/man/systemd.exec.html https://www.freedesktop.org/software/systemd/man/systemd.exe... * [3] https://www.freedesktop.org/software/systemd/man/systemd.unit.html https://www.freedesktop.org/software/systemd/man/systemd.uni... * [4] https://www.freedesktop.org/software/systemd/man/systemd.service.html https://www.freedesktop.org/software/systemd/man/systemd.ser...

Karellen 4y ago

We have the following files in /lib/systemd/system/ to define our sidekiq unit templates, which are all just copies of the same content as above. [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] Wat

ehPReth 4y ago
Cloudflare weirdness I suppose. I see them all, but perhaps they think you're likely a bot or something and don't allow you to see it? (That or you have JS off?)
- Karellen 4y ago
  Ah yeah, if I enable JS for the page then I get the right content. sigh