31 ms·

And yet another gotcha with configuration file loading: Over-elaborated hierarchies where configuration files may reside, and their precedence schemes. E.g. f

by Eduard 3y ago

And yet another gotcha with configuration file loading: Over-elaborated hierarchies where configuration files may reside, and their precedence schemes. E.g. from systemd.unit manpage ( https://www.freedesktop.org/software/systemd/man/systemd.unit.html https://www.freedesktop.org/software/systemd/man/systemd.uni... ): >... > In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc/ take precedence over those in /run/ which in turn take precedence over those in /usr/lib/. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in. > ... And this is just an excerpt which doesn't cover the details. Read the manpage for all the hilarity.

nick__m 3y ago
systemd configuration lookups are lightweights and straightforward compared to ansible variables lookups rules : -- Understanding variable precedence Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables): 1 command line values (for example, -u my_user, these are not variables) 2 role defaults (defined in role/defaults/main.yml) 1 3 inventory file or script group vars 2 4 inventory group_vars/all 3 5 playbook group_vars/all 3 6 inventory group_vars/* 3 7 playbook group_vars/* 3 8 inventory file or script host vars 2 9 inventory host_vars/* 3 10 playbook host_vars/* 3 11 host facts / cached set_facts 4 12 play vars 13 play vars_prompt 14 play vars_files 15 role vars (defined in role/vars/main.yml) 16 block vars (only for tasks in block) 17 task vars (only for the task) 18 include_vars 19 set_facts / registered vars 20 role (and include_role) params 21 include params 22 extra vars (for example, -e "user=my_user")(always win precedence) ‐‐ that is a RedHat-isim
- oblio 3y ago
  Meh, it's not Red Hat. Configuration management tools do this. Chef is similar.
  - eropple 3y ago
    Chef is worse. It's discouraged to use `override` in Chef cookbooks but nothing says that you can't, and introducing spooky action at a distance is just part of your life now.
- pferde 3y ago
  IIRC, Ansible had this exact variable precedence long before they were acquired by RH. I get that hating RH is in vogue currently, but let's keep it grounded in reality please. EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.
  - TheNewsIsHere 3y ago
    As an Ansible person, I couldn’t agree more. Being able to manage variables at basically any level of precedence is extremely useful. I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.
- hedora 3y ago
  I love how there are 21 places to look if you're trying to figure out why a CLI parameter is being ignored. Does it at least print a warning or something?
  - faangsticle 3y ago
    Other way around. Cli params have highest precedence.
crote 3y ago
That doesn't look too bad to me. ".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server. /usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.
- andrewaylett 3y ago
  /run is useful for configuration generation. Sure, you could maybe poke an API and have it do the right thing, but if "everything is a file" then why not put your configuration in a file? That way it's easier to see what's actually happening. Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.
Arnavion 3y ago
>Over-elaborated hierarchies where configuration files may reside, and their precedence schemes. >Read the manpage for all the hilarity. If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr. Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr. Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file. User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc. Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy. The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings. BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them. >And this is just an excerpt which doesn't cover the details. The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.
- stinos 3y ago
  and the user doesn't touch /usr. So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.
  - bayindirh 3y ago
    Yes, "The user (syadmin) doesn't touch /usr, except /usr/local", where compiled in applications reside with full path so /usr/local/{bin,sbin,share,...}. If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically). We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet. There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.
qbasic_forever 3y ago
Drop in files (config in .d folders) are a core extension point of systemd and not a symptom of an overly engineered config system. These are how services are meant to be custom tailored to specific machines and even user's needs. Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again. Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.

by Eduard 3y ago

nick__m 3y ago

systemd configuration lookups are lightweights and straightforward compared to ansible variables lookups rules : -- Understanding variable precedence Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables): 1 command line values (for example, -u my_user, these are not variables) 2 role defaults (defined in role/defaults/main.yml) 1 3 inventory file or script group vars 2 4 inventory group_vars/all 3 5 playbook group_vars/all 3 6 inventory group_vars/* 3 7 playbook group_vars/* 3 8 inventory file or script host vars 2 9 inventory host_vars/* 3 10 playbook host_vars/* 3 11 host facts / cached set_facts 4 12 play vars 13 play vars_prompt 14 play vars_files 15 role vars (defined in role/vars/main.yml) 16 block vars (only for tasks in block) 17 task vars (only for the task) 18 include_vars 19 set_facts / registered vars 20 role (and include_role) params 21 include params 22 extra vars (for example, -e "user=my_user")(always win precedence) ‐‐ that is a RedHat-isim

oblio 3y ago
Meh, it's not Red Hat. Configuration management tools do this. Chef is similar.
- eropple 3y ago
  Chef is worse. It's discouraged to use `override` in Chef cookbooks but nothing says that you can't, and introducing spooky action at a distance is just part of your life now.
pferde 3y ago
IIRC, Ansible had this exact variable precedence long before they were acquired by RH. I get that hating RH is in vogue currently, but let's keep it grounded in reality please. EDIT: And for the record, these Ansible precedence rules are not a bad thing. Once you are somewhat familiar with them, it allows you to write incredibly flexible and extensible playbooks and roles.
- TheNewsIsHere 3y ago
  As an Ansible person, I couldn’t agree more. Being able to manage variables at basically any level of precedence is extremely useful. I am also ever more deeply concerned about IBM’s hell-bent purge of all the good left at Redhat. I had thought it would take them less time to destroy Redhat, but I’m sure they wanted to make sure they spent time “listening” before making all these ecosystem-eroding changes.
hedora 3y ago
I love how there are 21 places to look if you're trying to figure out why a CLI parameter is being ignored. Does it at least print a warning or something?
- faangsticle 3y ago
  Other way around. Cli params have highest precedence.

crote 3y ago

That doesn't look too bad to me. ".d" directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn't want the config for your web server in the same file as the mail server. /usr/lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. /etc/ is where the administrator can place config files, providing system-specific overrides. /run/ is for files "created at runtime", I am not quite sure what the point of those is.

andrewaylett 3y ago
/run is useful for configuration generation. Sure, you could maybe poke an API and have it do the right thing, but if "everything is a file" then why not put your configuration in a file? That way it's easier to see what's actually happening. Systemd is also a useful demonstration of how to show people where configuration comes from -- `systemctl status <unit>` will include a list of all the files found during lookup, and `systemctl show <unit>` will give you the resolved unit configuration including all the defaults.

Arnavion 3y ago

>Over-elaborated hierarchies where configuration files may reside, and their precedence schemes. >Read the manpage for all the hilarity. If you find it hilarious it's because you don't understand the reason. /usr belongs to the distribution and /etc belongs to the user. The distribution doesn't touch /etc and the user doesn't touch /usr. Upstream ships a unit with its own opinion of the defaults. That unit goes in /usr. Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into /usr too since it's a distribution file. User would like to override a default. They should not edit /usr because that belongs to the distribution. They drop their override in /etc. Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in /run. These are functionally "system-provided units" because users must be able to override them, so they're below /etc in the hierarchy. The "defaults" and "overrides" we're talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets / changes it. If they want a whole file to be different, they can add a new file with the same name that is empty / has different values for its settings. BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them. >And this is just an excerpt which doesn't cover the details. The only detail your excerpt doesn't include is the consideration of `service.d/`, `socket.d/` etc directories which provide overrides for all units of that type. It's just one point, not commonly used, and not that big of a deal either.

stinos 3y ago
and the user doesn't touch /usr. So it's not hilarious statements like this make it hard to understand the reason? I almost cannot tell whether your whole post is sarcasm, or just showcasing irony.
- bayindirh 3y ago
  Yes, "The user (syadmin) doesn't touch /usr, except /usr/local", where compiled in applications reside with full path so /usr/local/{bin,sbin,share,...}. If you're deploying an especially big application, you may also prefer /opt/$APPNAME/{bin,etc,usr,...} (A full filesystem tree, basically). We manage a fleet of systems at work, and I don't remember editing or touching a file under /usr, unless the period I was developing .deb or .rpm packages to deploy on said fleet. There are many written and unwritten rules in Linux world, and sometimes people disregard or ignore these rules so blatantly, so I can't understand whether the post I'm replying to is a sarcasm or irony.

qbasic_forever 3y ago

Drop in files (config in .d folders) are a core extension point of systemd and not a symptom of an overly engineered config system. These are how services are meant to be custom tailored to specific machines and even user's needs. Let's say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again. Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn't live inside it or conflict with upstream changes.