19 ms·

Everything that uses configuration files should report where they're located

• vaughands 3y ago
  A quick and dirty way to do this is run the program through `strace` and find out what syscall it issues to read it. I had to do this to figure out where Cold Steel was reading save files from years back: https://vaughanhilts.me/blog/2018/02/16/playing-trails-of-cold-steel-on-linux-wine-guide.html https://vaughanhilts.me/blog/2018/02/16/playing-trails-of-co...
  • rafark 3y ago
    Interesting. I didn’t know you could do this. I can see how this can come in handy. Apparently there’s dtruss as an alternative for OS X. https://8thlight.com/insights/dtrace-even-better-than-strace-for-os-x https://8thlight.com/insights/dtrace-even-better-than-strace...
    • sharikous 3y ago
      No more. At least with apple Silicon dtrace is completely unusable.
      • comex 3y ago
        It still works if you turn off SIP. This is the same on Apple Silicon and Intel. However, for these purpose of tracking file accesses, I recommend using `eslogger` instead, as it doesn’t require disabling SIP and is faster, among other advantages.
        • djxfade 3y ago
          You can also use Activity Monitor and view process details to see all open file handles etc
          • lost_tourist 3y ago
            Sure but I don't think that most programs keep their config files open the whole time they're running.
        • viraptor 3y ago
          On M2 it is completely broken. Any usage of dtrace will hang the whole desktop. (Not sure if just the desktop or is it the whole kernel crashing) Feel free to reference my feedback entry FB12061147 if you're reporting the same.
        • k4ch0w 3y ago
          Use fs_usage. https://ss64.com/osx/fs_usage.html https://ss64.com/osx/fs_usage.html It’ll show all file events. No need to disable SIP. SIP is doing a lot of good work for users and unless you’re doing kernel work or low level coding I’d keep it enabled. Obv. There are other cases but for the general public keep it on.
          • zarzavat 3y ago
            It’s personal preference. If the security people had their way we’d all be developing on iPads. If SIP interferes with your work: turn it off. Linux doesn’t have SIP and it’s just fine to develop on Linux as well.
            • k4ch0w 3y ago
              100% agree. We would put everyone on chromebooks if we had our way. I don’t think it’s good for productivity and generating new ideas for a company so I never advocate for it.
      • tanelpoder 3y ago
        There's also fs_usage that doesn't require disabling SIP: https://mohit.io/blog/fs_usage-trace-file-system-calls-on-mac-os-x/ https://mohit.io/blog/fs_usage-trace-file-system-calls-on-ma...
      • chiggsy 3y ago
        There was a man who never existed named Thomas Covenant, created by Stephen R Donaldson. Decades ago this character said something that has stayed and will stay with me my entire life: "The best way to hurt somebody who's lost everything is to give him back something that is broken." For us, this thing is MacOS. I miss dtrace every damn day. DTrace allowed you to ask the damn os what it was doing, since the man pages are random and do not match the command line help, new daemons constantly appear with docs like this: NAME rapportd – Rapport Daemon. SYNOPSIS Daemon that enables Phone Call Handoff and other communication features between Apple devices. Use '/usr/libexec/rapportd -V' to get the version. Dogshit.
        • gjvc 3y ago
          OS X was the last chance the industry had to make a "commercial unix workstation", and Apple came >< close. Stallman was right.
  • p-e-w 3y ago
    strace uses ptrace, which some programs disable with prctl as a security measure. So while this can be a useful hack, it doesn't always work.
    • chungy 3y ago
      Can't disable DTrace :) Though DTrace is "only" available on Windows, Mac OS X, Solaris, illumos, and FreeBSD. Oracle has relicensed DTrace to be Linux-friendly and even made kernel patches, but it'll probably never end up in the mainline kernel.
      • viraptor 3y ago
        On Linux you can use bpftrace which is basically the same thing with slightly different names.
      • oblio 3y ago
        > Though DTrace is "only" available on Windows Is it stable and production ready for heavy duty workloads?
        • chungy 3y ago
          Microsoft ships it, so I assume so? I haven't personally ran any versions beyond Windows 7, I can't directly comment on any features after that.
    • patrec 3y ago
      You can use sysdig instead, which doesn't use ptrace[1] and is much faster (as well as generally more pleasant and powerful). [1]It used to have its own kernel extension but is eEBF based these days.
      • everybodyknows 3y ago
        Users should be aware that `sudo apt install sysdig` may require reconfiguration of UEFI Secure Boot, and that there is no apparent clean abort from this possible. The raw-mode terminal screen contains the text: ┌────────────────────────┤ Configuring Secure Boot ├────────────────────────┐ │ │ │ Your system has UEFI Secure Boot enabled. │ │ │ │ UEFI Secure Boot requires additional configuration to work with │ │ third-party drivers. │ │ │ │ The system will assist you in configuring UEFI Secure Boot. To permit │ │ the use of third-party drivers, a new Machine-Owner Key (MOK) has been │ │ generated. This key now needs to be enrolled in your system's firmware. │ │ │ │ To ensure that this change is being made by you as an authorized user, │ │ and not by an attacker, you must choose a password now and then confirm │ │ the change after reboot using the same password, in both the "Enroll │ │ MOK" and "Change Secure Boot state" menus that will be presented to you │ │ when this system reboots. │ │ │ │ If you proceed but do not confirm the password upon reboot, Ubuntu will │ │ still be able to boot on your system but any hardware that requires │ │ third-party drivers to work correctly may not be usable. │ │ │ │ <Ok> │ │ │ └───────────────────────────────────────────────────────────────────────────┘ This leads to a second screen asking for a new password. There is a <Cancel> option, but selecting it merely loops back to the first screen. Hitting C-c (Control-c) has no effect. There are validity rules for the password, but they are presented only after you've entered an invalid password, twice, the second time to confirm. After this installation proceeds, then terminates with a warning: DKMS: install completed. Processing triggers for man-db (2.9.1-1) ... Processing triggers for libc-bin (2.31-0ubuntu9.9) ... W: Operation was interrupted before it could finish Where the "W:" above is colored red. Distro was Ubuntu 20.04.
  • w4rh4wk5 3y ago
    +1 for putting together that blog post on Cold Steel. Greetings from PH3 ;)
    • vaughands 3y ago
      Thank you for your efforts! These days Proton makes a lot of this unnecessary. I hope Trails games will continue to at least function on Linux. :)
  • EdwardDiego 3y ago
    All that time I spent dicking around with lsof...
  • userbinator 3y ago
    On Windows, Process Monitor does the same thing --- with the additional feature that, since configuration is often stored in the registry there, it will also show which registry keys it touches.
  • hoherd 3y ago
    I do this with Brendan Gregg's opensnoop, which is standard on macOS. https://www.brendangregg.com/blog/2014-07-25/opensnoop-for-linux.html https://www.brendangregg.com/blog/2014-07-25/opensnoop-for-l... https://github.com/brendangregg/perf-tools/blob/master/opensnoop https://github.com/brendangregg/perf-tools/blob/master/opens...
  • kevincox 3y ago
    I do this much more frequently than I would like. But occasionally you have issues where the program lists a directory then only acts on specific names. So the strace output won't tell you what the expected name is.
  • westurner 3y ago
    ps wxa | grep <cmd> # or pgrep strace -f -f -e trace=file <cmd> | grep -v '-1 ENOENT (No such file or directory)$' IIRC there's some way to filter ENOENT messages with strace instead of grep? Strace: https://en.wikipedia.org/wiki/Strace https://en.wikipedia.org/wiki/Strace
    • msm_ 3y ago
      I use quick and dirty greps all the time, even when there is a "better" option available. It just works and is very intuitive in interactive contexts. Probably GP works in a similar way.
      • westurner 3y ago
        Remembered to add the `2>&1` stderr>stdout output redirection: ps wxa | grep <cmd> # or pgrep strace -f -f -e trace=file <cmd> 2>&1 | grep -v '-1 ENOENT (No such file or directory)$' Bash manual > 3. Basic Shell Features > 3.6 Redirections: https://www.gnu.org/software/bash/manual/html_node/Redirections.html https://www.gnu.org/software/bash/manual/html_node/Redirecti... In lieu of strace, IDK how fast `ls -al /proc/${pid}/fd/*` can be polled and logged in search of which config file it is; almost like `lsof -n`: pid=$$ # this $SHELL's pid help set; # help help (set -B; cat /proc/"${pid}"/{cmdline,environ}) \ | tr '\0' $'\n' (while 1; do echo \#$pid; ls -al /proc/${pid}/fd/*; done) | tee "file_handle_to_file_path.${pid}.log" # Ctrl-C It's good Configuration Management practice to wrap managed config with a preamble/postamble indicating at least that some code somewhere wrote that and may overwrite whatever a user might manually change it to (prior to the configuration management system copying over or re-templating the config file content, thus overwriting any manual modifications) ## < managed by config_mgmt_system_xyz (in path.to.module.callable) > ## </ managed>
        • hawski 3y ago
          I prefer to use process substitution (not in POSIX): strace -f -e trace=file -o >(grep -v ENOENT) <cmd>
• malf 3y ago
  strace -e open
• p-e-w 3y ago
  I don't think I've ever encountered a widely used program that doesn't have that information in its manpage. The manpage is the canonical source the article calls for. With the exception of New Age software like Go, that for some reason doesn't use manpages, I'm not convinced that the problem implied here actually exists.
  • vaughands 3y ago
    It's more common to have to guess in blackbox / proprietary software, like games.
  • BaculumMeumEst 3y ago
    Doesn't the man page often list several potential locations where it looks for the file and the ordering behavior, leaving you to investigate and find which is actually loaded? It's still annoying.
    • p-e-w 3y ago
      But that is exactly the information you need in order to determine which config file is being read in all situations. A single location doesn't cut it if the reality is more complicated. If the goal is to find out which particular file is being used at this particular time, on this particular system, the maximum verbosity log might have that information, and otherwise strace can help, as described in the other comments.
      • eptcyka 3y ago
        I really do not care about all the possible paths a config file might be read from. I want a flag that I can pass to a binary to see which paths it will actually read, and ideally, I'd like those paths to be logged to STDOUT on startup.
        • em-bee 3y ago
          but it reads all of them in a particular order and in case of conflicting settings you have to figure out which one takes precedence.
• sleepytimetea 3y ago
  Let me introduce you to the nightmare that is the Windows Registry...
  • poisonborz 3y ago
    Luckily modern programs rarely use it.
  • fsckboy 3y ago
    don't give the systemd crowd any ideas, the Windows Registry is right up their alley
    • RandomBK 3y ago
      Dbus says hi
    • fsh 3y ago
      All parts of systemd are configured entirely using plain text config files and symlinks.
      • mike_hock 3y ago
        It's indeed "configured using symlinks" but not in the sense that you're free to use symlinks the way they're supposed to work but in the sense that if you don't want to use systemctl to place them, you can place them "by hand" if you know all the magic ways in which systemd interprets them and will remove them as it sees fit when you invoke a management command.
        • fsh 3y ago
          I have no idea what you mean by that. A systemd target is a folder in /etc/systemd/system/ which contains a bunch of symlinks to unit config files. When the target is requested, systemd makes sure that all these units are running. This is how symlinks have been used for configurations since forever (for example by apache on Debian).
• smcin 3y ago
  The title read without context sounds like it's suggesting they should report which IP address/country/jurisdiction they're in. To any random user on the internet. Which would be insane. But it's really only saying "which system path they're in, to e.g. a sys admin".
  • mdekkers 3y ago
    Possibly if you are not familiar with how software is configured. It was immediately clear to me.
    • smcin 3y ago
      I am familiar with how software is configured, and the implied context is not clear from only the headline, hence the headline needs fixing. "Everything" is massive overstatement: bank websites use configuration files, so if hackers try to log in to the bank, the website should report where the config files are located? Nope. Headline way too broad.
• tomaaron 3y ago
  Configuration files are like parents’ advice - carefully thought out, well-intentioned, but often overwritten by the ‘environment variables’ of real life experiences!
  • formerly_proven 3y ago
    I have this tool, which is used in a few different environments, so I gave it a “xxx config” command which tells which config files were read, which environment variables were used, which environment variables from the tool’s XXX_ namespace are unknown, what configuration values are effective etc. I don’t think anyone ever used that function.
• guggle 3y ago
  All the software I write is configured through env vars... why use anything else ?
  • withinboredom 3y ago
    You can’t change env vars after a program is running, but I can change a file and send a SIGHUP.
    • deleted 3y ago
      [deleted]
    • ilyt 3y ago
      Programs reconfigurable at runtime are by far rarity, as that's generally pretty hard thing to do. But yes, just have a config file, ENVs are nice as a backup for small apps, but never as only way
      • withinboredom 3y ago
        Anything running as a daemon/service should support this capability IMHO, especially if it is servicing active traffic.
        • ilyt 3y ago
          It's definitely nice when it does but it's also a whole lot of complexity to add to the code. You have to re-create all structures relying on configs, reconnect if needed, clear all previous ones, but only after any ongoing processing of requests received before the reload signal finishes. Some go half-way, like HAProxy where it does spawn a new process but that process just binds with SO_REUSEPORT and signal old one to close the socket and finish all remaining connections. So you effectively get hitless reload, it's just that old process is the one finishing in-flight requests
          • withinboredom 3y ago
• tester756 3y ago
  Also, I do believe that programs should be capable of self-configuration like if you need LLVM and didnt manage to find it under "app_folder/llvm" then let's go find it somewhere on the disk(s) and ask user if he wants to use it
  • hughesjj 3y ago
    System environment variables/system registry sounds like the best way to do that from a pragmatic perspective
• lewo 3y ago
  The main issue is to use configuration files residing somewhere in the filesystem. This looks like a global variable in a codebase (something we generally try to avoid). Instead, the configuration file should be explicitly provided as a command line argument. Systemd sandboxing can also be useful to ensure the program only uses the expected set of files. For instance, on my NixOS machine, the Nginx configuration is not in `/etc/nginx` but explicitly provided and can then be known with ps: $ ps aux | grep nginx nginx: master process /nix/store/9iiqv6c3q8zlsqfp75vd2r7f1grwbxh7-nginx-1.24.0/bin/nginx -c /nix/store/9ffljwl3nlw4rkzbyhhirycb9hjv89lr-nginx.conf
  • ris58h 3y ago
    It's not a bad idea but it's not applicable to every piece of software. I don't think that passing a config file for every git command would be convenient.
    • viraptor 3y ago
      You can change the commandline string at runtime. You could inject a fake "-c correct/path" even if it's not there. (That's useful for other things too, like injecting the git revision of the app into the commandline)
  • jamann 3y ago
    How long does it take you to type out that path?
    • em-bee 3y ago
      /nix/store/9ii<tab>nginx.conf
  • Karellen 3y ago
    > This looks like a global variable in a codebase (something we generally try to avoid). Aren't they more like global constants than variables? Loaded at startup, and never change during that run of the program. (With the exception of only explicitly being re-read on SIGUSR1 for daemon-like programs.) And global consts, or #defines, or whatever, are things we generally don't try to avoid?
• antirez 3y ago
  $ redis-cli INFO ... executable:/Users/antirez/hack/redis/src/./redis-server config_file:/etc/redis/redis.conf ... In the case of Redis, it is possible to do much more. With the CONFIG command you can check specific configuration parameters at runtime, and even change most of them. There are commands to get reports about latency and memory usage, performing a self-analysis of the instance condition. You can see counters for commands called, and even obtian a live stream of the commands the instance is executing. This should all be obvious features to make system software approachable, observable, and to make people able to act easily when shit happens, without having to guess where the configuration file is from the system startup scripts or alike. UX is not only about GUIs.
  • themoonisachees 3y ago
    Redis has consitently been the nicest program to admin everytime i have to interact with it as a sysadmin
• Eduard 3y ago
  Programs should not only report where the configuration files are located, but also report how configuration options contained therein are read in: Some programs use a "last occurrence wins" approach, while other programs (e.g. sshd) use a "first occurrence wins" approach. Buried in https://man.freebsd.org/cgi/man.cgi?sshd_config(5) https://man.freebsd.org/cgi/man.cgi?sshd_config(5) and awkwardly expressed: "Unless noted otherwise, for each keyword, the first obtained value will be used." That's why e.g. Debian's /etc/ssh/sshd_config ( https://salsa.debian.org/ssh-team/openssh/-/blob/debian/1%259.3p1-1/sshd_config?ref_type=tags#L13 https://salsa.debian.org/ssh-team/openssh/-/blob/debian/1%25... ) starts with Include /etc/ssh/sshd_config.d/*.conf , so that custom configurations in that directory win over the distribution-provided defaults set up after this Include directive inside /etc/ssh/sshd_config.
  • Eduard 3y ago
    Also: processes / running programs should provide a straightforward way to show their configuration. While "sshd -T" shows the effective configuration from all configuration file parsing, there is no straightforward way to see the configuration of the/a currently running sshd process. This has footgun potential, as someone could think their sshd is secured and hardened by checking with "sshd -T", while the actually running sshd process still uses old configurations. Apache also makes it needlessly hard to find out its effective (and active / currently running) configuration. I don't know of any better way then awkwardly using mod_info ( https://httpd.apache.org/docs/2.4/mod/mod_info.html https://httpd.apache.org/docs/2.4/mod/mod_info.html ) to get such insights.
    • mike_hock 3y ago
      Not sure I get the reasoning here. That requires you to actually go and proofread the configuration of each and every running process, at which point it's easier to just verify the effective configuration on disk and restart the service. SSH is a special case because it leaves active sessions alive when it restarts, but that doesn't generalize to other daemons. Just restart if in doubt.
  • Shrezzing 3y ago
• klabb3 3y ago
  I maintained a CLI tool for a long time. I came to the conclusion that good config is a bit harder and more subtle than most people think. I agree with the premise of the article, but there are more layers to it, literally. Depending on how much config you have, sooner rather than later you’ll have multiple sources: system defaults, user config files, sometimes profiles, env vars, and flags. Configuration may be sourced from a combination of these, so overrides need to be handled and naming needs to be consistent. Another source of mess is deeply nested config, such as for a user-defined project. Those are very awkward to add flags for, in which case it’s better to enforce a file (like kubernetes config). To me, tools like git strikes a good balance. And by that I mean that project-local config is in a project root directory that’s not necessarily checked in to VCS, but picks up the config even if you’re in a subdir. It rarely causes any unwanted surprises, but at the cost of being somewhat flag heavy, which is then alleviated with user-defined aliases.
• el_snark 3y ago
  Its got my vote. Speaking as someone contemplating a Solr installation left without any documentation by a previous employee.
• est 3y ago
  and every configurtion file should be read-writable by some kind of UI. Especially those out-of-context .yaml files with lots of DSLs embedded.
• ilyt 3y ago
  As a system administrator I want 2 things from config files: * loading a directory of config files, so various Configuration Management tools have easier time deploying config (especially for multi-use apps like say syslog daemon, where multiple disparate apps might want to put a part of the config in) * a function to dry-run config validation, so CM can alert about that instead of restarting an app to a wrong config Finding where is one has never been a problem.
  • themoonisachees 3y ago
    Seconding dry-run, while looking intently aws-cli for haveing the worst possible dry-run option imaginable.
• personjerry 3y ago
  Also log files!
• quickthrower2 3y ago
  So a “registry”, of sorts :-)
• atoav 3y ago
  I'd add: everything should use configuration files that are stored on the machine. I know services that use configuration files and you are expected to upload them into the service using an IDE and a plugin to then store the config in it's database.
  • em-bee 3y ago
    not supporting configuration files on disk allows a service to work without needing read or worse write access to the disk. storing configuration in the database is a benefit. (potentially even for security)
    • atoav 3y ago
      Yeah but then at least include a cli interface that runs on that machine which allows you to edit the config.
• hamburglar 3y ago
  Lately I've written all my tools to not only announce where they're reading their configs from (which isn't precisely what OP is asking for but is close), but also to have an option to generate the default config that will be presumed if a config file is not found. "mytool --gen-config /tmp/newconfig.txt" should produce a fully-formed config with all defaults explicitly set which can be moved into /etc/mytool.conf or wherever it's supposed to go, all ready for me to tweak without having to look up settings names.
• c0nsumer 3y ago
  In my day job I do a lot of troubleshooting. That's most of my job... helping other IT folks figure out what the heck is happening with something. I'd say that in 70% of things I work on, part of that time is figuring out what config files are used because the app/service/process/whatever is doing something the owner/tech/whatever doesn't expect, yet is rational, and they "didn't tell it to do that". It'd be great if everything reported as the article mentions -- amazing, in fact -- but I feel like it's a pie in the sky wish. Like asking that all programs always exit cleanly or something. I feel like there'll always be inevitable edge cases -- libraries that themselves pick up config files or registry values or environment variables that aren't expected, who knows -- that'll need to be discovered. If things are already not going right with a program, I'm less likely to trust what it says it's doing and more likely to just watch it and see what it's actually reading and doing.
  • CoastalCoder 3y ago
    If any of the frequent offenders are open source, I wonder if ot would be worth your while to submit (or get someone else to) a change to add this functionality.
    • coldtea 3y ago
      "Submit a change and they'll include it" is the demo/promotional version of FOSS
      • CoastalCoder 3y ago
        I suspect that varies from one project to the next.
  • johnchristopher 3y ago
    Yesterday I was wondering why the self signed root certificate and the certificates I added to my debian install didn't work with httpie (ssl warning) but did work with curl (and the browsers I added the RootCA to). I found the explanation: https://github.com/httpie/httpie/issues/480 https://github.com/httpie/httpie/issues/480 httpie doesn't look into /usr/local/share/ca-certificates but apparently it's not httpie's fault, it's the python requests library that doesn't check that folder. I don't know what to think of it and who is supposed to fix it. But I am back to curl with my tail between the legs (because I oversold httpie to my coworkers) for now. There's only one place where the bucket stops with curl.
• paveworld 3y ago
  just try using the unix command lsof if you know the PID aka Process ID for the application yr running try lsof -p PID_NUM_HERE it will give you a list of every open file your application is using along with it's full path finally, if that gave you too many lines try lsof -p PID_NUM_HERE | grep -E 'txt|properties|config' I know you wanted the files listed it in a help file however this will work 100% of the time Best, Stephen
  • rossmohax 3y ago
    this will work 0% of the time if app closes config after reading
• YourDadVPN 3y ago
  I think the best way to achieve this is by providing an OS API that results in the files always being created in the same place. Applications/libraries could still choose their own filenames and syntax, just the location would be OS controlled. I think there is room for a new desktop/laptop OS to emerge and one good idea from mobile OS design I would like to see is having everything be an API call that allows the OS to impose standardisation, permissions and user preferences rather than the free-for-all desktop OSes have (though I propose letting the user run non-compliant applications, and not porting the iOS app store only model into the OS).
  • AeroNotix 3y ago
    You're basically suggesting the windows registry.
    • angulardragon03 3y ago
      macOS has something similar also with defaults [1]. https://macos-defaults.com/#%F0%9F%99%8B-what-s-a-defaults-command https://macos-defaults.com/#%F0%9F%99%8B-what-s-a-defaults-c...
    • aeonik 3y ago
      The problem with the Windows registry, at least back in the day, was that it was a single file that could be corrupted and it would wreck your whole system. I think having a standard utility API for *nix configs makes a lot of sense. I'm surprised it doesn't exist. I tried to find one, and there are some libraries for reading and parsing in every language, but nothing that seems to cover everything. This bash script seems to be a fairly "built in" way to parse them: https://unix.stackexchange.com/questions/441076/which-is-the-standard-configuration-parser-library-used-in-linux https://unix.stackexchange.com/questions/441076/which-is-the...
      • feldrim 3y ago
        > The problem with the Windows registry, at least back in the day, was that it was a single file that could be corrupted and it would wreck your whole system. Exactly. But Registry has been using NTFS and it's capabilities for rollbacks and recovery. Therefore, the problem is mostly solved. There are occasions [0] of bugs causing corruptions though but they are very rare. 0. https://www.bleepingcomputer.com/news/microsoft/windows-10-ntfs-corruption-bug-gets-unofficial-temporary-fix/ https://www.bleepingcomputer.com/news/microsoft/windows-10-n...
• joelthelion 3y ago
  A good and underused tool to figure this out when the tool doesn't help is strace. It allows you to trace system calls, and, crucially, file opens.
• mixcocam 3y ago
  The config should also come with a short explanation on how to use it on it!
• MattPalmer1086 3y ago
  I just implemented a "--config" command in my latest code that reports exactly what was used for configuration, files and environment variables, and where the files are actually located. Makes life a lot easier imho.
• kdeldycke 3y ago
  If you maintain a Python CLI, you can use Click-Extra [1]. It provides a ready-to-use --config option which reports how it sources the configuration file [2]: $ my-cli --verbosity DEBUG subcommand debug: Load configuration matching ~/.config/my-cli/*.{toml,yaml,yml,json,ini,xml} debug: Pattern is not an URL. debug: Search local file system. debug: No configuration file found. (...) It also adds an auto-magic --show-params [3] to help you see where parameters are coming from: $ cli --int-param1 3 --show-params ╭─────────────────┬─────────────────────────────────────────┬─────────────────────────────────────────┬──────┬──────────────────┬─────────┬─────────────────┬─────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────┬─────────────╮ │ ID │ Class │ Spec. │ Type │ Allowed in conf? │ Exposed │ Env. vars. │ Default │ Value │ Source │ ├─────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼──────┼──────────────────┼─────────┼─────────────────┼─────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼─────────────┤ │ cli.color │ click_extra.colorize.ColorOption │ --color, --ansi / --no-color, --no-ansi │ bool │ │ │ CLI_COLOR │ True │ True │ DEFAULT │ │ cli.config │ click_extra.config.ConfigOption │ -C, --config CONFIG_PATH │ str │ │ │ CLI_CONFIG │ /home/runner/.config/cli/*.{toml,yaml,yml,json,ini,xml} │ /home/runner/.config/cli/*.{toml,yaml,yml,json,ini,xml} │ DEFAULT │ │ cli.help │ click_extra.colorize.HelpOption │ -h, --help │ bool │ │ │ CLI_HELP │ False │ False │ DEFAULT │ │ cli.int_param1 │ cloup._params.Option │ --int-param1 INTEGER │ int │ │ │ CLI_INT_PARAM1 │ 10 │ 3 │ COMMANDLINE │ │ cli.int_param2 │ cloup._params.Option │ --int-param2 INTEGER │ int │ │ │ CLI_INT_PARAM2 │ 555 │ 555 │ DEFAULT │ │ cli.show_params │ click_extra.parameters.ShowParamsOption │ --show-params │ bool │ │ │ CLI_SHOW_PARAMS │ False │ True │ COMMANDLINE │ │ cli.time │ click_extra.timer.TimerOption │ --time / --no-time │ bool │ │ │ CLI_TIME │ False │ False │ DEFAULT │ │ cli.verbosity │ click_extra.logging.VerbosityOption │ -v, --verbosity LEVEL │ str │ │ │ CLI_VERBOSITY │ WARNING │ Debug │ COMMANDLINE │ │ cli.version │ click_extra.version.VersionOption │ --version │ bool │ │ │ CLI_VERSION │ False │ False │ DEFAULT │ ╰─────────────────┴─────────────────────────────────────────┴─────────────────────────────────────────┴──────┴──────────────────┴─────────┴─────────────────┴─────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────┴─────────────╯ [1]: https://github.com/kdeldycke/click-extra https://github.com/kdeldycke/click-extra [2]: https://kdeldycke.github.io/click-extra/config.html https://kdeldycke.github.io/click-extra/config.html [3]: https://kdeldycke.github.io/click-extra/parameters.html#show-params-option https://kdeldycke.github.io/click-extra/parameters.html#show...
• thecosmicfrog 3y ago
  I wish more programs had an obvious way of showing what configuration values are currently set. Something like `program --config` with no arguments should print all configuration keys and their current values.
  • reidacdc 3y ago
    Seconding this, for sure. "/usr/sbin/sshd -T" (T for "test") has been a life-saver more than once. My strongest version of this is that any application that manages any state should be able to report on that state in a human- and machine-readable way.
    • efreak 3y ago
      This doesn't tell you the current running configuration, it tells you the current configuration on disk. These should be the same, but aren't always. I personally have a nasty habit of forgetting to reload configs. I make sure to leave them in a syntactically valid state, but I really don't care if someone can't access my webserver for a while; if I have to go, I add a quote and a curly bracket, save, and close my ssh session. If my 5 minutes away runs into tomorrow, then I no longer remember what's been changed (I manage this these days by making /etc a git repo with an ignore list and stashing incomplete changes...then forgetting to commit them when I'm done)
• rewmie 3y ago
  This is a high quality blog post. It raises awareness of a usecase that's always been there but it might never surfaced, it goes straight to the point, and provides clear examples. It's a blog post where everyone ends up being slightly better after reading it.
• nilslindemann 3y ago
  Basically the location of everything should be exposed to the user, especially if he is the reader of the source code.
• nraynaud 3y ago
  thanks for pointing that out, I'll make sure to include it next time I write a new application.
• throwawaaarrgh 3y ago
  We could use pkg-config for that. It's meant for build systems but you could store any metadata in there, in theory. https://en.m.wikipedia.org/wiki/Pkg-config https://en.m.wikipedia.org/wiki/Pkg-config https://people.freedesktop.org/~dbn/pkg-config-guide.html https://people.freedesktop.org/~dbn/pkg-config-guide.html
• smcameron 3y ago
  List them in the FILES section of the man page. You have a man page, right? Also, follow the XDG base directory specification[1]. [1] https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-...
  • JdeBP 3y ago
    systemd has manual pages. I pointed out some missing configuration file locations back in 2016. They are still undocumented today. * https://jdebp.uk/FGA/systemd-documentation-errata.html https://jdebp.uk/FGA/systemd-documentation-errata.html
  • Rediscover 3y ago
    And also `strace`.
• yread 3y ago
  Especially if there are multiple places to read them from: current dir, PATH, classpath, $PROJECT_HOME, database, env vars... With env vars it would be handy to also know where they come from, is there a way? like having a timestamp when it was set, by what user and process.
• sgtnasty 3y ago
  Are there any good examples of projects that do this really well?
• hyperman1 3y ago
  I also vastly prefer it when error messages give a hint about what reconfiguration you can do to fix it, including correct syntax and googleable terms. So don't do: farblewarble too low. Instead: FarbleWarble is 1234 but need at least 5678. Edit /etc/whatever.conf and add: FarbleWarble="5678" There is a special hell for programs that talk abut how the Burble is configured incorrectly , without any mention of FarbleWarble
• itomato 3y ago
  This is how we wind up with three implementations of a “Windows-style Registry” while also having all the other management modes; local dotfiles,system dotfiles, all of /etc and whatever is allowed under /opt. Be smarter about package management and this is a non-issue; checksums, install-time dumps of file modifications, and the like. It’s why tools like CFEngine and the rest were invented, and I’m upset this is still an issue for people like OP twenty years later.
• usrbinbash 3y ago
  Not only that, but past a certain complexity of the config, programs should have a way to dump their runtime config to the command line. Yes, I am talking to you, server daemons that take configs from 2-3 different rc-file locations, hardcoded defaults, envvars, the moon phase and the current market value of dried parsnips. If you don't have a way to simply tell me what your current config is, then you have no earthly right to run it.
  • hans_castorp 3y ago
    I like the way that Postgres handles this. There is a system view on top of their configuration file(s) that reports from which file and line number the currently active value was taken from.
• xbmcuser 3y ago
  I feel no standards for config files is probably the biggest reason app containers got traction with advanced home users. Save your docker containers nuke your server install move to a new distro and reimport back the app container.
  • laeri 3y ago
    There are standards. The article even mentions the xdg base dir standard. The problem is that not everyone follows them but a large number still do and then it is easy to find them. I still agree that app containers are still a good idea.
    • metaltyphoon 3y ago
      When standards are not enforced, they became optional and IMO not really a standard.
      • laeri 3y ago
        The main culprits are programs that were written before the standard existed in addition to many people that just don't know about it. Who should enforce this standard? I would say community pressure does it pretty well at the moment. A standard does not need to be enforced and the XDG base directory one is still useful. The more awareness exists and the more people want this and think it is useful the better. It also has the advantage that no other competing standard exists as far as I know but it still has some downsides.
• egberts1 3y ago
  Fastest way to identify hidden and dynamic configuration filenames: strace -f <executable-file> | grep openat Yeah, I was just looking at the biggest offender that I have encountered: Debian's update-ca-certificates, 37 hidden configuration files.
  • egberts1 3y ago
    Details of hidden files opened by Debian's update-ca-certificates: https://egbert.net/blog/articles/ca-certificates-rebuild-on-debian.html https://egbert.net/blog/articles/ca-certificates-rebuild-on-... /etc/ca-certificates.conf file /usr/share/ca-certificates/* directory /usr/share/ca-certificates/mozilla/* directory /usr/share/ca-certificates// directory /usr/local/share/ca-certificates/* directory /usr/local/share/ca-certificates// directory /etc/ssl/certs directory $CWD/<all-certs-read-before> files /usr/lib/ssl/openssl.cnf file /etc/ca-certificates/update.d directory /etc/ca-certificates/update.d/jks-keystore directory /etc/default/cacerts directory /etc/java-11-openjdk/security/nss.cfg file /usr/share/ca-certificates-java directory /usr/lib/jvm/java-11-openjdk-amd64/lib/jvm.cfg file /usr/share/ca-certificates-java/ca-certificates-java.jar file /etc/ca-certificates/update.d/mono-keystore directory /etc/mono/4.5/machine.config file /etc/mono/assemblies/cert-sync/cert-sync.config file /etc/mono/assemblies/Mono.Security/Mono.Security.config file /etc/mono/assemblies/mscorlib/mscorlib.config file /etc/mono/assemblies/System/System.config file /etc/mono/config file /etc/ssl/certs/ca-certificates.crt file $HOME/.mono/config file /usr/lib/mono/4.5/cert-sync.exe.config file /usr/lib/mono/4.5/cert-sync.exe.config file /usr/lib/mono/4.5/mscorlib.dll.config file /usr/lib/mono/gac/Mono.Security/4.0.0.0_0738eb9f132ed765/Mono.Security.dll.config file /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll.config file /usr/share/.mono/certs/Trust/ski-*.cer file /usr/share/.mono/certs/new-certs/XXXXXXXX.0 file $CWD/openssl EXECUTABLE!!! (why look in $CWD?) /usr/bin/openssl /usr/local/bin/openssl /usr/local/sbin/openssl /usr/sbin/openssl (VERY STRANGE ordering of /usr/[local/][s]bin/
    • efreak 3y ago
      > $CWD/openssl EXECUTABLE!!! (why look in $CWD?) debootstrap, maybe?
• nikau 3y ago
  Alternately, every aysadmin should learn how to drive strace
• em500 3y ago
  I wonder if there is any mainstream program with a more convoluted configuration scheme than bash? Just when think you understand the various invocations of .profile, .bash_profile, .bashrc in login/non-login/interactive/non-interactive modes in home and/or /etc after digesting an article with flowcharts[1], you realize that any invocation can execute arbitrary code from any location, and different distributions (not to mention, macOS, WSL, git-for-Windows, etc) all source different stuff in their default configs (like executing .bashrc in .profile). [1] https://blog.flowblok.id.au/2013-02/shell-startup-scripts.html https://blog.flowblok.id.au/2013-02/shell-startup-scripts.ht...
  • nxpnsv 3y ago
    That graph is wild! Bash is crrrraazy
    • ElevenLathe 3y ago
      I love this post. Such a crazy diagram for something that intuitively we believe should be simple. You can easily imagine most of the the compromises and feature requests over the years that brought it to this state. All software that survives decades of care and upkeep tends to end up in a similar state.
  • hnlmorg 3y ago
    This is a why Murex (my shell) has a ‘runtime’ builtin which lets you query which script updates what. Every function, variable etc can be queried to find where they were set and even when (time stamped) too.
• WeylandYutani 3y ago
  I use Pcgamingwiki so that I can find out where the hell the developers decided to put their config and save game location. Windows is wild.
• mberning 3y ago
  It would also be nice to have an “explain” option where it shows how or why a value was updated. I am specifically thinking of Ansible which has a multitude of locations from which it sources variables. It would be nice to be able to spy on a variable and report every value change and where that change came from.
• bighoki2885000 3y ago
  [dead]
• bartq 3y ago
  I disagree it's responsibility of the tool. Program should just reach out for files and some higher order or meta runner should trace what the process does. One of the commenters mentioned `strace`, this is the correct approach, but should be done in well engineered way. Other way is to run program in symbomlic interpreter and trace what it wants to read in any of the branches.
  • ufo 3y ago
    Might work for config files but would it work for environment variables?
    • bartq 3y ago
      It would for env vars as well, why not? Just trace program trying to reach out for some var X from env.
      • mvnuweucxqokii 3y ago
        afaik reading env vars requires no sys calls and thus will not appear in strace
        • bartq 3y ago
          So there should be a different tool used which will intercept environment variables reads and will log them.
  • hermannj314 3y ago
    This seems to be an interesting approach. Why do you suggest this path over documentating the behavior explicitly? What makes one approach better the other? Is it faster to run a trace? Should a developer trace their own code and auto-generate documentation post-build?
• kaba0 3y ago
  Easier option: sandbox each executable, so every output file will be located in a special folder.
• crabbone 3y ago
  Wasn't Windows Registry a (failed) attempt at unifying this stuff? Now, hear me out. The problem is not the location of configuration. The problem is the crippled and anemic way UNIX (and the likes) programs take configuration. These can only predictably take environment variables and command-line arguments. Both ways are crippled due to size limit (especially the command-line arguments) and structure (you cannot even have lists / arrays in environment variables). This is the reason why every application author believes they must write their own configuration, with their own (crippled) configuration format and logic for locating their configuration. On the bright side, I think, systemd is moving towards making this mess more uniform. On the not so bright side, systemd doesn't appreciate the difficulties associated with the problem and doesn't seem to have any sort of plan moving forward. So, unit files are kind of configuration, but because the format is so crippled and anemic, they immediately added a way to source random external files for process environment etc. Finally, systemd, even if they understood the problem well and put a good effort towards solving it are still powerless against the format in which system supplies input to processes. What really needs to change is all the family of execve, execlp and so on. This was always a stop-gap / bad idea, but it was OK for a while, before programs started to require more complex input. It should've died in the 80s... but today it is so deeply entrenched that removing it will take a very long time. Another problem here is the approach that "everything is a file". So, the only way to provide configuration beyond command-line arguments and environment variables is a file... But, imagine an alternative reality in which OS is capable of providing basic data-structures, which can also be persisted and queried? Kind of like you do it with a relational database? I think this idea is so much not novel that it's older than most people commenting in this thread... And, even though this would make it so much easier for so many application authors to provide consistent and easy to inspect, better tested interface to their programs... nobody's doing that :) Because files are good enough.
  • Roark66 3y ago
    Having spent many years in the windows land I definitely get where you're coming from. However, I find the Unix way where everything is a file, much simpler to work with. Just take a simple task of copying a subset of some system's configuration to another one. On Unix it's just a matter of copying some files. On Windows (and our imaginary is that provides object storage) we would need to find the relevant objects, then write a script to export/import it (or mess with export/import tools for a while). Personally I see "everything is a file" as strength not weakness. In addition to my personal preference having a centralised configuration system not only restricts in many ways, presents a potential performance bottleneck, but also is an extra point of failure. Additionally, I think most Unix software uses a fairly simple text format for config files. There really is little reason for using anything else. If you need complex data structures use YAML. I've found many more proprietary binary config file formats on Windows (despite registry) than on Unix. So if the os would make something like this available there would still be no guarantee all apps use it. It would just be an extra thing to deal with for little benefit IMO.
• Aissen 3y ago
  That's one thing I like about "systemctl cat <service/unit name>": it shows both the content and the path of the config file.
• paulddraper 3y ago
  Like cat /var/lib/dpkg/info/<package>.conffiles ?
• tgv 3y ago
  I'm of the opinion that most programs should get their config files explicitly from the command line. None of that ~/.apprc or $APP_PORT=8080. That's inconvenient for tools you use often, like ls or grep. If they read config files, an option like --ls-config would be very welcome.
  • remram 3y ago
    Interestingly `LS_COLORS` is set by default on most distributions... and you definitely wouldn't want to type that in every time.
    • tgv 3y ago
      Every new account/os installation, I have to hunt around to turn that off. I absolutely dislike colored output in almost all cases. Might be because I'm color-blind. But they keep moving those settings around. Perhaps that part is done in their spare time by the MS Office menu bar team.
• jtode 3y ago
  Have my upvote. So many hours lost to "[software package] config file location".
  • wspeirs 3y ago
    Agreed! Few things anger me more than when a program simply reports, "File not found"... OK, which file? Where were you looking for said file? I know how we get here, people simply report the error from the OS w/out adding context. This is why I love anyhow reporting in Rust, you can attach context (like the file name and path) to the error. In log-store.com I report the default name of the config file (you can specify it via cmdline if you want), and the 3 locations and order searched, if the config file isn't found. I also have it report the file being used, so if you expected the file in /etc, but you accidentally had one in your home directory, you'll know on startup. These things seem like table-stakes in 2023.
• bennettnate5 3y ago
  I've found strace | grep "open" particularly useful for locating config files. They're usually opened right at the start of the program as well. (For reference, strace prints out every system call used by a program with the arguments that were called; occasionally, a program may use something other than `open` or `fopen` to read in a file, so that should be kept in mind when using this method)
  • remram 3y ago
    strace has its own filtering mechanism, so you can do `strace -e open`. Though you might have to add `,openat,openat2` so maybe your way is easier in practice.
• timokoesters 3y ago
  In Conduit (https://conduit.rs https://conduit.rs) there is no default config path. In order to start Conduit, you need to specify the CONDUIT_CONFIG environment variable, which is the path to your config. This will typically be done in a systemd service. This has multiple benefits: - You can't accidentally start Conduit with a wrong config (e.g. the config in the current working directory) - You can have multiple Conduits running at the same time with different config - It's easier to understand for the system administrator because there is no hidden information
  • magarnicle 3y ago
    That seems like a really good solution, for the cost of a little more mucking around to begin with.
• sammy2255 3y ago
  The popular VLC alternative MPV is very guilty of this. I got shut down when I tried to discuss it in a Github issue
• aaron695 3y ago
  [dead]
• kazinator 3y ago
  If I need to know where some program is looking for startup files and in what exact order, I just use strace on it.
• runeks 3y ago
  I would go one step further and say that all sources of runtime configuration should be reportable: * Config files * CLI options * Environment variables And what takes precedence when.
  • bbkane 3y ago
    I try to write all my CLIs this way - to the point of writing my own Go CLI parsing library [0] to do it. Now, when I run `cmd -h`, I get output like the following for all flags (example from [1]): --count , -c : Number of times to dig type : int default : 1 configpath : dig.combine.count required : true currentvalue (set by config) : 1 It's sometimes more verbose than I'd like, but it's quite useful for figuring out how a flag got its current value [0] https://github.com/bbkane/warg https://github.com/bbkane/warg [1] https://github.com/bbkane/shovel https://github.com/bbkane/shovel
  • nmz 3y ago
    this would also really help the CLI in terms of auto completion, I see people go above and beyond writing completion scripts, which could be simplified instead by running --autocomplete or checking some file somewhere.
• morkalork 3y ago
  That's something I think anyone who remembers what being a newbie linux user is like will agree with.
• js2 3y ago
  I needed to update /etc/resolv.conf under RedHat the other day and I was surprised by how much this has changed. You don't edit /etc/resolv.conf directly anymore because it's maintained by Network Manager[1]. Instead you edit either one of the interface configuration files and tell NM (via nmcli) to reload it, or you use nmcli to update the interface config file. Fine, I can deal with that. What really surprised me were a couple programs that needed to be restarted to see the updated /etc/resolv.conf as they apparently don't use glibc for DNS resolution. I used lsof to figure out what was sending DNS requests to the old name server IPs. It turned out to be osqueryd and Apache Traffic Server. 1. There are various ways to put /etc/resolv.conf back under manual control if you don't want NM to maintain it.
  • marcosdumay 3y ago
    > Instead you edit either one of the interface configuration files and tell NM (via nmcli) to reload it Yeah, and this would be fine if both /etc/resolv.conf told you what is creating that file and NM didn't rewrite its own configuration files all the time. Anyway, it would be not that bad if NM was kept restricted to RedHat, because the entire philosophy of that distro is that configuration files are just suggestions and the system is free to change anything at any time. But instead, that behavior infected everything.
    • js2 3y ago
      The first line of resolv.conf is a comment indicating it's written by Network Manager. So that part didn't trip me up at least.
      • marcosdumay 3y ago
        On Debian it only says it's auto-generated and you shouldn't edit. It's not necessarily created by NM either¹, so it would be really helpful to have it. 1 - The policy of whatever configures your network last rewrites this file is spreading. Apparently systemd creates it first, and Debian has more than one network configurator for you to choose.
• dschuetz 3y ago
  Read The Freaking Manpages.
• bombcar 3y ago
  php.ini is my experience of the worst of this, finding which of the fifty five billion config files is the actual one controlling the web server’s php and not the console’s is always a fun journey. Luckily ctrl-r helps the second time.
• vaughan 3y ago
  This is the central problem with all programming today. You can think of everything as configuration. Parameters of a function are just configuration for that function. What we need is traceability of every value in a system, not just configuration. You click on any value and you see a graph/chain of all the transformations this data goes through right back to the source of truth. This is how we should code. Think about how difficult this is to do today with your current IDE/editor. Every task in software is wanting to change what is displayed in a user interface of some sort. Usually you have to make your way through a labyrinth to figure out what to change where, and also what side-effects will occur. If this is not smack bang in front of your face when you are coding, then you are just adding more and more tangled data flows and complexity to your system.
  • yawboakye 3y ago
    i wonder whether our current strong compile-time/runtime distinction can support this. it may be slightly easier for interpreted and vm-based languages but i’m not sure we can have it more broadly. lately i’ve been playing with prolog and revisiting computer science/computer programming before imperative languages drew a curtain on innovation and i think we have lost a lot. our modern programming isn’t the programming the founders had in mind, it seems.
    • cmrdporcupine 3y ago
      I think it could be done for Rust
    • marcosdumay 3y ago
      Compiled languages absolutely can support this. But imperative ones create a bit of a problem. Nothing impossible to deal with, but it becomes harder. But any choice of language, the OS can enforce it too, and maybe in a much easier way than messing with your language.
  • Blackthorn 3y ago
    This would be sooooo nice, especially in DI heavy languages like Java. It might not be turtles all the way down, but finding the final turtle can be so difficult.
• koromak 3y ago
  Managing environments / staging / cicd / configurations is like the hardest part of my job. Its fundamentally messy.
  • marcosnils 3y ago
    Have you seen https://dagger.io/ https://dagger.io/? Managing all that with code has been a life changer to me.
• markus23 3y ago
  It should also report the syntax and so on, and actually you want a way to directly modify configuration values. So if you think this through, you will end up with something like https://www.libelektra.org https://www.libelektra.org
• jdwyah 3y ago
  Strong agreement. I'm building a tool for dynamic configuration and that makes this doubly fun, because now the configuration can come from a remote, live updating source as well. In Prefab, the source can be a default file, env specific defaults, live values, or machine specific overrides when doing local development. Right now I've landed on printing out the name, value & source type and path on boot. It looks like https://gist.github.com/jdwyah/5ca4b0eacdc04414540af0761b8a0a14 https://gist.github.com/jdwyah/5ca4b0eacdc04414540af0761b8a0... I think it's allright, but I'm still thinking about how to make it better. More details in: https://docs.prefab.cloud/docs/explanations/bootstrapping https://docs.prefab.cloud/docs/explanations/bootstrapping
  • jdwyah 3y ago
    And if you are interested in making all of your configuration typesafe, please get in touch. I have things I want to show you.
• the__alchemist 3y ago
  This article (And most notably its title) uses a narrow view of programs, how they're interacted with, and storage models. Should your car's ECUs do this? Your washing machine? A GUI program? What should they output if storage is the last page of flash memory? What if it uses 2 pages? If it's offboard, should it report the device ID? If it's a GUI, should it have a button that displays this? Would an import/export feature meet the intent?
  • cafeinux 3y ago
    I find the author's intent and meaning to be rather straightforward to understand: he's a Unix sysadmin, so I think it's understandable that he's speaking about programs executed on a Unix-like system that a sysadmin is likely to administrate. So no, nor a washing machine nor a car ECU is expected to do this (not until sysadmins are paid to administrate them at least). As for technicians that might need to configure one someday, those people are usually expected to already know where to look, or at least to be able to read their documentation correctly. And anyway, car ECUs and washing machine usually provide a UI for configuration and don't rely on config files as a primary mode of configuration. As for GUI programs, if they use a human readable config file under the hood, or if they accept a config file as input (some let the user change some simple settings through their UI but uses a config file to change more advanced settings), yes they should have a way to show where it is (either through a command, or through the settings UI for example). I don't really know what you are talking about with the pages of a flash memory... Since in Unix, everything is a file, it will be read through a file anyway, not directly through pages. Same thing for offboard: just display the path to the file, wherever it might physically be. And if you really are in the most rare and hypothetical scenario where your program reads its config directly from the pages of a flash memory, just have it display "The config is read from the last [two] page[s] of <this flash memory> when invoqued with --help.
    • the__alchemist 3y ago
      The article makes sense in this context, and after a reread, the hint is identification of background in the first sentence. I misunderstood due to the title of "Everything that uses configuration files should report where they're located",and lines "if a program uses a configuration file (or several), it should have an obvious command line way to find out where it expects to find that configuration file.", and use of "program" in general. Specifically, "program" in this case refers to "CLI program that runs on Linux, used by the IT field".
• twunde 3y ago
  This brings to mind php's phpinfo. While often left publicly available, it provided (provides?) information on all loaded configurations.
• deleted 3y ago
  [deleted]
• radarsat1 3y ago
  It should be easier to track file opening history of any running program. There is strace, but grepping/parsing the output for this purpose gets very messy.
• clvx 3y ago
  If you are in k8s, I freaking love FluxCD for this reason. It points where your sources are.
• dumpster_fire 3y ago
  Wait till you get to template configs that automatically generate fan out config files during build time. You can report it, but you can't find it. Really enhances the debugging experience.
• opentokix 3y ago
  Tell me again that you run a terrible system and don't know what you are doing like 99.9% of the IT-"professionals" out there.
• m0rissette 3y ago
  Strace, filter in open calls? Lsof, find files open on running proc? Problem solved?
• WirelessGigabit 3y ago
  And... one of my pet peeves with configurations is that people delete everything that is commented. Like helm chart definitions. Now there is an upgrade of the chart, and I need to go and trace down which value changed. Well, if we would've kept the commented stuff I could overlay it with the new defaults showing me where the defaults have changed. At least that gives me some clue.
  • remram 3y ago
    I personally delete everything that I don't change. This reduces the amount of breaking changes that I have to deal with. If I'm only setting `persistence.storageClass` and `ingress.host`, my values.yaml will be two lines, not the original 2,000.
• dclowd9901 3y ago
  Maybe tangential to this, I’ve been thinking about the idea of “conceptually linked items” in a system and how to ensure things don’t become desynced. Static type systems help keep linked objects synced across code. But what about documentation? What about configuration code? It quickly falls out of sync. I was thinking a possible naive solution to this in a file system would be some sort of pragma you could put on a file: other files it should be synced with and a date of last change. If you update a file, any file it’s supposed to be synced with must be updated, if only at the very least, the date of last change. It’s sort of a forcing function to make sure everything’s up to date that needs to be. Could be onerous in a large system but I’m not sure.
• siliconc0w 3y ago
  "just make everything a protobuf" (or whichever system you prefer). The config file is essentially an API and APIs really benefit when they're typed. You can make a actual API to expose application configuration knobs too which is nice but you should tie this into your rollout/CICD system so you're just as careful pushing new configuration values as you are when pushing new binaries (multiple envs, slow rollouts, blue/green etc).
• baby 3y ago
  I think good programming languages enforce that. I really like that in Golang you import a library, and them have to qualify any calls with that library name. For example, id you imported library foo, calls will look like foo.Bar and foo.Gu. I think that’s one thing that Rust didn’t steal from Golang when they should have
• leecommamichael 3y ago
  This has been the default (pun intended) on Apple OSes for a long time. At the end of the day, the developer has the ability to create/destroy files at will.
• foolfoolz 3y ago
  i worked at a big company ($10B+) that was heavy on scala and akka. we had tons of bugs across almost every team trying to figure out which application.conf was setting something or getting overwritten from somewhere else. we had many unnecessary configs to force something we couldn’t figure out. we had maybe 2 people who actually understood it and could debug it. what a horrible system. no one ever walked away from akka and said i enjoy the decisions they made to configure it
  • asimpletune 3y ago
    Haha I wonder if we worked at the same company. Anyways, I also remember that a lot of those same teams never read the documentation that explained how any of that stuff worked. In general, most of Scala stuff was made out to be super hard and complicated by other people, when really they just needed someone to explain to them the basics in simple terms what was happening. I don't get to write Scala anymore and now I really miss it for certain things.
    • foolfoolz 3y ago
      i wrote scala at work for like 12 years straight. it’s a great language and i really do miss it at times. scala code always achieved its goal of being less code. but there were some things like this where scala made it more difficult. and i would like to chalk this up to “someone just needed to learn the basics”, but why was everyone having this issue? it’s a bad interface
• zapw 3y ago
  For git config, the --show-origin flag is really useful for this, e.g. git config --show-origin --list
• maxk42 3y ago
  Perhaps we could implement some sort of standard switches, such as --locate-conf for reporting which configuration file(s) are being read (and in which order) or --show-conf to produce a comprehensive list of set configurations.
• swayvil 3y ago
  A daemon that watches everything that reads and writes. Reports everything that looks like a config file, what accessed it, where it's located ... That might be useful.
• gavinray 3y ago
  It ought to be "$XDG_CONFIG_HOME/<program-name>" I have trouble seeing the rationale for other locations when the XDG standard exists
• elihu 3y ago
  I would go one further and say that the default should be that programs cannot read, open, or see files at all unless they're either explicitly allow-listed to open those files, or the user has given permission for them to open those files (i.e. by using the file name as a command-line argument, or selecting the file in an os-provided file selection dialog box).
  • jjtheblunt 3y ago
    Is that what macOS does? (I'm thinking it's coarser granularity, like asks if a program can see a filesystem tree, for a few such checked trees.)
    • elihu 3y ago
      I don't know; I don't use a Mac.
    • notatoad 3y ago
      i think that's only mac store apps.
  • theobeers 3y ago
    I like this about Deno (not to suggest that the runtime is without its flaws). Whenever I try to run a script and it tells me I need to allow e.g. read access to the filesystem, I’m more grateful than annoyed. And it lets you choose the granularity of the permissions that you grant. Hopefully this approach will become more common. https://deno.com/manual@v1.34.3/basics/permissions https://deno.com/manual@v1.34.3/basics/permissions
  • jml7c5 3y ago
    I see this as one of the big unintended benefits of AppArmor and SELinux: it's easy to filter for the relevant subset of files.
• gillesjacobs 3y ago
  Anaconda being by far the worst offender I have encountered. It places a ton of configuration spread out over your home, .condarc, and your shell profile dotfiles, as well as further configuration in the install folder and the of course the project environment.yml file. Good luck finding out the channel precedence.
• thayne 3y ago
  Perhaps there should be an semi-standard flag similar to --help and --version for showing help specifically about configuration and environment variables. --cfg-help maybe?
• digitalsanctum 3y ago
  As I read this I wonder if SBOM (Software Bill of Materials)[1] could be used? I'm not sure if it already has configuration files as part of it's definition but doesn't seem like a stretch to add? Btw, I've never used SBOM in practice but seems like something worthwhile implementing. [1] https://www.cisa.gov/sbom#:~:text=A%20%E2%80%9Csoftware%20bill%20of%20materials,that%20make%20up%20software%20components https://www.cisa.gov/sbom#:~:text=A%20%E2%80%9Csoftware%20bi....
• alentred 3y ago
  Agree! Also any other locations where any state that impacts the program behavior is stored: cache, temporary files, etc. Ultimately I want to be able to debug things or do a sort of a "factory reset". Or maybe I need to move to NixOS.
• parentheses 3y ago
  This is a great idea. I feel that the best total situation is 2 things: 1. A way to print the paths to all files visible that will be read for configuration. This means that if the user config file means the system config file is ignored, print only the user config file path. Alternatively print that the system file will not be read but show its path anyways. There's many ways to solve this and it depends on the way the app loads configs. 2. A way to print the final configuration after it's been loaded and validated. This should show what settings override others. For example if the user misunderstands the config merging or erroneously quotes a numeric value causing validation to fail, this would be a great way to triangulate that. I've seen quite a few apps implement #2 but rarely do they have #1.
• jmbwell 3y ago
  Further: programs should use well-defined libraries to manage config and data file location. As easy as it is to roll your own, a library can and should handle cases you don’t need to worry about. Python has platformdirs for this. It handles XDG and a number of corner cases. https://pypi.org/project/platformdirs/ https://pypi.org/project/platformdirs/ Ideally the OS you deploy to should handle this as well.
  • hooverd 3y ago
    Thanks! You find a new Python lib every day.
• samlittlewood 3y ago
  My goto for this is ‘strace’
• irrational 3y ago
  Log files too.
• firefoxd 3y ago
  phpinfo(); for all other languages.
• deleted 3y ago
  [deleted]
• fsckboy 3y ago
  The operating system should tell everything that uses configuration files where to put them, and what formats are acceptable. The operating system should tell systemd where to put them. The operating system does not use JSON. Users can then ask the operating system.
• deleted 3y ago
  [deleted]
• jmholla 3y ago
  This also applies to code generation. At my first job out of college, it was difficult to piece together where a particular bit auto-created code came from. At my second job, I pushed everyone to ensure that code generated files pointed to their sources (configuration files and generating code). Ultimately, one of our engineers designed it into our codegen framework so well, it basically came for free for our developers. And boy was it so nice to not wonder where the knobs were that changed your output. On a side note, we also checked in the codegened files and that was great for other reasons like reviewability and searchability.
• dmckeon 3y ago
  Programs that use configuration files should provide ways to: show which sources of config data were used: files, CLI, env vars, etc. with --verbose, also show which potential sources were not used show any conflicts or overrides from config sources show config settings differing from program defaults in the same format as their source, or in an appropriate format for a config file For the last item, it should be possible to capture the output showing non-default settings and append it to a config file, without any editing or processing, to reproduce the program configuration state exactly.
• seiferteric 3y ago
  I've mentioned this before in a comment long ago that I thought there should be a library for configurations, like libconfig. This would expose an API that a program could get it's config from, then the user could decide in which format they would like to keep their configs, like ini, json, yaml etc and could translate between them. Then you could get like a unified config for your system.
  • donatj 3y ago
    Lossless translation between config types is troublesome and rarely possible cleanly. Varying capabilities of different config file types is part of the reason for the variety that exists.
• peter_d_sherman 3y ago
  https://unix.stackexchange.com/questions/58887/how-do-i-monitor-opened-files-of-a-process-in-realtime https://unix.stackexchange.com/questions/58887/how-do-i-moni... https://stackoverflow.com/questions/27428150/linux-how-to-track-all-files-accessed-by-a-process https://stackoverflow.com/questions/27428150/linux-how-to-tr... https://superuser.com/questions/348738/continuously-monitor-files-opened-accessed-by-a-process https://superuser.com/questions/348738/continuously-monitor-... https://serverfault.com/questions/138297/how-to-track-all-files-a-process-opens-in-its-lifetime https://serverfault.com/questions/138297/how-to-track-all-fi...
• Footnote7341 3y ago
  I'm sick of programs dumping config files in random places in my home directory instead of .local, and then trying to read other system configuration files from .home even though I've gone through the proper steps to move them to .local and labeled the fact that I've done that with environment variables more and more im feeling software deserves to be dumped into a nix style sandbox if it refuses to behave properly. I've had to recompile shit from source just to get it to respect my environment variables regarding where to look for local configs.
• someonewhocar3s 3y ago
  [dead]
• markstos 3y ago
  A hack I use: run `strace someprog` and look for the system calls that open files. This works no matter how badly the thing is documented.
• spieglt 3y ago
  https://github.com/spieglt/whatfiles https://github.com/spieglt/whatfiles may be useful to find such files
• ayakang31415 3y ago
  I am curious, as UNIX user, why there is no global configuration directory in home path where all the configuration files are stored. I hate that my home directory is cluttered with lots of .file and .dir.
• avereveard 3y ago
  Amen, plus all the folders in the search path, so sweet know where to properly override defaults.
• londons_explore 3y ago
  I find "strace programname 2>&1 | grep open" is a good way to find out which files are being opened for config. It's quicker than digging out the man page.
• ACV001 3y ago
  just stick to a convention (e.g. Spring boot properties file conventions - https://docs.spring.io/spring-boot/docs/1.0.1.RELEASE/reference/html/boot-features-external-config.html https://docs.spring.io/spring-boot/docs/1.0.1.RELEASE/refere...).
• deleted 3y ago
  [deleted]
• TZubiri 3y ago
  Ptrace -e open
• zzbzq 3y ago
  Programs should just store all their files together. The traditional directory structure of Linux et al is wrong. I do not need a /bin folder full of unrelated programs. I need a /program_xyz folder for each program, with all the stuff in there. Granted when disk space was scarce, this reduces the ability to share dynamically linked dependencies. There are other ways around that. And in the modern world, there are some retrofitted solutions where package managers indeed do it this way. And a lot of other tools out there like Docker overlap this idea that part of what they do is disentangle the file system of unrelated programs. But the real giga-brained answer, which most programmer-type people probably aren't emotionally ready to accept, is that not only are the ways we organize our directories wrong, but tree-based directories are just dumb in general. Everything should just go in one big directory and you can use names and tags to sort them out. Programmers are culturally obsessed with trees.
  • amphitheatre 3y ago
    I would posit most things in a file system are better represented by graphs. A file type would have a program to run with, a program to edit with, maybe also referencing files in relatively placed directories/locations, and potentially pointed to by a crown job or other automation routines… Trees and hierarchical layouts do make sense for some things, but as soon as you start symlinking, a proper graph works better - though I imagine computationally harder (a sacrifice I’m willing to make on most desktop machines tbh). Your notion of sticking everything in a single space and using tags, searches, and filters is something I’ve wanted to explore for a while. Afaik it’s unexplored space, unless others could point me to some awesome tools or processes?
    • zvr 3y ago
      The UNIX filesystem has traditionally been a graph for ever. I haven't looked at details for a couple of decades, but definitely all UNIX/POSIX/Linux filesystems operate on a graph model. A distinction I used to make when I was teaching this stuff: on your filesystem tree, on Unix names (labels) are on the links (arrows), while on DOS/Windows names are on nodes (boxes). If you want to explore a tag-based system, take a look at https://www.tagspaces.org/ https://www.tagspaces.org/
• tqwhite 3y ago
  YEs. YEs. YEs. I always do and condemn everyone who doesn't. Also, it's nice to have a -showConfig flag that exposes the results of config file processing, minus secrets of course.
• jrochkind1 3y ago
  How many of us just upvoted on title alone? once you see it, it's just self-evident. Which doesn't mean we remember to do it when we are implementing things that use config files.
• captaincrisp 3y ago
  Just use strace to find out what files it's opening! /s
• activiation 3y ago
  ~/.config/appname should be the default
• coxley 3y ago
  No one can hide from strace!
• tapvt 3y ago
  Honestly, I’m probably going to update my own (mostly node) apps to dump such information to `stdout` when they’re first fired up, at the very least. It would have saved me time today when re-visiting and extending a service that I’ve not touched in two years.
• sacnoradhq 3y ago
  Consider writing a FUSE filesystem driver that generates configuration files on the fly in response to some secure source of truth. It skips one of the middlemen in terms of configuration management and makes programmatic reconfiguration more accessible.