4 ms·

I have found that managing dotfiles is not enough. The dotfiles serve no purpose without the software that uses them. My method is to write small scripts, I cal

by mongol 6y ago

I have found that managing dotfiles is not enough. The dotfiles serve no purpose without the software that uses them. My method is to write small scripts, I call them setuplets, that install the software and then symlink the dotfile to its master that I manage in git. In the simplest case, it is just a two line script in a directory, but I have one for each program, and a tool to select which I run when setting up a new linux machine. The more complex ones may install a number of tools and set up environment variables etc. For environment variables, the script drops a script file in a directory, where my bashrc sources it.

lambda_obrien 6y ago
I've been logging each install as a markdown document with directions, and manually symlinking my dotfiles, but tiny scripts that do both are a great idea!
- mongol 6y ago
  Yes, as soon as I have installed a program which I like and think is a "keeper", I write my setuplet script, usually by copying and pasting from the bash history.
  - iib 6y ago
    I am not an expert in Ansible, or any other software of the kind, but I think this is one of the easier use-cases of such a program. I wrote an ansible script to compile emacs from source, after some initial configuration, and it serves me pretty well.
    - mongol 6y ago
      I started with Ansible for automation, and still use it, but have abandoned it for setting up my personal environment. I still use it to set up services, such as sshd, webserver, printer, samba, smartd etc. I found it was more cumbersome for personal environment setup than my script method. My method of re-initializing my environment rests on three pillars: Ansible, for server/OS setup. Things run by init/systemd, involving anything outside of $HOME basically Backup, using borgmatic. I use this to restore most things in $HOME, except random dotfiles. My documents, my checked out git repos, etc. Script "setuplets". These I run on demand, to set up my environment piece by piece. Perhaps I do not want to restore my programming environment just because I want to have my custom prompt on a host, for example. Finding the balance between these have been difficult. What bootstraps what, and especially, how to handle credentials. My backup is encrypted, but how would I make sure I had the keys to restore it? I could restore it from my pass password store, but how would I get the gnupg keys in place first? I have not solved this completely satisfactorily yet
      - iib 6y ago
        I am you in the past then, I am still trying to find the balance. Thank you for the insights!
- mattalbie 6y ago
  I do this but include the symlink command in the document. That way all the links become one copy and paste.
  - smichel17 6y ago
    May I recommend Makefiles? Nothing fancy, just a bunch of .PHONY targets. It's a converient way to bundle a bunch of scripts (or single arcane commands) into one file, with autocompletion.
    - mattalbie 6y ago
      That’s a good idea. Thanks!
Kototama 6y ago
Interesting! https://github.com/nix-community/home-manager https://github.com/nix-community/home-manager uses Nix to solve this problem. It adds reproducibility and rollbacks but with the cost of learning the Nix DSL.
- jpetrucc 6y ago
  +1 for nix and home-manager - I recently switched to this setup (still learning/experimenting a lot, link here [0]) and have been very happy! [0]: https://github.com/jpetrucciani/nix https://github.com/jpetrucciani/nix
  - kohlerm 6y ago
    +1 for nix and home-manager. Most other solutions make it hard to manage installing packages on different flavors of Linux. You can get even more fancy by using the direnv integration to have specific versions of binaries available when entering a directory. See https://www.mathiaspolligkeit.de/dev/exploring-nix-on-macos/ https://www.mathiaspolligkeit.de/dev/exploring-nix-on-macos/ . By checking in the nix files into your projects directory, you get really reproducible projects, because all you have to do is to checkout the project then run direnv allow and nix/home-manager will install all tools needed to run/build that specific projects. Different projects can use different versions of tools.
codethief 6y ago
> My method is to write small scripts, I call them setuplets, Finally I have a proper term for what I, too, have been doing all these years! :-) It's indeed the best-possible approach I've found, though there are a number of things that I haven't yet solved for myself in a satisfactory manner: - With shell scripts there are no idempotency guarantees and there is no easy undoing / uninstalling / clean-up, especially after updating a setuplet. - With shell scripts everything is defined imperatively as opposed to declaratively. In particular, setuplets usually operate on the filesystem directly and testing and dry runs are almost impossible. - No status report as to what a setuplet wants to set up (software, configs, cronjobs…) and what is already set up on the current machine. That is, no diffs. This makes sharing setuplets and configs between multiple machines (say, personal and work laptop) rather cumbersome. For instance, I might forget to re-execute a setuplet on the second machine which could then lead to a missing software dependency or a mismatch between config and software. - No simple, out-of-the-box way to have different configs/dotfiles for different machines, in particular: no config templating. - It's hard to share common settings across applications without duplicating them everywhere. For instance, I would like to define a common set of colors / a common theme for my window manager, my terminal, my editor and so on. Similarly, (some) keybindings should be the same across applications. Moreover, I have a set of common directories in my home dir (for binaries, logs, cache etc.) that all my setuplets & dotfiles should use. - Dependencies and interactions between setuplets are often implicit. They interact with and depend on one another through a myriad of ways, like software dependencies (of course) but also PATH modifications, cronjobs, bash aliases, file system modifications … These are very hard to recognize and, even worse, to refactor. - Bash scripts are error-prone and cumbersome to write and debug (and refactor). - My setuplets don't have a common command line interface and their relation to one another is unclear. (In which order should they get executed?) I tend to write scripts that invoke all the setuplets in the right order but it still seems messy and error-prone. I've tried solutions like Ansible but I've found that its purely declarative DSL is not flexible enough to cover all my use cases in an elegant manner. …which is why I'm currently working on a small Python library that will hopefully solve or at least ease the above pain points for me. Once the library is finished, I will rewrite my setuplets in Python (using the library to do the hard and tedious work), so that I end up with one single Python project of dotfiles and setuplets (exposed through one single command line tool) that, once executed, will automatically set up an entire machine for me within a few minutes. One nice thing would be that all inter-setuplet dependencies would be expressed through Python code (with proper typing, encapsulation in modules and everything) which could then easily be explored (and also refactored) with an IDE. Sure, this sounds like a lot of work but given that I intend to use my dotfiles for a couple more decades, it seems well worth it. Of course, whether my approach will ultimately be able to solve all the challenges above remains to be seen but I'm at a point right now where I'm convinced that proper software engineering methods (especially dependency injection, type checks, tests etc.) would be a real boon for managing my (hundreds of) dotfiles.
rakoo 6y ago
I don't want to sound condescendant, but are you not basically doing packages ? Instead of standard packages your distro gives you, you have "custom" packages that also contain more specific installation instructions (write file with specific content, set environment variables, ...). I know that PKGBUILDs in Arch allow you to do such things if needed. Bonus: now even configuration files can be linked to the software that needs it; when you remove a software you also remove all the files related to it and there are no leftovers.
- maple3142 6y ago
  Because package manager are different across different distro? Also, custom setup allows you to have those tools even if you don't have root permission to install them with package manager.
- mongol 6y ago
  No. My script calls the package manager, sudo zypper install..., and symlinks a dotfile. That is usually all.
maple3142 6y ago
I have a similar setup, but I don't have to write that script myself. Instead, I uses Zinit (https://github.com/zdharma/zinit https://github.com/zdharma/zinit) to let it setup the tools for me.
siraben 6y ago
I used to do this for macOS and Linux but it was a fragile approach, since using Nix and NixOS I've had great success using home-manager[0] which takes care of what software is installed and any overrides or patches I desire. I also use GNU stow in my dotfiles[1] especially so that my setup still works on systems without Nix installed. [0] https://github.com/nix-community/home-manager https://github.com/nix-community/home-manager [1] https://github.com/siraben/dotfiles https://github.com/siraben/dotfiles

by mongol 6y ago

lambda_obrien 6y ago

I've been logging each install as a markdown document with directions, and manually symlinking my dotfiles, but tiny scripts that do both are a great idea!

mongol 6y ago
Yes, as soon as I have installed a program which I like and think is a "keeper", I write my setuplet script, usually by copying and pasting from the bash history.
- iib 6y ago
  I am not an expert in Ansible, or any other software of the kind, but I think this is one of the easier use-cases of such a program. I wrote an ansible script to compile emacs from source, after some initial configuration, and it serves me pretty well.
  - mongol 6y ago
    I started with Ansible for automation, and still use it, but have abandoned it for setting up my personal environment. I still use it to set up services, such as sshd, webserver, printer, samba, smartd etc. I found it was more cumbersome for personal environment setup than my script method. My method of re-initializing my environment rests on three pillars: Ansible, for server/OS setup. Things run by init/systemd, involving anything outside of $HOME basically Backup, using borgmatic. I use this to restore most things in $HOME, except random dotfiles. My documents, my checked out git repos, etc. Script "setuplets". These I run on demand, to set up my environment piece by piece. Perhaps I do not want to restore my programming environment just because I want to have my custom prompt on a host, for example. Finding the balance between these have been difficult. What bootstraps what, and especially, how to handle credentials. My backup is encrypted, but how would I make sure I had the keys to restore it? I could restore it from my pass password store, but how would I get the gnupg keys in place first? I have not solved this completely satisfactorily yet
    - iib 6y ago
      I am you in the past then, I am still trying to find the balance. Thank you for the insights!
mattalbie 6y ago
I do this but include the symlink command in the document. That way all the links become one copy and paste.
- smichel17 6y ago
  May I recommend Makefiles? Nothing fancy, just a bunch of .PHONY targets. It's a converient way to bundle a bunch of scripts (or single arcane commands) into one file, with autocompletion.
  - mattalbie 6y ago
    That’s a good idea. Thanks!

Kototama 6y ago

Interesting! https://github.com/nix-community/home-manager https://github.com/nix-community/home-manager uses Nix to solve this problem. It adds reproducibility and rollbacks but with the cost of learning the Nix DSL.

jpetrucc 6y ago
+1 for nix and home-manager - I recently switched to this setup (still learning/experimenting a lot, link here [0]) and have been very happy! [0]: https://github.com/jpetrucciani/nix https://github.com/jpetrucciani/nix
- kohlerm 6y ago
  +1 for nix and home-manager. Most other solutions make it hard to manage installing packages on different flavors of Linux. You can get even more fancy by using the direnv integration to have specific versions of binaries available when entering a directory. See https://www.mathiaspolligkeit.de/dev/exploring-nix-on-macos/ https://www.mathiaspolligkeit.de/dev/exploring-nix-on-macos/ . By checking in the nix files into your projects directory, you get really reproducible projects, because all you have to do is to checkout the project then run direnv allow and nix/home-manager will install all tools needed to run/build that specific projects. Different projects can use different versions of tools.

codethief 6y ago

> My method is to write small scripts, I call them setuplets, Finally I have a proper term for what I, too, have been doing all these years! :-) It's indeed the best-possible approach I've found, though there are a number of things that I haven't yet solved for myself in a satisfactory manner: - With shell scripts there are no idempotency guarantees and there is no easy undoing / uninstalling / clean-up, especially after updating a setuplet. - With shell scripts everything is defined imperatively as opposed to declaratively. In particular, setuplets usually operate on the filesystem directly and testing and dry runs are almost impossible. - No status report as to what a setuplet wants to set up (software, configs, cronjobs…) and what is already set up on the current machine. That is, no diffs. This makes sharing setuplets and configs between multiple machines (say, personal and work laptop) rather cumbersome. For instance, I might forget to re-execute a setuplet on the second machine which could then lead to a missing software dependency or a mismatch between config and software. - No simple, out-of-the-box way to have different configs/dotfiles for different machines, in particular: no config templating. - It's hard to share common settings across applications without duplicating them everywhere. For instance, I would like to define a common set of colors / a common theme for my window manager, my terminal, my editor and so on. Similarly, (some) keybindings should be the same across applications. Moreover, I have a set of common directories in my home dir (for binaries, logs, cache etc.) that all my setuplets & dotfiles should use. - Dependencies and interactions between setuplets are often implicit. They interact with and depend on one another through a myriad of ways, like software dependencies (of course) but also PATH modifications, cronjobs, bash aliases, file system modifications … These are very hard to recognize and, even worse, to refactor. - Bash scripts are error-prone and cumbersome to write and debug (and refactor). - My setuplets don't have a common command line interface and their relation to one another is unclear. (In which order should they get executed?) I tend to write scripts that invoke all the setuplets in the right order but it still seems messy and error-prone. I've tried solutions like Ansible but I've found that its purely declarative DSL is not flexible enough to cover all my use cases in an elegant manner. …which is why I'm currently working on a small Python library that will hopefully solve or at least ease the above pain points for me. Once the library is finished, I will rewrite my setuplets in Python (using the library to do the hard and tedious work), so that I end up with one single Python project of dotfiles and setuplets (exposed through one single command line tool) that, once executed, will automatically set up an entire machine for me within a few minutes. One nice thing would be that all inter-setuplet dependencies would be expressed through Python code (with proper typing, encapsulation in modules and everything) which could then easily be explored (and also refactored) with an IDE. Sure, this sounds like a lot of work but given that I intend to use my dotfiles for a couple more decades, it seems well worth it. Of course, whether my approach will ultimately be able to solve all the challenges above remains to be seen but I'm at a point right now where I'm convinced that proper software engineering methods (especially dependency injection, type checks, tests etc.) would be a real boon for managing my (hundreds of) dotfiles.

rakoo 6y ago

I don't want to sound condescendant, but are you not basically doing packages ? Instead of standard packages your distro gives you, you have "custom" packages that also contain more specific installation instructions (write file with specific content, set environment variables, ...). I know that PKGBUILDs in Arch allow you to do such things if needed. Bonus: now even configuration files can be linked to the software that needs it; when you remove a software you also remove all the files related to it and there are no leftovers.

maple3142 6y ago
Because package manager are different across different distro? Also, custom setup allows you to have those tools even if you don't have root permission to install them with package manager.
mongol 6y ago
No. My script calls the package manager, sudo zypper install..., and symlinks a dotfile. That is usually all.

maple3142 6y ago

I have a similar setup, but I don't have to write that script myself. Instead, I uses Zinit (https://github.com/zdharma/zinit https://github.com/zdharma/zinit) to let it setup the tools for me.

siraben 6y ago

I used to do this for macOS and Linux but it was a fragile approach, since using Nix and NixOS I've had great success using home-manager[0] which takes care of what software is installed and any overrides or patches I desire. I also use GNU stow in my dotfiles[1] especially so that my setup still works on systems without Nix installed. [0] https://github.com/nix-community/home-manager https://github.com/nix-community/home-manager [1] https://github.com/siraben/dotfiles https://github.com/siraben/dotfiles