6 ms·

I started my career in a world where we did everything using shell scripts running directly on bare metal servers, usually running Solaris, and later SuSe or Re

by hliyan 2y ago

I started my career in a world where we did everything using shell scripts running directly on bare metal servers, usually running Solaris, and later SuSe or RedHat. I never understood the "how would you reproduce your setup without Docker (or X, where X is some other technology)". The scripts were deterministic. The dependency versions were locked. The configurations were identical. The input arguments were identical. The order of execution was identical. It all ran on a deterministic computational device. How could it not be reproducible?

ookblah 2y ago
reproducibility isn't just on your deployments, it's for development too. got old REAL fast when your fancy build doesn't work the same on every devs device or some one off issue with how your dev has setup their environment steals hours from everyone. it was a big reason why we moved to containers at the bare minimum, because its quick and easy to spin up and destroy and you are guaranteed what runs locally runs on prod. no more "well it worked on my system!".
- ghomem 2y ago
  >reproducibility isn't just on your deployments, it's for development too Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it. You can achieve this with Docker as well, if the arrangement is not too complex. The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.
ghomem 2y ago
Well that's exactly the point! Creating complex cloud resources with, for instance, Terraform, is less reproducible than a shell script on an LTS system like Ubuntu or RHEL - that's because the cloud provider interfaces drifts and from time to time stops accepting the terraform manifests that previously worked. And to fix it, you have to interrupt your normal work for yet another unplanned intervention in the terraform code - this happened to my teams several times. This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken. I tried to explain this topic in the article linked above. Not sure how far I succeeded.
- kbolino 2y ago
  Leaning into LTS is nice until you near EOL and have to migrate everything in an often Herculean effort to work with the next LTS release.
  - deleted 2y ago
    [deleted]
  - ghomem 2y ago
    Like 12 years of life cycle is not enough for you to plan a transition? You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.
    - kbolino 2y ago
      I don't really think we're in disagreement here. The longer you wait, the harder the transition will be. LTS is a good foundation, and usually the right choice for "enterprise" or "business" settings, but you should not rely overmuch on any one LTS release's way of doing things, when the wider Linux ecosystem moves much faster.
      - ghomem 2y ago
        The longer you wait the harder the pain. The less you wait the more frequent the pain. So it depends on the function that converts intensity and frequency to suffering :p But, most importantly, the fact that LTS gives you a choice is what I was highlighting. For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.
  - minkles 2y ago
    It's not terrible in my experience of doing it several times now. It is definitely less terrible than trying to unfuck tangles of terraform / terragrunt / yaml / bits of cloud infra.
altdataseller 2y ago
Wouldnt there be slight differences in different Unix flavors so that the script couldnt run in all of them? If it only worked on Solaris, what would happen if Solaris retired? (Like what happened to Centos)
- pxc 2y ago
  You will likely have to adapt your scripts for OS-specific or installation-specific tasks like package management and modifying filesystems. In the past I've used Nix (either via `nix run` and `nix shell` or templating in Nixpkgs' `writeScript` or similar) for this stuff to guarantee that I'm always running the same tools regardless of what's installed on the base system. This can free you up to use a different shell, rely on recent features of Bash, use GNUisms in coreutils, sed, grep, find, etc., fix a specific version of jq, use external templating tools, etc. For systemd-based distros, you can even use Nix to manually install system-wide services: just install a package to the default or system profile, and then symlink the included unit files from the profile (not the direct store path) into /etc/systemd. `systemctl daemon-reload` and you can manipulate them in all the usual ways one would with systemd. Other Unix distros don't have first-class support with Nix so you may need to take some additional care when working out your script (especially the part of it that installs Nix), but if you don't need to set up services this way you can write portable scripts with few limitations that will work across all Linux distros, macOS, probably FreeBSD and maybe NetBSD. I've never been so lucky as to work at a place that used any Unix flavors other than Linux and macOS, though.
- stcroixx 2y ago
  That's what POSIX was for. Keep your scripts and system calls POSIX compliant and you could move from something like AIX to Linux easily.
  - kbolino 2y ago
    POSIX never specified things like disk partitioning or package management, so this still requires something else to give you a working system in the first place.
- 2y ago
eastbound 2y ago
You said it: Your versions were locked. Therefore it is not constantly up-to-date. I was pinched myself: Security. - With the cloud threats, everything needs to be constantly up-to-date. Docker images make it easier than permanent servers that need to be upgraded. We used to upgrade every week, now we’re upgraded by default. So yes, sometimes our images don’t start with the latest version of xyz. But this is rare, downgrade is easy with Docker, and reproduction on a dev engine easier. - With the cloud threats, everything needs to be isolated. Docker makes it easy to have an Alpine with no other executable than strictly necessary, and only open ports to the required services. I hate the cloud because 4GB/2CPU should be way enough to run extremely large workloads, but I had to admit that convenience made me switch.
- j45 2y ago
  What needs to be constant and up to date is reviewing the new patches and which ones can be released and not locked. The versions that are not locked can be a test or dev environment that constantly updates and checks for errors. Security threats are a thing, how we do and don't use technologies as well which ones can also factor in to how much is exposed.
- Spivak 2y ago
  A container is locking the whole OS, on this axis it's not an improvement either direction. You still need a way to update deps.
- hliyan 2y ago
  We did upgrades periodically, each time a conscious choice after reviewing the release notes of the dependency. Occasionally a script would need to be updated, but that was it.
consteval 2y ago
To be fair there's real issues with this approach, too. For example, shell scripts aren't actually very portable. GNU awk vs nawk vs... multiply that by all your tools, and yeah those scripts don't run deterministically (they rely too much on the environment). This alone was a big reason why systemd exists today. But there's a middle ground here too. To me there's a HUGE gap between Kubernetes distributed systems and shell script free for all.

by hliyan 2y ago

ookblah 2y ago

reproducibility isn't just on your deployments, it's for development too. got old REAL fast when your fancy build doesn't work the same on every devs device or some one off issue with how your dev has setup their environment steals hours from everyone. it was a big reason why we moved to containers at the bare minimum, because its quick and easy to spin up and destroy and you are guaranteed what runs locally runs on prod. no more "well it worked on my system!".

ghomem 2y ago
>reproducibility isn't just on your deployments, it's for development too Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it. You can achieve this with Docker as well, if the arrangement is not too complex. The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.

ghomem 2y ago

Well that's exactly the point! Creating complex cloud resources with, for instance, Terraform, is less reproducible than a shell script on an LTS system like Ubuntu or RHEL - that's because the cloud provider interfaces drifts and from time to time stops accepting the terraform manifests that previously worked. And to fix it, you have to interrupt your normal work for yet another unplanned intervention in the terraform code - this happened to my teams several times. This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken. I tried to explain this topic in the article linked above. Not sure how far I succeeded.

kbolino 2y ago
Leaning into LTS is nice until you near EOL and have to migrate everything in an often Herculean effort to work with the next LTS release.
- deleted 2y ago
  [deleted]
- ghomem 2y ago
  Like 12 years of life cycle is not enough for you to plan a transition? You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.
  - kbolino 2y ago
    I don't really think we're in disagreement here. The longer you wait, the harder the transition will be. LTS is a good foundation, and usually the right choice for "enterprise" or "business" settings, but you should not rely overmuch on any one LTS release's way of doing things, when the wider Linux ecosystem moves much faster.
    - ghomem 2y ago
      The longer you wait the harder the pain. The less you wait the more frequent the pain. So it depends on the function that converts intensity and frequency to suffering :p But, most importantly, the fact that LTS gives you a choice is what I was highlighting. For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.
- minkles 2y ago
  It's not terrible in my experience of doing it several times now. It is definitely less terrible than trying to unfuck tangles of terraform / terragrunt / yaml / bits of cloud infra.

altdataseller 2y ago

Wouldnt there be slight differences in different Unix flavors so that the script couldnt run in all of them? If it only worked on Solaris, what would happen if Solaris retired? (Like what happened to Centos)

pxc 2y ago
You will likely have to adapt your scripts for OS-specific or installation-specific tasks like package management and modifying filesystems. In the past I've used Nix (either via `nix run` and `nix shell` or templating in Nixpkgs' `writeScript` or similar) for this stuff to guarantee that I'm always running the same tools regardless of what's installed on the base system. This can free you up to use a different shell, rely on recent features of Bash, use GNUisms in coreutils, sed, grep, find, etc., fix a specific version of jq, use external templating tools, etc. For systemd-based distros, you can even use Nix to manually install system-wide services: just install a package to the default or system profile, and then symlink the included unit files from the profile (not the direct store path) into /etc/systemd. `systemctl daemon-reload` and you can manipulate them in all the usual ways one would with systemd. Other Unix distros don't have first-class support with Nix so you may need to take some additional care when working out your script (especially the part of it that installs Nix), but if you don't need to set up services this way you can write portable scripts with few limitations that will work across all Linux distros, macOS, probably FreeBSD and maybe NetBSD. I've never been so lucky as to work at a place that used any Unix flavors other than Linux and macOS, though.
stcroixx 2y ago
That's what POSIX was for. Keep your scripts and system calls POSIX compliant and you could move from something like AIX to Linux easily.
- kbolino 2y ago
  POSIX never specified things like disk partitioning or package management, so this still requires something else to give you a working system in the first place.
2y ago

eastbound 2y ago

You said it: Your versions were locked. Therefore it is not constantly up-to-date. I was pinched myself: Security. - With the cloud threats, everything needs to be constantly up-to-date. Docker images make it easier than permanent servers that need to be upgraded. We used to upgrade every week, now we’re upgraded by default. So yes, sometimes our images don’t start with the latest version of xyz. But this is rare, downgrade is easy with Docker, and reproduction on a dev engine easier. - With the cloud threats, everything needs to be isolated. Docker makes it easy to have an Alpine with no other executable than strictly necessary, and only open ports to the required services. I hate the cloud because 4GB/2CPU should be way enough to run extremely large workloads, but I had to admit that convenience made me switch.

j45 2y ago
What needs to be constant and up to date is reviewing the new patches and which ones can be released and not locked. The versions that are not locked can be a test or dev environment that constantly updates and checks for errors. Security threats are a thing, how we do and don't use technologies as well which ones can also factor in to how much is exposed.
Spivak 2y ago
A container is locking the whole OS, on this axis it's not an improvement either direction. You still need a way to update deps.
hliyan 2y ago
We did upgrades periodically, each time a conscious choice after reviewing the release notes of the dependency. Occasionally a script would need to be updated, but that was it.

consteval 2y ago

To be fair there's real issues with this approach, too. For example, shell scripts aren't actually very portable. GNU awk vs nawk vs... multiply that by all your tools, and yeah those scripts don't run deterministically (they rely too much on the environment). This alone was a big reason why systemd exists today. But there's a middle ground here too. To me there's a HUGE gap between Kubernetes distributed systems and shell script free for all.