6 ms·

If it's local-first then it's already subject to rot. Unless they're running it all in containers? In which case local doesn't matter. If you want to record a

If it's local-first then it's already subject to rot. Unless they're running it all in containers? In which case local doesn't matter. If you want to record a runbook, then record a runbook. You can do that a million ways. Text file, confluence doc, screen recording, shell script, etc. People already don't do that; they're not gonna suddenly start doing it more because your UI is fancier. Personally, I don't want to sit around all day writing code (or docs) to try to get the system to be like X state. I want to manually make it have X state, and then run a tool to dump the state, and later re-run the tool to create (or enforce) that state again. I do not want to write code to try to tell the computer how to get to that state. Nor do I want to write "declarative configuration", which is just more code with a different name. I want to do the thing manually, then snapshot it, then replay it. And I want this to work on any system, anywhere, without dependence on monitoring a Bash shell for commands or something. Just dump state and later reapply state.

milkshakes 2y ago
what happens when you want to tweak something you did in the middle of this process? do you have to go through the whole flow again manually to make a single change?
- 0xbadcafebee 2y ago
  I imagine you could either A) just modify the dumped state, B) paramaterize it, C) have the program split up the state into transactions and modify those. The program will probably have to take more than one step, in order, in order to accomplish everything. If it fails, you'd want it to try to undo it, ala transactions. And since it can do all that, it can stop, start, or resume at specific steps. Like, Terraform has always sucked because there was no way to dump existing resources as new code. So a team at Google made a tool to do it (Terraform-er). If Terraform had already had that feature, and if it didn't rely on having pre-existing state to manage resources, that would be like 90% of the way to what I'd want. Just dump resources as code, then let me re-run the code, and if I want I can modify the code to ask me for inputs or change things. (People think of Terraform as only working on Cloud resources, but you could (for example) make an Ubuntu Linux provider that just configures Ubuntu for you, if you wanted)
  - kiitos 2y ago
    Any notion of state that satisfies requirements like > Just dump state and later reapply state is necessarily declarative. > Just dump resources as code, What is the code for this resource? VM foo1 Memory 16GiB Network mynet1 It depends on the current state of the system where the resource is applied. If VM foo1 already exists, with 16GiB of memory, and connected to network mynet1, then the code is a no-op, no code at all. Right? Anything else would be a mistake. For example if the code would delete any matching VM and re-create it, that would be disastrous to continuity and availability, clearly a non-starter. Or, if VM foo1 exists, with 16GiB of memory, but connected to anothernet3, then the code should just change the network for that VM from anothernet3 to mynet1, and should definitely not destroy and re-create the VM entirely. And so on.
    - 0xbadcafebee 2y ago
      It depends what you're talking about; Terraform specifically has a flawed model where it assumes nothing in the world exists that it didn't create itself. Other configuration management tools don't assume that; they assume that you just want an item to exist; if it does exist, great, if it doesn't exist, you create it. But for a moment I'll assume you're talking about the other problem with configuration management tools, which is "which of the existing resources do I actually want to exist or modify?" That's a solved problem. Anything that you use on a computer that controls a resource, can uniquely identify said resource, through either a key or composite key. This has to be the case, otherwise you could create things that you could never find again :) (Even if you created an array of things with no name, since it exists as an item in a list, the list index is its unique identifier) Taking Terraform as example again, the provider has code in it that specifies what the unique identifier is, per-resource. It might be a single key (like 'id', 'ASN', 'Name', etc) or a composite key ( {'id' + 'VPC' + 'Region'} ). If the code you've dumped does not have the unique identifier for some reason, then the provider has to make a decision: either try to look up existing resources that match what you've provided and assume the closest one is the right one, or error out that the unique identifier is missing. Usually the unique identifier is not hard to look up in the first place (yours has a composite identifier: {VM:"foo1", Network:"mynet1"}). But it's also (usually) not fool-proof. Imagine a filesystem. You actually have two unique identifiers: the fully-qualified file path, and the inode number. The inode number is the actual unique identifier in the filesystem, but we don't tend to reference it, as 1) it's not that easy to remember/recognize an inode number, 2) it can be recycled for another file, 3) it'll change across filesystems. We instead reference the file path. But file paths are subtly complex: we have sym-links, hard-links and bind-mounts, so two different paths can actually lead to the same file, or different files! On top of that, you can remove the file and then create an identically-named file. Even if the file had identical contents, removing it and creating a new one is technically a whole new resource, and has impact on the system (permissions may be different, open filehandles to deleted files are a thing, etc). So what all of us do, all day, every day, is lie to ourselves. We pretend we can recognize files, that we have a unique identifier for them. But actually we don't. What we do is use a composite index and guess. We say, "well it looks like the right file, because it's in the right file path, with the right size, and right name, and right permissions, and (maybe) has the right inode". But actually there's no way to know for sure it's the same file we expect. We just hope it is. If it looks good enough, we go with it. So that's how you automate managing resources. For each type of resource, you use whatever you can as a unique (or composite) identifier, guesstimate, and prompt the user if it's impossible to get a good enough guess. Because that's how humans do it anyway.
jimbokun 2y ago
That was the Docker manifesto.
- x-complexity 2y ago
  > That was the Docker manifesto. It essentially still is. Unless the Dockerfiles are kept secret, any container can be replicated from the given Dockerfile. Barring extreme (distro/system/hardware)-level quirks, a Docker container should be able to run anywhere that Linux can.
  - spott 2y ago
    A container can very rarely be reproduced by a dockerfile. I imagine with a lot of discipline (no apt update, no “latest” tag, no internet access) you can make a reproducible docker file…. But it is far from normal.
    - taberiand 2y ago
      Well sure, making a 100% reproducible build is hard - but Docker makes it easier, not harder. If 100% reproducible is the goal, what's easier than docker?
      - zelphirkalt 2y ago
        Tools that have been designed with reproducibility in mind. Like Guix. Beware, I am definitely not claiming those are easy to use in general. Just that you can get to reproducibility using them more reliably and maybe easier than with docker.
      - otabdeveloper4 2y ago
        > but Docker makes it easier, not harder Incorrect. Step one of reproducibility is "disable unconstrained downloading from the internet". Docker does the opposite.
        2y ago
LinXitoW 2y ago
So you then have binary blobs of state without any documentation of how or why it is the way it is? That doesn't seem maintainable. Dockerfiles are basically this, but with a file documenting the different steps you took to get to that state.
- dewey 2y ago
  That’s not what they are saying. They are saying that the system where you have to declare everything manually is annoying (which it is), ideally it would record the changes while you make changes and then deduplicate them, remove unnecessary ones to arrive at the final playbook that can be replayed if needed.
  - naikrovek 2y ago
    yes it would be nice to have a computer that could read your mind flawlessly.
- 0xbadcafebee 2y ago
  Then I should explain why "a blob" is, in a weird way, actually superior to a Dockerfile. Step 1. You write a Dockerfile. You build it. You test with it. Hey, it works! You push it to production. Step 2. Years later, you need to patch that container in production. So you change the Dockerfile, rebuild, and re-test. Step 3. Uh-oh! The tests don't work anymore! What's going on? I changed one line in the Dockerfile but now the app isn't working (or perhaps the build isn't). What's going on? What's going on is a reproducibility failure. Just having the instructions (or what someone thought was the instructions, or what they were years ago) isn't enough to ensure you get the same results a second time. A million little things you didn't think of may change between builds (and the more time in-between, the more things change), and they may end up breaking things. Without a truly reproducible build, you are likely to have problems trying to rebuild with just the instructions. That's why with Docker containers, we rely on build artifacts. The container you had two years ago? However that was built, with whatever weird combination of arguments and files-on-the-internet-being-pulled-at-build-time and everything else, it was built successfully. If you want to be really sure you patch it correctly, you pull the old container image (not Dockerfile), modify that, and push that to production as a new container. No rebuilding, just patching. This avoids reproducibility failures. That same idea is why you'd want to just download a blob and later re-apply it. The blob was the state of things when it was working. If you tried to just write down the instructions to replicate it, it's likely you'd either 1) get it wrong (it was only working because of some unrelated changes somebody else made and forgot about) or 2) you'd get a reproducibility error. So "the blob" I'm talking about doesn't have to be a literal binary blob. It could be whatever, we're talking about a theoretical idea here. It could be layers like a container, or metadata in a JSON file, or configuration/code that gets auto-generated, etc. I don't care what it is. It just has to describe the state as it was when X was working. It's then up to the tool to figure out how to get back to that state. People already write this as "declarative code" for configuration management tools to do the same thing. I'm saying, I don't want to have to write the code. Just dump it out for me.
pram 2y ago
Sounds like you want autoexpect! https://linux.die.net/man/1/autoexpect https://linux.die.net/man/1/autoexpect
- 0xbadcafebee 2y ago
  Actually no, that's still just monitoring a series of steps to eventually lead to a thing. I don't want to record a series of steps, I just want to dump existing state of a thing.
retrochameleon 2y ago
It kind of sounds like you're describing Ansible. You use modules for common tasks like ensuring a package is installed, a file is present or has certain content, etc. It's declarative and imdempotent.
- bobthecowboy 2y ago
  I've written some fairly complex stuff in Ansible. It is mostly declarative but you should be careful with assumptions about its idempotency, especially if you reach out for community modules.
- 0xbadcafebee 2y ago
  No, I don't want to write Ansible configuration. I don't want to write any configuration. I just want to dump state and restore it. (To put it another way: I want to auto-generate ansible configuration based on an existing system, and then apply that configuration again later)
  - retrochameleon 2y ago
    Well now it just sounds like you're describing a system backup and restore.
zelphirkalt 2y ago
Such a process is rarely portable though, and will need to be repeated for each different system, at which point it would be great to already have a declarative description, that can automatically be translated into those steps required to get to state X.
sorrythanks 2y ago
> If it's local-first then it's already subject to rot. Can you expand on this?
- 0xbadcafebee 2y ago
  When most people say "local" what they mean is "i'm running something on my laptop". That is to say, it's a random operating system, with a randomly set-up environment, with randomly installed tools. Could be different tools that have the same name but incompatible options. Could be any version (version 3 might be incompatible with version 4, but it's the same name for the command). And it will definitely change over time. This will lead whatever steps you've recorded to 1) stop working on the existing "local" machine, and 2) be incompatible with other people's "local" machines. So the instructions have "rotted". To avoid this you could write Ansible/Puppet to install and configure all the same tools, but that will break too over time too, and be a maintenance hassle. The only reliable solution is to use containers to run it all in; that guarantees the same version of everything. But at that point it's not really "running locally" anymore, it's running in the container, which is sort of its own bag of issues. At that point you might as well have a SaaS tool to run your runbook in a cloud environment in containers or something, as that's way easier to set up and manage than either ansible/puppet, or Docker on everyone's machine (there's still a million tech "engineers" out there who don't understand containers).
  - IOT_Apprentice 2y ago
    Pre Cloud at eBay we managed 30k windows servers via a tool called site controller that referenced ALL servers as objects with properties of configuration state, build version and operational state. ALL databases had their startup configuration parameters defined per instance across datacenters. Furthermore SRE had tools to rate limit connections or restart of a database so that it was not overwhelmed by incoming connection requests. We also built tools to do fine grained definition of load balancers and what services were behind them to be able to redirect traffic on the fly and then reset to the original mappings once connectivity of the LBs was resolved. These tools weee centralized and available to both SRE and senior system administrators. These things evolved to new tools which accommodated a private cloud, then docker then Kubernetes. I left prior to kubernetes implementation.

milkshakes 2y ago

what happens when you want to tweak something you did in the middle of this process? do you have to go through the whole flow again manually to make a single change?

0xbadcafebee 2y ago
I imagine you could either A) just modify the dumped state, B) paramaterize it, C) have the program split up the state into transactions and modify those. The program will probably have to take more than one step, in order, in order to accomplish everything. If it fails, you'd want it to try to undo it, ala transactions. And since it can do all that, it can stop, start, or resume at specific steps. Like, Terraform has always sucked because there was no way to dump existing resources as new code. So a team at Google made a tool to do it (Terraform-er). If Terraform had already had that feature, and if it didn't rely on having pre-existing state to manage resources, that would be like 90% of the way to what I'd want. Just dump resources as code, then let me re-run the code, and if I want I can modify the code to ask me for inputs or change things. (People think of Terraform as only working on Cloud resources, but you could (for example) make an Ubuntu Linux provider that just configures Ubuntu for you, if you wanted)
- kiitos 2y ago
  Any notion of state that satisfies requirements like > Just dump state and later reapply state is necessarily declarative. > Just dump resources as code, What is the code for this resource? VM foo1 Memory 16GiB Network mynet1 It depends on the current state of the system where the resource is applied. If VM foo1 already exists, with 16GiB of memory, and connected to network mynet1, then the code is a no-op, no code at all. Right? Anything else would be a mistake. For example if the code would delete any matching VM and re-create it, that would be disastrous to continuity and availability, clearly a non-starter. Or, if VM foo1 exists, with 16GiB of memory, but connected to anothernet3, then the code should just change the network for that VM from anothernet3 to mynet1, and should definitely not destroy and re-create the VM entirely. And so on.
  - 0xbadcafebee 2y ago
    It depends what you're talking about; Terraform specifically has a flawed model where it assumes nothing in the world exists that it didn't create itself. Other configuration management tools don't assume that; they assume that you just want an item to exist; if it does exist, great, if it doesn't exist, you create it. But for a moment I'll assume you're talking about the other problem with configuration management tools, which is "which of the existing resources do I actually want to exist or modify?" That's a solved problem. Anything that you use on a computer that controls a resource, can uniquely identify said resource, through either a key or composite key. This has to be the case, otherwise you could create things that you could never find again :) (Even if you created an array of things with no name, since it exists as an item in a list, the list index is its unique identifier) Taking Terraform as example again, the provider has code in it that specifies what the unique identifier is, per-resource. It might be a single key (like 'id', 'ASN', 'Name', etc) or a composite key ( {'id' + 'VPC' + 'Region'} ). If the code you've dumped does not have the unique identifier for some reason, then the provider has to make a decision: either try to look up existing resources that match what you've provided and assume the closest one is the right one, or error out that the unique identifier is missing. Usually the unique identifier is not hard to look up in the first place (yours has a composite identifier: {VM:"foo1", Network:"mynet1"}). But it's also (usually) not fool-proof. Imagine a filesystem. You actually have two unique identifiers: the fully-qualified file path, and the inode number. The inode number is the actual unique identifier in the filesystem, but we don't tend to reference it, as 1) it's not that easy to remember/recognize an inode number, 2) it can be recycled for another file, 3) it'll change across filesystems. We instead reference the file path. But file paths are subtly complex: we have sym-links, hard-links and bind-mounts, so two different paths can actually lead to the same file, or different files! On top of that, you can remove the file and then create an identically-named file. Even if the file had identical contents, removing it and creating a new one is technically a whole new resource, and has impact on the system (permissions may be different, open filehandles to deleted files are a thing, etc). So what all of us do, all day, every day, is lie to ourselves. We pretend we can recognize files, that we have a unique identifier for them. But actually we don't. What we do is use a composite index and guess. We say, "well it looks like the right file, because it's in the right file path, with the right size, and right name, and right permissions, and (maybe) has the right inode". But actually there's no way to know for sure it's the same file we expect. We just hope it is. If it looks good enough, we go with it. So that's how you automate managing resources. For each type of resource, you use whatever you can as a unique (or composite) identifier, guesstimate, and prompt the user if it's impossible to get a good enough guess. Because that's how humans do it anyway.

jimbokun 2y ago

That was the Docker manifesto.

x-complexity 2y ago
> That was the Docker manifesto. It essentially still is. Unless the Dockerfiles are kept secret, any container can be replicated from the given Dockerfile. Barring extreme (distro/system/hardware)-level quirks, a Docker container should be able to run anywhere that Linux can.
- spott 2y ago
  A container can very rarely be reproduced by a dockerfile. I imagine with a lot of discipline (no apt update, no “latest” tag, no internet access) you can make a reproducible docker file…. But it is far from normal.
  - taberiand 2y ago
    Well sure, making a 100% reproducible build is hard - but Docker makes it easier, not harder. If 100% reproducible is the goal, what's easier than docker?
    - zelphirkalt 2y ago
      Tools that have been designed with reproducibility in mind. Like Guix. Beware, I am definitely not claiming those are easy to use in general. Just that you can get to reproducibility using them more reliably and maybe easier than with docker.
    - otabdeveloper4 2y ago
      > but Docker makes it easier, not harder Incorrect. Step one of reproducibility is "disable unconstrained downloading from the internet". Docker does the opposite.
      - 2y ago

LinXitoW 2y ago

So you then have binary blobs of state without any documentation of how or why it is the way it is? That doesn't seem maintainable. Dockerfiles are basically this, but with a file documenting the different steps you took to get to that state.

dewey 2y ago
That’s not what they are saying. They are saying that the system where you have to declare everything manually is annoying (which it is), ideally it would record the changes while you make changes and then deduplicate them, remove unnecessary ones to arrive at the final playbook that can be replayed if needed.
- naikrovek 2y ago
  yes it would be nice to have a computer that could read your mind flawlessly.
0xbadcafebee 2y ago
Then I should explain why "a blob" is, in a weird way, actually superior to a Dockerfile. Step 1. You write a Dockerfile. You build it. You test with it. Hey, it works! You push it to production. Step 2. Years later, you need to patch that container in production. So you change the Dockerfile, rebuild, and re-test. Step 3. Uh-oh! The tests don't work anymore! What's going on? I changed one line in the Dockerfile but now the app isn't working (or perhaps the build isn't). What's going on? What's going on is a reproducibility failure. Just having the instructions (or what someone thought was the instructions, or what they were years ago) isn't enough to ensure you get the same results a second time. A million little things you didn't think of may change between builds (and the more time in-between, the more things change), and they may end up breaking things. Without a truly reproducible build, you are likely to have problems trying to rebuild with just the instructions. That's why with Docker containers, we rely on build artifacts. The container you had two years ago? However that was built, with whatever weird combination of arguments and files-on-the-internet-being-pulled-at-build-time and everything else, it was built successfully. If you want to be really sure you patch it correctly, you pull the old container image (not Dockerfile), modify that, and push that to production as a new container. No rebuilding, just patching. This avoids reproducibility failures. That same idea is why you'd want to just download a blob and later re-apply it. The blob was the state of things when it was working. If you tried to just write down the instructions to replicate it, it's likely you'd either 1) get it wrong (it was only working because of some unrelated changes somebody else made and forgot about) or 2) you'd get a reproducibility error. So "the blob" I'm talking about doesn't have to be a literal binary blob. It could be whatever, we're talking about a theoretical idea here. It could be layers like a container, or metadata in a JSON file, or configuration/code that gets auto-generated, etc. I don't care what it is. It just has to describe the state as it was when X was working. It's then up to the tool to figure out how to get back to that state. People already write this as "declarative code" for configuration management tools to do the same thing. I'm saying, I don't want to have to write the code. Just dump it out for me.

pram 2y ago

Sounds like you want autoexpect! https://linux.die.net/man/1/autoexpect https://linux.die.net/man/1/autoexpect

0xbadcafebee 2y ago
Actually no, that's still just monitoring a series of steps to eventually lead to a thing. I don't want to record a series of steps, I just want to dump existing state of a thing.

retrochameleon 2y ago

It kind of sounds like you're describing Ansible. You use modules for common tasks like ensuring a package is installed, a file is present or has certain content, etc. It's declarative and imdempotent.

bobthecowboy 2y ago
I've written some fairly complex stuff in Ansible. It is mostly declarative but you should be careful with assumptions about its idempotency, especially if you reach out for community modules.
0xbadcafebee 2y ago
No, I don't want to write Ansible configuration. I don't want to write any configuration. I just want to dump state and restore it. (To put it another way: I want to auto-generate ansible configuration based on an existing system, and then apply that configuration again later)
- retrochameleon 2y ago
  Well now it just sounds like you're describing a system backup and restore.

zelphirkalt 2y ago

Such a process is rarely portable though, and will need to be repeated for each different system, at which point it would be great to already have a declarative description, that can automatically be translated into those steps required to get to state X.

sorrythanks 2y ago

> If it's local-first then it's already subject to rot. Can you expand on this?

0xbadcafebee 2y ago
When most people say "local" what they mean is "i'm running something on my laptop". That is to say, it's a random operating system, with a randomly set-up environment, with randomly installed tools. Could be different tools that have the same name but incompatible options. Could be any version (version 3 might be incompatible with version 4, but it's the same name for the command). And it will definitely change over time. This will lead whatever steps you've recorded to 1) stop working on the existing "local" machine, and 2) be incompatible with other people's "local" machines. So the instructions have "rotted". To avoid this you could write Ansible/Puppet to install and configure all the same tools, but that will break too over time too, and be a maintenance hassle. The only reliable solution is to use containers to run it all in; that guarantees the same version of everything. But at that point it's not really "running locally" anymore, it's running in the container, which is sort of its own bag of issues. At that point you might as well have a SaaS tool to run your runbook in a cloud environment in containers or something, as that's way easier to set up and manage than either ansible/puppet, or Docker on everyone's machine (there's still a million tech "engineers" out there who don't understand containers).
- IOT_Apprentice 2y ago
  Pre Cloud at eBay we managed 30k windows servers via a tool called site controller that referenced ALL servers as objects with properties of configuration state, build version and operational state. ALL databases had their startup configuration parameters defined per instance across datacenters. Furthermore SRE had tools to rate limit connections or restart of a database so that it was not overwhelmed by incoming connection requests. We also built tools to do fine grained definition of load balancers and what services were behind them to be able to redirect traffic on the fly and then reset to the original mappings once connectivity of the LBs was resolved. These tools weee centralized and available to both SRE and senior system administrators. These things evolved to new tools which accommodated a private cloud, then docker then Kubernetes. I left prior to kubernetes implementation.