4 ms·

> Anyone see a flaw in this? Perhaps not a flaw, but some issues with your setup are implied. If you're rebuilding from scratch because you're not sure that y

by Jedd 10y ago

> Anyone see a flaw in this? Perhaps not a flaw, but some issues with your setup are implied. If you're rebuilding from scratch because you're not sure that you can update things, then you're probably in need of a configuration management tool (I'm a big fan of saltstack[1], mostly because I don't like Ruby or DSL's, but there's lots of options out there[2]) If you're worried you're going to lose transitory data, it sounds like you don't have a trusted and tested backup/archival/recovery process in place. So having it stored on a single EBS / DO BS / etc means you're still exposed. If you're rebuilding and rolling data over, in this scenario, I'd be copying, rather than relocating, any precarious data repositories. [1] https://saltstack.com/ https://saltstack.com/ [2] https://en.wikipedia.org/wiki/Comparison_of_open-source_configuration_management_software https://en.wikipedia.org/wiki/Comparison_of_open-source_conf...

vidarh 10y ago
I tend to avoid those config management tools other than for basic bootstrapping exactly because while you can use those too to recreate from scratch, when you don't do that, you leave the door open for undocumented, unknown state, since most of them basically take a system in an unknown-but-hopefully-mostly-consistent state and try to bring them to a known state. But they'll only be in a known state in that case if your setup is extremely comprehensive. In practice I've seen too many config management tools where long running servers have ended up in unknown states because changes have been applied, and subsequently changes have either been made outside of the toolchain, or changes have been done to the config in ways that doesn't let the tool know what has changed, or the tools simply doesn't have a way of comparing machine states without comprehensively enumerating everything on the server (e.g. people running Ansible playbooks that adds X, subsequently removing the requirement X from the playbook, and going about without considering whether or not X will interact with Y which they've added later). As a result, I see rebuilding from scratch as largely orthogonal to whether or not you use a configuration management tool or e.g. build VM images that you replace wholesale, or whatever you do: You should rebuild from scratch regularly, as coupled with a test-suite it's the only realistic way of knowing whether or not you've left anything out of your build process. My biggest caveat with config management systems is that they tend to end up encouraging live changes to a setup, instead of a build-test-deploy cycle. Sometimes that's necessary, but to me that's a last resort.
- brazzledazzle 10y ago
  I actually agree with your overall point but generally speaking if someone is making changes outside of your standardized toolchain you have a human problem. Emergencies aside, you use those tools for a reason. Straddling the fence is almost the worst of both worlds.
  - vidarh 10y ago
    Agreed. But in my experience, the easier you make going outside the process the easier it becomes to invent excuses for why it is ok. A lot of my job involves making the right thing to do the path of least resistance, because when it isn't, because humans overall tend to be the cause of a whole lot more of the problems than the servers.
andybak 10y ago
I probably wasn't clear: 1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook. 2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change. I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a running instance should result in the same state as a genuine 'build from scratch' but there are many places this can fail: 1. Someone has inadvertently modified the environment at some point. A quick-fix or accidental change or an alteration in my playbook that isn't clearly reflected in the running instance 2. Not all ansible modules are perfectly idempotent. It's a fairly leaky abstraction. 3. If you never build from scratch apart from the first time then you're not really testing your automated deployment. The next time you really need to build from scratch you might get a shock.
- Jedd 10y ago
  > 1. Someone has inadvertently modified the environment at some point. > A quick-fix or accidental change or an alteration in my playbook that > isn't clearly reflected in the running instance This will cause you comparable problems whether you're building from scratch or upgrading in place (iff you're using a tool like ansible). As noted by another responder, you've got a human problem there, not necessarily a technical one. > 2. Not all ansible modules are perfectly idempotent. It's a > fairly leaky abstraction. I've not used ansible, but have used a few other tools, so I understand how this is (easily) possible. As in other cases, the trick, of course, is to fix this. ; ) > 3. If you never build from scratch apart from the first time then > you're not really testing your automated deployment. The next time > you really need to build from scratch you might get a shock. Totally agreed. Your original post suggested the problem you had was with transitory data - evidently your concern is more to do with the ability to consistently and reliably rebuild a known and trusted environment. Persistent block storage won't help you with that.
  - andybak 10y ago
    What I was hoping it would do is enable me to use a single droplet but get a fast, minimal down-time 'from scratch' deployment that retains user data. The reason I want a single droplet is mainly cost. I use one droplet per client rather than a multitenant setup. I could use S3 and a separate hosted db but for various reasons I'd prefer not to.

by Jedd 10y ago

vidarh 10y ago

I tend to avoid those config management tools other than for basic bootstrapping exactly because while you can use those too to recreate from scratch, when you don't do that, you leave the door open for undocumented, unknown state, since most of them basically take a system in an unknown-but-hopefully-mostly-consistent state and try to bring them to a known state. But they'll only be in a known state in that case if your setup is extremely comprehensive. In practice I've seen too many config management tools where long running servers have ended up in unknown states because changes have been applied, and subsequently changes have either been made outside of the toolchain, or changes have been done to the config in ways that doesn't let the tool know what has changed, or the tools simply doesn't have a way of comparing machine states without comprehensively enumerating everything on the server (e.g. people running Ansible playbooks that adds X, subsequently removing the requirement X from the playbook, and going about without considering whether or not X will interact with Y which they've added later). As a result, I see rebuilding from scratch as largely orthogonal to whether or not you use a configuration management tool or e.g. build VM images that you replace wholesale, or whatever you do: You should rebuild from scratch regularly, as coupled with a test-suite it's the only realistic way of knowing whether or not you've left anything out of your build process. My biggest caveat with config management systems is that they tend to end up encouraging live changes to a setup, instead of a build-test-deploy cycle. Sometimes that's necessary, but to me that's a last resort.

brazzledazzle 10y ago
I actually agree with your overall point but generally speaking if someone is making changes outside of your standardized toolchain you have a human problem. Emergencies aside, you use those tools for a reason. Straddling the fence is almost the worst of both worlds.
- vidarh 10y ago
  Agreed. But in my experience, the easier you make going outside the process the easier it becomes to invent excuses for why it is ok. A lot of my job involves making the right thing to do the path of least resistance, because when it isn't, because humans overall tend to be the cause of a whole lot more of the problems than the servers.

andybak 10y ago

I probably wasn't clear: 1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook. 2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change. I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a running instance should result in the same state as a genuine 'build from scratch' but there are many places this can fail: 1. Someone has inadvertently modified the environment at some point. A quick-fix or accidental change or an alteration in my playbook that isn't clearly reflected in the running instance 2. Not all ansible modules are perfectly idempotent. It's a fairly leaky abstraction. 3. If you never build from scratch apart from the first time then you're not really testing your automated deployment. The next time you really need to build from scratch you might get a shock.

Jedd 10y ago
> 1. Someone has inadvertently modified the environment at some point. > A quick-fix or accidental change or an alteration in my playbook that > isn't clearly reflected in the running instance This will cause you comparable problems whether you're building from scratch or upgrading in place (iff you're using a tool like ansible). As noted by another responder, you've got a human problem there, not necessarily a technical one. > 2. Not all ansible modules are perfectly idempotent. It's a > fairly leaky abstraction. I've not used ansible, but have used a few other tools, so I understand how this is (easily) possible. As in other cases, the trick, of course, is to fix this. ; ) > 3. If you never build from scratch apart from the first time then > you're not really testing your automated deployment. The next time > you really need to build from scratch you might get a shock. Totally agreed. Your original post suggested the problem you had was with transitory data - evidently your concern is more to do with the ability to consistently and reliably rebuild a known and trusted environment. Persistent block storage won't help you with that.
- andybak 10y ago
  What I was hoping it would do is enable me to use a single droplet but get a fast, minimal down-time 'from scratch' deployment that retains user data. The reason I want a single droplet is mainly cost. I use one droplet per client rather than a multitenant setup. I could use S3 and a separate hosted db but for various reasons I'd prefer not to.