5 ms·

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it. For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them. On a production system, do not keep the database on the same server as the application.

snuxoll 11y ago
It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.
- vinceguidry 11y ago
  Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.
  - snuxoll 11y ago
    So go ahead and vendor them in, outside of inclusion in distribution software packages (with exceptions) there's nothing stopping you from including your own dependencies. Packaging your application properly makes deployment a hell of a lot easier and less error-prone.
lqdc13 11y ago
What? Why is it a bad thing? I guess it depends on someone's definition of a database.
- vinceguidry 11y ago
  Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains. If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then. On systems I build, cron jobs are added as part of the deployment process and managed as code, in the git repository. That's just one example. Others include iptables rules, FTP server configuration, startup scripts, and such. If it's not in your codebase, it's unmanaged state on the server that you will lose if your cloud provider takes a shit. All of that needs to be managed as code or as configuration and deploy scripts should contain idempotent commands that add them if they are not present. I consider shelling into a server for any reason to be unideal and will think of ways to avoid it. If you are asking about database being on the same server as the application, it's because it makes infrastructure management more difficult. Application servers are set up quickly, torn down quickly, and work as soon as you deploy. Database servers are fortresses, you don't build new ones and tear them down half as quickly, unless you've got scaling requirements. It is tempting to want to put data on the same server as the application early on in order to save on hosting costs, with the understanding that you'll separate them later, but that's a rookie move. It's much less work to separate them now, when there's nothing riding on it, much less headache.
  - kasey_junk 11y ago
    I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database). That said you could easily use the same advice for that data. But in some cases the application itself needs to be on the same server as the database. Think if you were building postgres for instance...
    - vinceguidry 11y ago
hinkley 11y ago
And the nice thing about DO and some of their cohort is that they charge by the hour for servers. Fire up new droplets and install your upgrades on them, smoke test the system, swap your routing over to the new machines, back up the old droplets (just in case you missed something) and and then destroy them. If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.

snuxoll 11y ago

It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.

vinceguidry 11y ago
Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.
- snuxoll 11y ago
  So go ahead and vendor them in, outside of inclusion in distribution software packages (with exceptions) there's nothing stopping you from including your own dependencies. Packaging your application properly makes deployment a hell of a lot easier and less error-prone.

lqdc13 11y ago

What? Why is it a bad thing? I guess it depends on someone's definition of a database.

vinceguidry 11y ago
Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains. If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then. On systems I build, cron jobs are added as part of the deployment process and managed as code, in the git repository. That's just one example. Others include iptables rules, FTP server configuration, startup scripts, and such. If it's not in your codebase, it's unmanaged state on the server that you will lose if your cloud provider takes a shit. All of that needs to be managed as code or as configuration and deploy scripts should contain idempotent commands that add them if they are not present. I consider shelling into a server for any reason to be unideal and will think of ways to avoid it. If you are asking about database being on the same server as the application, it's because it makes infrastructure management more difficult. Application servers are set up quickly, torn down quickly, and work as soon as you deploy. Database servers are fortresses, you don't build new ones and tear them down half as quickly, unless you've got scaling requirements. It is tempting to want to put data on the same server as the application early on in order to save on hosting costs, with the understanding that you'll separate them later, but that's a rookie move. It's much less work to separate them now, when there's nothing riding on it, much less headache.
- kasey_junk 11y ago
  I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database). That said you could easily use the same advice for that data. But in some cases the application itself needs to be on the same server as the database. Think if you were building postgres for instance...
  - vinceguidry 11y ago

hinkley 11y ago

And the nice thing about DO and some of their cohort is that they charge by the hour for servers. Fire up new droplets and install your upgrades on them, smoke test the system, swap your routing over to the new machines, back up the old droplets (just in case you missed something) and and then destroy them. If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.