3 ms·
RDS is very useful for companies which are big enough to employ, say, 2 programmers, but still too small to employ a DBA. The hard part of running a database,
by ekidd 3y ago
RDS is very useful for companies which are big enough to employ, say, 2 programmers, but still too small to employ a DBA.
The hard part of running a database, in my experience, isn't setting up or running it. The hard part isn't even configuring backups.
The hard part is noticing that your backups have been broken for years, before you actually need to restore from them. Yes, yes, you know how to do this correctly. But you delegated it to the sysadmin, the sysadmin subtly broke the backup scripts, the scripts have been silently doing nothing for 18 months, and then the sysadmin got a new job.
This is the main value proposition of RDS: Your data will be backed up, your backups will be restorable, and most of your normal admin tasks can be performed by pushing a button.
- abrookewood 3y agoAgreed ... which is why we test our restores periodically. Feels annoyingly like 'wasted' time, but it's worth it.
- spiffytech 3y ago> Your data will be backed up, your backups will be restorable Well, usually... https://news.ycombinator.com/item?id=13620622 https://news.ycombinator.com/item?id=13620622 Pretty sure I've read similar stories more recently. Probably still better odds of successful restore with RDS than with a home-rolled setup. But there's no substitute for verifying backups.
- KingOfCoders 3y agoIn my startup (until our exit) I've run Postgres on my own, with streaming backups, daily backup tests (automatic restores and checks), offsite backups and (slow) upgrades. Data wasn't mission critical and customers could live with 5min downtimes for upgrades once a year. No problem for years. When we had hosted Mongo, we had more problems. If I have the money I'll use a managed database. But running Postgres in a startup up to $10m ARR / ~1TB data seems not like a problem.
- ssd532 3y agoHow do you test backup restoration on daily basis?
- KingOfCoders 3y agoA script creates a database instance from a container, restores backup into it, makes some checks (e.G. table sizes above some value, audit table contains data up to the backup point etc.), sends out email that everything looks ok.
- KingOfCoders 3y agoAlso unzipping and decrypting, already makes sure encryption worked and zipping worked and you did not end up with a 0 byte backup file (because of permission etc.). Same goes of course for snapshots and backups in the cloud. I have several clients which had backup problems because of misconfiguration in AWS/GCP.
- ssd532 3y agosuper. I have been thinking of doing something similar for my productions db systems as well but was struggling to come up with some efficient way to test restoration. This gives me some starting point.
- KingOfCoders 3y agoYou will be very happy when restoration testing fails. Better to find out in a test than - as some of my clients have - when restoring a backup in an emergency.
- sharts 3y agoWhile true, you'd think something as important as a DB would be something worthy of creating some kind of nightly automated testing/verification of a restore procedure (or multiple).
- neilv 3y agoExactly, on the backups. There was off-the-shelf open source PITR software that I looked at and could've configured, but I couldn't justify spending all the time to test that setup, given that (supposedly) RDS was rock-solid turn-key. There were other engineering and ops things that needed my time more.