3 ms·
I use rds Aurora but i run a daily cron job to export the entire thing with mysqldump + gzip to s3. I pass some flags to mysqldump to avoid locking the db and
by etaioinshrdlu 7y ago
I use rds Aurora but i run a daily cron job to export the entire thing with mysqldump + gzip to s3.
I pass some flags to mysqldump to avoid locking the db and otherwise interfering with production. It also dumps from the reader node not the writer.
I also clear some tables and restore it daily to a development DB.
Sadly this looks like it only supports parquet, a rather usual db dump format. Even if, I'm sure, it's way more efficient to process using some modern tools than a text sql dump.
I just like open formats and interoperability and AWS's rds offerings are always just slightly 'off'.
- sudhirj 7y agoParquet is an open format. It’s part of the Apache foundation. Would have preferred CSV as well, though, easier to work with. There’s also a little known feature called S3 Select that allows you to run SQL on S3 files and selectively retrieve data, which is what Parquet is for in the first place.
- etaioinshrdlu 7y agoCan you load that dump using any open source tools outside of AWS and recreate the database identically? Because you can do that with mysqldump.
- derision 7y agoI don't think this is meant for recreating the database, but rather exporting large sets for analysis somewhere else. But as far as tools yes, parquet can be parsed by any tool that knows the format. It's not too uncommon