6 ms·

Show HN: EDB – A framework to make and manage backups of your database

31 points by RoxasShadow 11y ago

dchest 11y ago
Filed issues related to crypto: https://github.com/RoxasShadow/EDB/issues/created_by/dchest https://github.com/RoxasShadow/EDB/issues/created_by/dchest
- RoxasShadow 11y ago
  Thanks for the good points raised!
inquist 11y ago
Simple, clean, straightforward. I've done similar things in bash before but not bothered to publish :) It's a good idea though to have an ecosystem of simple but flexible tools like this. Here's another idea we implemented alongside a backup system like this. Say you run a backup daily; you can run another script to prune the archives, keeping D most recent daily archives, W most recent weekly archives, M most recent monthly archives, and all yearly archives.
- RoxasShadow 11y ago
  Thank you for your feedback. That would suite well as module when used alongside the FTP module. I'll keep it in mind :)
mickeyben 11y ago
I've recently created a similar tool at our company. We found out that dumping the database table by table in a parallelized way is much faster than a full database mysqldump. There's also mysqlpump soon available in mysql 5.7 to replace mysqldump and works in parallel.
- falcolas 11y ago
  Until then, have a look at "mydumper", which does both parallel dumps at a table level, but also chunking, which makes de-duplication possible.
falcolas 11y ago
The mysql driver needs additional work. In particular, it needs a '--single-transaction' flag, or a global lock, to ensure that the dump is consistent - particularly if you want to dump multiple databases concurrently. Doing one dump per table with chunking (each file has N rows) would help with both speed and disk sizes of backups by allowing S3 or some other program to implement de-duplication between incremental backups. It also wouldn't hurt to capture the binlog position, if available, to enable point in time recovery. Have a look at mydumper for an idea of how another tool implemented these: https://launchpad.net/mydumper https://launchpad.net/mydumper
- RoxasShadow 11y ago
  I will open an issue with your suggestion. Thanks.

31 points by RoxasShadow 11y ago

Filed issues related to crypto: https://github.com/RoxasShadow/EDB/issues/created_by/dchest https://github.com/RoxasShadow/EDB/issues/created_by/dchest

RoxasShadow 11y ago
Thanks for the good points raised!

inquist 11y ago

Simple, clean, straightforward. I've done similar things in bash before but not bothered to publish :) It's a good idea though to have an ecosystem of simple but flexible tools like this. Here's another idea we implemented alongside a backup system like this. Say you run a backup daily; you can run another script to prune the archives, keeping D most recent daily archives, W most recent weekly archives, M most recent monthly archives, and all yearly archives.

RoxasShadow 11y ago
Thank you for your feedback. That would suite well as module when used alongside the FTP module. I'll keep it in mind :)

mickeyben 11y ago

I've recently created a similar tool at our company. We found out that dumping the database table by table in a parallelized way is much faster than a full database mysqldump. There's also mysqlpump soon available in mysql 5.7 to replace mysqldump and works in parallel.

falcolas 11y ago
Until then, have a look at "mydumper", which does both parallel dumps at a table level, but also chunking, which makes de-duplication possible.

falcolas 11y ago

The mysql driver needs additional work. In particular, it needs a '--single-transaction' flag, or a global lock, to ensure that the dump is consistent - particularly if you want to dump multiple databases concurrently. Doing one dump per table with chunking (each file has N rows) would help with both speed and disk sizes of backups by allowing S3 or some other program to implement de-duplication between incremental backups. It also wouldn't hurt to capture the binlog position, if available, to enable point in time recovery. Have a look at mydumper for an idea of how another tool implemented these: https://launchpad.net/mydumper https://launchpad.net/mydumper

RoxasShadow 11y ago
I will open an issue with your suggestion. Thanks.