2 ms·
Ugh. A very painful occurrence. A couple of recommendations, in the event that Kickstarter reads this thread: - Run pt-table-checksum on a daily or weekly cron
by falcolas 14y ago
Ugh. A very painful occurrence. A couple of recommendations, in the event that Kickstarter reads this thread:
- Run pt-table-checksum on a daily or weekly cron, and fix what it finds immediately.
* Switch off RDS.
* Your db admins will thank you for taking off the kid gloves
* You can restore from binary backups quickly.
* Performance will get better
* You can use something like pt-slave-restart to get replication running quickly and re-sync the DBs after the immediate crisis is over.
* If you don't already have it, hook up Nagios to RDS with some MySQL monitoring plugins like pmp.
Data drift, and the resultant replication downtime, is an unfortunate reality of asynchronous replication. It doesn't have to be multiple hours worth of scramble, however - if you catch it early (with monitoring) and are prepared to handle it (with pt-table-sync or binary backups).