3 ms·
pt-online-schema change is a great tool, but not a panacea. I've seen a serious issue where under moderately heavy load, the OSC tool would trigger a deadlock
by oasisbob 14y ago
pt-online-schema change is a great tool, but not a panacea.
I've seen a serious issue where under moderately heavy load, the OSC tool would trigger a deadlock on the final atomic table swap, blocking all transactions against the table, including the final `RENAME TABLE`.
All operations (including reads) would block, waiting on a table metadata lock. Killing the offending RENAME thread was the only way to restore access.
IIRC, the ultimate cause was due to a bug in our code that would occasionally leave stale transactions open in edge cases. Happened a few times, and caused a fairly severe outage the first time we hit it.
Not sure where the blame here lies -- and what can fundamentally be improved -- but OSC techniques and MySQL still leave plenty of room for a moderately savvy user to get into trouble.