3 ms·
I agree with most points but would like to add one about "tunability". This article focuses mostly on correctness, which is of course important (and without whi
by WJW 2y ago
I agree with most points but would like to add one about "tunability". This article focuses mostly on correctness, which is of course important (and without which the whole system would be useless) but in itself is often not sufficient for large systems. I've seen it happen more than once that a big table rewrite causes so much load the database might as well be down completely.
One thing I would love to see implemented more thoroughly in database systems is the ability to throttle and/or pause the load that DML can impose. Normal SQL only says "what" the ALTER TABLE should do, but the "how" is often rather lacking. At most you get a CREATE INDEX CONCURRENTLY in postgres or an "ALGORITHM=instant" in mysql, but rarely do you get finegrained enough to say "use at most XYZ iops for this task", let alone that you can vary that XYZ variable dynamically or to assign priorities to load caused by different queries.
AFAIK TiDB and pt-osc provide ways to pause a running migration, gh-ost can also throttle a migration dynamically. Vitess also has several ways to manage migration, as it leverages gh-ost. For postgress I don't think any of the currently popular tools have good ways to manage this, but I would love to be proven wrong.
- YorickPeterse 2y agoGood point, I forgot about tunability. At GitLab we did have a mechanism for this for background migrations, IIRC it looked at the vacuuming statistics of PostgreSQL and would back off if it deemed there was too much load.
- shlomi-noach 2y agoVitess maintainer here. While Vitess can still use gh-ost, the recommended way is to use Vitess' internal VReplication mechanism. With this, your can throttle, pause, resume, cancel, retry, revert a migration, as well as auto-resume from failover (migration continues from point of interruption once a new primary is instated). Vitess throttling is by default based on replication lag, but you can use different metrics, such as load average, or indeed multiple metrics combined, to define what constitutes a load.