4 ms·

Not to be rude but this isn't how this works at all. Things like 'run an "INSERT ... SELECT ..."' can't happen at scale due to locking. How they actually do it

Not to be rude but this isn't how this works at all. Things like 'run an "INSERT ... SELECT ..."' can't happen at scale due to locking. How they actually do it is super rad: https://www.percona.com/doc/percona-toolkit/3.0/pt-online-schema-change.html https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc... tl;dr; They setup a system of triggers (updates,inserts,etc) , copy the data over, then run through all the data in the trigger system. percona developed all these fancy features as well to monitor replica data etc. Another way with cloud vms (terabyte+ tables), you image a replica, do the alter, let the replica catch up, image it, build replicas off that, promote this to master. Facebook's internal one I hear is close to this.

derefr 5y ago
"This isn't how things work at all" implies that more than 0.0000001% of DBs are running "at scale." Most DBs people will ever deal with — including within big enterprises! — are things like customer databases that have maybe 100k records in them. The tables lock waiting for lower-xact-ID read traffic to clear, yes — for about 400ms. Because queries running on the DB are all point queries that don't run any longer than that. Or they're scheduled batch jobs that happen during a maintenance window. At scale, you're hopefully not using an RDBMS as a source of truth in the first place; but rather, using it as a CQRS/ES aggregate, downstream of a primary event store (itself likely some combination of a durable message-queue, and archival object storage for compacted event segments.) In that kind of setup, data migrations aren't the job of the RDBMS itself, but rather the job of the CQRS/ES framework — which can simply be taught a new aggregate that computes a single new column, and will start catching that aggregate up and making the data from it available. If your RDBMS is columnar (again, hopefully), you can just begin loading that new column in, no locking on the rest of the table required. IMHO, the trigger-based approach is a weak approximation of having a pre-normalization primary source. It's fine if you want to avoid rearchitecture, but in a HOLAP system (which is usually inevitable for these sorts of systems when their data architects have tried to focus on simplicity, as this leads to eschewing denormalized secondary representations in OLAP-focused stores) it will cause your write perf to degrade + bottleneck, which is actually the worst thing for locking. (I should know; I'm dealing with a HOLAP system right now with large numbers of computed indices, append-only OLTP inserts, and random OLAP reads; where the reads hold hundreds of locks each due to partitioning. Any time a write tx stays open for more than a few hundred milliseconds, the whole system degrades due to read locks piling up to the point that the DB begins to choke just on allocating and synchronizing them. The DB is only a few TB large, and the instance it's on has 1TB of memory and as many cores as one can get... but locking is locking.)

derefr 5y ago

"This isn't how things work at all" implies that more than 0.0000001% of DBs are running "at scale." Most DBs people will ever deal with — including within big enterprises! — are things like customer databases that have maybe 100k records in them. The tables lock waiting for lower-xact-ID read traffic to clear, yes — for about 400ms. Because queries running on the DB are all point queries that don't run any longer than that. Or they're scheduled batch jobs that happen during a maintenance window. At scale, you're hopefully not using an RDBMS as a source of truth in the first place; but rather, using it as a CQRS/ES aggregate, downstream of a primary event store (itself likely some combination of a durable message-queue, and archival object storage for compacted event segments.) In that kind of setup, data migrations aren't the job of the RDBMS itself, but rather the job of the CQRS/ES framework — which can simply be taught a new aggregate that computes a single new column, and will start catching that aggregate up and making the data from it available. If your RDBMS is columnar (again, hopefully), you can just begin loading that new column in, no locking on the rest of the table required. IMHO, the trigger-based approach is a weak approximation of having a pre-normalization primary source. It's fine if you want to avoid rearchitecture, but in a HOLAP system (which is usually inevitable for these sorts of systems when their data architects have tried to focus on simplicity, as this leads to eschewing denormalized secondary representations in OLAP-focused stores) it will cause your write perf to degrade + bottleneck, which is actually the worst thing for locking. (I should know; I'm dealing with a HOLAP system right now with large numbers of computed indices, append-only OLTP inserts, and random OLAP reads; where the reads hold hundreds of locks each due to partitioning. Any time a write tx stays open for more than a few hundred milliseconds, the whole system degrades due to read locks piling up to the point that the DB begins to choke just on allocating and synchronizing them. The DB is only a few TB large, and the instance it's on has 1TB of memory and as many cores as one can get... but locking is locking.)