6 ms·
PostgreSQL Data Migration Tips
- NDizzle 12y agoI like how SQL is becoming cool again.
- ngoel36 12y agoDid it ever really go out of style?
- gdulli 12y agoNot for me. I've used ORMs for the simplest of CRUD operations from web apps but couldn't imagine ever working without SQL for everything else.
- agopaul 12y agoTo be fair, when it comes to data consistency especially on structured dbs in high load environments, all-SQL solutions are usually better than, say, client side scripts wrapped in transactions
- mizerable 12y agonice ! thank you for this
- aanari 12y agoNo problem mizerable! Glad you enjoyed it.
- xtrumanx 12y agoIt was bugging me that the target of the "rosser" link was not pointing to the comment he shared his bulk update technique so I dug it up for anyone else interested: https://news.ycombinator.com/item?id=9018756 https://news.ycombinator.com/item?id=9018756
- aanari 12y agoAuthor here. Great point, I just updated this link to point directly to his comment rather than his HN profile.
- bhahn 12y agoFOR UPDATE NOWAIT immediately locks the rows being retrieved in the first step (as if they were to be updated) Technically "FOR UPDATE" only makes an attempt to lock rows, and the "NOWAIT" instructs postgres, in the case that another transaction already has a lock on the row, to raise an error immediately instead of the default behavior of waiting for the lock to become available. http://www.postgresql.org/docs/9.4/static/sql-select.html http://www.postgresql.org/docs/9.4/static/sql-select.html
- aanari 12y agoThat's a good clarification. I think it's better for the data migration to error out when acquiring the lock on rows and then retry, rather than waiting indefinitely, but YMMV (your mileage may vary).
- steventhedev 12y agoBone to pick with the CTE that archives users. I'd rather see it insert them into the archive table first, then delete them. Not much of a difference, but my instinct is to cover all possible failures, and be especially careful around deleting rows. It also means that god forbid it did die halfway through, and PG isn't smart enough to pick up where it left off safely, you won't lose any data, and at worst would end up with a duplicate archived row (easy enough to catch with some maintenance scripts and origin ids)
- jpitz 12y agoShould be a transaction. I bet if you could a way to corrupt data on safe hardware with this query and a power-plug test, the pg developers would treat it as a high-priority bug.
- aanari 12y agoThat's correct, as jpitz mentioned if the code is run inside a transaction block, then we don't have to worry about the failing DELETE causing the INSERTs to fail.
- jpitz 12y agoThe transaction is implict - in fact, I don't know of a way to do this outside the scope of a transaction. Is there?
- steventhedev 12y agoMy instinct here is to write the query in such a way that even if pg changes in the future, or if I migrate to an almost compatible alternative tomorrow, it should work the same way. Perhaps more importantly, it sends the message to future devs (including me) that rows should only be deleted after they've already been archived. It also covers that 0.000000001% chance that everything else went wrong and the intern did it manually and managed to screw it up. In the end, queries are code, and code is our way to communicate or intent to the next developer, so it's better to do the delete after the insert
- bremac 12y agoI'm not sure I understand the purpose of the loop in the last example. AFAIK top-level plpgsql statements (including DO blocks run in psql) execute in a single transaction, so it seems like you end up slowly locking the entire table, as the transaction won't commit until the loop completes. (I learned this the hard way by trying to "batch"-update a table with tens of millions of rows in production.) The normal way to handle batch updates is to perform the loop outside of postgresql, so that each batch is in its own transaction.
- bhahn 12y agoAFAIK top-level plpgsql statements (including DO blocks run in psql) execute in a single transaction This is true according to the docs. Anonymous code blocks are "transient anonymous functions", and functions are executed within a transaction. it seems like you end up slowly locking the entire table The selected rows would be locked for update, delete, and select for updates, but not for regular reads. Perhaps his users table is used primarily for reads, which made this command run with negligible consequences? http://www.postgresql.org/docs/9.4/static/sql-do.html http://www.postgresql.org/docs/9.4/static/sql-do.html http://www.postgresql.org/docs/9.4/static/plpgsql-structure.html http://www.postgresql.org/docs/9.4/static/plpgsql-structure....
- aanari 12y agoAuthor here. That's a great point bhahn, I just updated my gist to properly handle the case that you just outlined: https://gist.github.com/aanari/349c7d97ed50c6f69930#file-batch_at_will_commander-sql https://gist.github.com/aanari/349c7d97ed50c6f69930#file-bat... By creating a separate function for the locking and updating of rows, we ensure that the `BEGIN/END` transaction is handled per iteration rather than at the very end, so we only lock rows while they are being processed. Since Postgres does not support nested transaction blocks, calling a defined function from within an anonymous function block seemed to be the easiest and clearest path to achieve this.
- bhahn 12y agohttps://gist.github.com/aanari/349c7d97ed50c6f69930#file-batch_at_will_commander-sql-L22-L28 https://gist.github.com/aanari/349c7d97ed50c6f69930#file-bat... The anonymous block is implicitly called in a transaction, so all the calls to batch_at_will will be executed in the parent transaction; there's no way around this (except for using db_link but that's pretty smelly imo). The only way to batch update while only locking rows in the batch is to run the loop outside of postgres like OP suggested.