4 ms·

It seems grossly irresponsible to encourage the use of serializable without even mentioning deadlocks. I guess you won't have deadlocks if your db interprets as

It seems grossly irresponsible to encourage the use of serializable without even mentioning deadlocks. I guess you won't have deadlocks if your db interprets as serializable as "lock the whole database on transaction start", but that means literally no concurrency whatsoever, which also seems like a grossly irresponsible recommendation. I dunno, maybe I didn't get the memo that the database of the future will be single-threaded.

dmm 7y ago
In postgresql at least, serializable cannot deadlock. It uses predicate locks to provide serializable consistency with high levels of concurrency. You can have serialization failures though.
- basetop 7y ago
  I'm not too familiar with postgresql, but isn't that serializable snapshot rather than serializable? I'm pretty sure all RDMBs have deadlocks in serializable. But in serialiable snapshot, a transaction doesn't deadlock, but simply fails.
  - convolvatron 7y ago
    a transaction system in which its impossible for transactions to ever fail kind of isn't a transaction system. see above comment from abadid - its entirely possible to impose a global ordering of all transactions either up front or retroactively without admitting deadlocks.
  - freels 7y ago
    Serializable Snapshot an implementation strategy for providing Serializable isolation. In PostgreSQL if you ask for Serializable you get SSI.
abadid 7y ago
As mentioned in the post: "There are several ways to achieve [serializability] — such as via locking, validation, or multi-versioning." Deadlock happens under some, but not all implementations of serializability via locking. There have been several database systems developed in my lab that use locking to achieve serializability, but yet never deadlock. Examples include: (1) Calvin: http://www.cs.umd.edu/~abadi/papers/calvin-sigmod12.pdf http://www.cs.umd.edu/~abadi/papers/calvin-sigmod12.pdf (2) Orthrus: http://www.cs.umd.edu/~abadi/papers/orthrus-sigmod16.pdf http://www.cs.umd.edu/~abadi/papers/orthrus-sigmod16.pdf (3) PWV: http://www.cs.umd.edu/~abadi/papers/early-write-visibility.pdf http://www.cs.umd.edu/~abadi/papers/early-write-visibility.p... Bottom line: serializability does not necessarily mean deadlock. Deadlock can be avoided via non-locking implementations, or even in well-designed locking implementations. One of the points in the conclusion of the post warrants being reiterated at this point: "If you find that the cost of serializable isolation in your system is prohibitive, you should probably consider using a different database system earlier than you consider settling for a reduced isolation level."
- kerblang 7y ago
  What I'm talking about here is not academic database technologies in laboratories, but the typical mysql/postgres/oracle/sqlserver production world regular programmers live in. So yes, I know pragmatic strategies for avoiding deadlocks, but in all the databases I've used, you have to proactively employ such a strategy. Many if not most of these programmers have at best a rudimentary knowledge of transactions and/or whether they are even using them (thanks to ORMs). My concern is that someone reads a blog like this, concludes that serializable is best, and proceeds towards disastrous consequences.
  - Rapzid 7y ago
    Those databases all allow and detect deadlock as an optimistic concurrency strategy out of the box.
  - ztorkelson 7y ago
    You appear to suggest that programmers with “rudimentary knowledge of transactions” should prefer lower isolation levels which sacrifice correctness for performance. If anything is “grossly irresponsible” here, it’s that. Such isolation levels are notoriously difficult to reason about—even for experienced practitioners—and their misuse can and does introduce persistent data anomalies that can be costly to remediate. Generally speaking, performance issues are significantly easier to diagnose and resolve than data anomalies, and they may be address in a targeted fashion as the need arises. There’s no substitute for thinking. But if I had to prescribe general advice, it’d be this: (1) When given the choice, select a modern database system which supports scalable and efficient serializable and snapshot transaction isolation levels. (2) Use serializable isolation, by default, for all transactions. (3) In the event that your transactions are not sufficiently performant, stop and investigate. Profile the system to identify the bottleneck. (4) If the bottleneck is contention due to the transaction isolation level, stop. Assess whether the contention is inherent or whether it is incidental to the implementation or data model. (5a) If the contention is incidental, do not lower the isolation level. Instead, refactor to eliminate the contention point. Congratulations; you are now done. (5b) Otherwise, lower the isolation level—only for one or more of the transaction(s) in question—by a single step. Carefully assess the anomalies you have now introduced and the ramifications on the system as a whole. Look for other transactions which could intersect concurrently in time and space. Implement compensatory controls as necessary to accommodate the new behavior. (6) Repeat only as necessary to achieve satisfaction.
magicalhippo 7y ago
At work we use Sybase SQLAnywhere, and we added serializable access to one particular table. Fine if you have a few dozen rows as we did in testing, but after a while in production the DB struggled hard the row count went beyond a hundred k... Turned out the serialized access caused it to issue a lock on every single row (for reasons I still don't understand). The serialized access was used was to basically implement a fancy autoinc ourselves. I rewrote the code to use a unique constraint and retry-loop instead, which eliminated the speed issue entirely.
- blr246 7y ago
  To implement the serializable isolation level, the database system must track access to every single row you access, even the ones you read or filter out. (The need to track reads, even for rows filtered out of a select statement is surprising but necessary.) Consider a common scenario where you SELECT a set of rows and take a SUM over a column. Suppose your query and another query begin reading from the same committed state of the same table. Suppose that the other query uses an UPDATE command on a set of rows in the table, and that the other query commits before yours does. In order to be consistent, the database system must detect the situation where the other query updated a row that affects the filter WHERE you scanned the table, otherwise your sum could be incorrect if the other query's committed state would cause your query to compute a SUM over a different set of rows or over modified values in your SUM. The only way for the database system to guarantee there is no conflict is to keep track of every single row your query accesses, even if it is a row passed over by a WHERE clause! Serializability is a well studied concept. You can find lots of good resources about algorithms for implementing concurrency control and for detecting whether or not two transactions are serializable. The high-level summary is that it takes a lot of operational bookkeeping to guarantee that two queries have no conflicts, especially when you are using real-world examples having many filters and joins.
  - magicalhippo 7y ago
    > The only way for the database system to guarantee there is no conflict is to keep track of every single row your query accesses, even if it is a row passed over by a WHERE clause! Why is putting a lock on the table itself not an option?

dmm 7y ago

In postgresql at least, serializable cannot deadlock. It uses predicate locks to provide serializable consistency with high levels of concurrency. You can have serialization failures though.

basetop 7y ago
I'm not too familiar with postgresql, but isn't that serializable snapshot rather than serializable? I'm pretty sure all RDMBs have deadlocks in serializable. But in serialiable snapshot, a transaction doesn't deadlock, but simply fails.
- convolvatron 7y ago
  a transaction system in which its impossible for transactions to ever fail kind of isn't a transaction system. see above comment from abadid - its entirely possible to impose a global ordering of all transactions either up front or retroactively without admitting deadlocks.
- freels 7y ago
  Serializable Snapshot an implementation strategy for providing Serializable isolation. In PostgreSQL if you ask for Serializable you get SSI.

abadid 7y ago

As mentioned in the post: "There are several ways to achieve [serializability] — such as via locking, validation, or multi-versioning." Deadlock happens under some, but not all implementations of serializability via locking. There have been several database systems developed in my lab that use locking to achieve serializability, but yet never deadlock. Examples include: (1) Calvin: http://www.cs.umd.edu/~abadi/papers/calvin-sigmod12.pdf http://www.cs.umd.edu/~abadi/papers/calvin-sigmod12.pdf (2) Orthrus: http://www.cs.umd.edu/~abadi/papers/orthrus-sigmod16.pdf http://www.cs.umd.edu/~abadi/papers/orthrus-sigmod16.pdf (3) PWV: http://www.cs.umd.edu/~abadi/papers/early-write-visibility.pdf http://www.cs.umd.edu/~abadi/papers/early-write-visibility.p... Bottom line: serializability does not necessarily mean deadlock. Deadlock can be avoided via non-locking implementations, or even in well-designed locking implementations. One of the points in the conclusion of the post warrants being reiterated at this point: "If you find that the cost of serializable isolation in your system is prohibitive, you should probably consider using a different database system earlier than you consider settling for a reduced isolation level."

kerblang 7y ago
What I'm talking about here is not academic database technologies in laboratories, but the typical mysql/postgres/oracle/sqlserver production world regular programmers live in. So yes, I know pragmatic strategies for avoiding deadlocks, but in all the databases I've used, you have to proactively employ such a strategy. Many if not most of these programmers have at best a rudimentary knowledge of transactions and/or whether they are even using them (thanks to ORMs). My concern is that someone reads a blog like this, concludes that serializable is best, and proceeds towards disastrous consequences.
- Rapzid 7y ago
  Those databases all allow and detect deadlock as an optimistic concurrency strategy out of the box.
- ztorkelson 7y ago
  You appear to suggest that programmers with “rudimentary knowledge of transactions” should prefer lower isolation levels which sacrifice correctness for performance. If anything is “grossly irresponsible” here, it’s that. Such isolation levels are notoriously difficult to reason about—even for experienced practitioners—and their misuse can and does introduce persistent data anomalies that can be costly to remediate. Generally speaking, performance issues are significantly easier to diagnose and resolve than data anomalies, and they may be address in a targeted fashion as the need arises. There’s no substitute for thinking. But if I had to prescribe general advice, it’d be this: (1) When given the choice, select a modern database system which supports scalable and efficient serializable and snapshot transaction isolation levels. (2) Use serializable isolation, by default, for all transactions. (3) In the event that your transactions are not sufficiently performant, stop and investigate. Profile the system to identify the bottleneck. (4) If the bottleneck is contention due to the transaction isolation level, stop. Assess whether the contention is inherent or whether it is incidental to the implementation or data model. (5a) If the contention is incidental, do not lower the isolation level. Instead, refactor to eliminate the contention point. Congratulations; you are now done. (5b) Otherwise, lower the isolation level—only for one or more of the transaction(s) in question—by a single step. Carefully assess the anomalies you have now introduced and the ramifications on the system as a whole. Look for other transactions which could intersect concurrently in time and space. Implement compensatory controls as necessary to accommodate the new behavior. (6) Repeat only as necessary to achieve satisfaction.

magicalhippo 7y ago

At work we use Sybase SQLAnywhere, and we added serializable access to one particular table. Fine if you have a few dozen rows as we did in testing, but after a while in production the DB struggled hard the row count went beyond a hundred k... Turned out the serialized access caused it to issue a lock on every single row (for reasons I still don't understand). The serialized access was used was to basically implement a fancy autoinc ourselves. I rewrote the code to use a unique constraint and retry-loop instead, which eliminated the speed issue entirely.

blr246 7y ago
To implement the serializable isolation level, the database system must track access to every single row you access, even the ones you read or filter out. (The need to track reads, even for rows filtered out of a select statement is surprising but necessary.) Consider a common scenario where you SELECT a set of rows and take a SUM over a column. Suppose your query and another query begin reading from the same committed state of the same table. Suppose that the other query uses an UPDATE command on a set of rows in the table, and that the other query commits before yours does. In order to be consistent, the database system must detect the situation where the other query updated a row that affects the filter WHERE you scanned the table, otherwise your sum could be incorrect if the other query's committed state would cause your query to compute a SUM over a different set of rows or over modified values in your SUM. The only way for the database system to guarantee there is no conflict is to keep track of every single row your query accesses, even if it is a row passed over by a WHERE clause! Serializability is a well studied concept. You can find lots of good resources about algorithms for implementing concurrency control and for detecting whether or not two transactions are serializable. The high-level summary is that it takes a lot of operational bookkeeping to guarantee that two queries have no conflicts, especially when you are using real-world examples having many filters and joins.
- magicalhippo 7y ago
  > The only way for the database system to guarantee there is no conflict is to keep track of every single row your query accesses, even if it is a row passed over by a WHERE clause! Why is putting a lock on the table itself not an option?