12 ms·

Clarification on “Call Me Maybe: MariaDB Galera Cluster”

62 points by crivabene 11y ago

bakhy 11y ago
doesn't this "solution" here actually still leave wide open the possibility of reading inconsistent data?
wglb 11y ago
How is inconsistent different than just plain wrong? This may be another example of reactions to Aphyr's reports telling about the mental model of the developers.
creshal 11y ago
> If you use this in a real life, the more obvious way to write these transactions is: Is it? Do ORMs really do that, or is it one of those "SQL was designed to be used this way, but nobody using SQL read the design documents" cases?
- bsaul 11y ago
  Good point, but imho anyone that relies on transactionnal properties to ensure validity of its operation should really not be using ORMs ( at least not for the sensitive operations). You shouldn't need to read an ORM documentation to understand what kind of locking is happening at a given time. It should all be there right in your code.
  - ealexhudson 11y ago
    This approach only works if your logic is codable in SQL format in the way this trivial balance change can be. If the change is happening in app business logic, this is always going to be a problem. The fact the balance+=25 type approach works is kind of a hack, and I don't think Aphyr is wrong to call it "corrupted" (in this specific example, in one transaction, you could say that it is inconsistent - but you only need a few inconsistent operations in the same place and suddenly your data is complete rubbish). This feels a bit like handwaving the problem away, to me. Yes, InnoDB has the problem, but with that backend you can set SERIALIZABLE isolation. You can't do that in Galera. So if you don't have the option of rewriting the SQL (either because it's not your SQL, or there is no SQL alternative available) then you're screwed.
    - bsaul 11y ago
      Sure, my remark wasn't meant to dismiss aphyr blog. I completely agree that there is a big problem, and calling it corrupted seems valid to me as well. It was more of a sidenote about orms and transactions in general.
- deleted 11y ago
  [deleted]
- viraptor 11y ago
  They don't do that automatically. In sqlalchemy for example you still have to call an extra function explicitly. (http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_09.html#new-for-update-support-on-select-query http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_09...) I know of only one developer who knew about "for update", but then again they just read the mysql book with chapter dedicated to locking behaviour.
- needusername 11y ago
  > Do ORMs really do that I haven't seen any, but they do use optimistic locking.
  - otis_inf 11y ago
    (Disclaimer: I write ORMs for a living) ORMs might issue lock hints but in general they simply start the transaction with the desired isolation level and let the RDBMS take care of the locking of affected rows/tables by statements executed. Lock hints are sometimes used by naive ORMs who think you can implement optimistic concurrency using row locking. Their users will find out eventually that this whole row-locking doesn't help anyone as it simply reverses the tables on 'first write wins' vs. 'last write wins': there's still someone who loses, so the locking doesn't solve the problem of no-one losing their changes while it does give slower performance (and on SQL Server even the risk of deadlocks). What developers often overlook is that there are two types of transactions: business transactions and DB transactions. A business transaction can span multiple DB transactions and a DB transaction can span multiple SQL statements. They're not the same, seeing a business transaction as equal to a DB transaction makes things get messed up and often gives food to the need of explicit lock hints for some queries to e.g. get the false sense of being able to implement optimistic concurrency. If you consider a business transaction, locking doesn't make any sense: it can take some time to complete it, so you have to deal with the side effects of stale data: it immediately becomes apparent that e.g. optimistic concurrency has no place here, one needs other ways to avoid people overwriting work of each other. TFA re-orders statements to get the desired locks in place, and it IS possible to do so, e.g. some ORMs offer when to start a transaction or implement Unit of works which allow you to specify which batches to execute first (e.g. first deletes, then inserts). However the developer using the ORM isn't working at that abstraction level, as the ORM offers an abstraction level above all that; so re-ordering statements to get the desired read locks or avoid dirty reads within a transaction is abusing knowledge of how the abstraction offered by the ORM internally works. IMHO one shouldn't do that: the code utilizing the ORM is db agnostic: expecting certain DB behavior in DB agnostic code is going to give unpleasant side effects when the DB agnostic code will be used to e.g. utilize another RDBMS instead of the current one.
- dack 11y ago
  I think in general when writing applications, you should assume that any data you keep in memory between SQL queries could become stale and change before the next query/update. Anytime you have to update multiple individual records and rely on calculating state from both of them at once, alarm bells should start going off. Yes, you can start dropping into special transactions but there's also a potential opportunity for a better design. Unfortunately, all that feels like a huge accidental complexity.
  - acveilleux 11y ago
    To some extent, isn't that the whole point of specifying the transaction isolation level you need? So you can make these assumptions?
    - dack 11y ago
      Sure, but there are tradeoffs in performance then. I'm not saying transactions are bad (although maybe my strong wording in the parent implies that), just that, especially if you're using an ORM, you shouldn't make those assumptions by default.
      - rcthompson 11y ago
        The only assumption that was made is that Galera cluster's SNAPSHOT ISOLATION actually implements SNAPSHOT ISOLATION. How is this not an assumption that should be made by default? If the database user has specified a specific isolation level, then they have already selected a specific trade-off in performance vs consistency. Giving them a weaker isolation level in the name of performance is like giving someone a car when they asked for a train ticket.
  - bboreham 11y ago
    > Yes, you can start dropping into special transactions Transactions aren't "special" in SQL. You expect that reads and writes within a transaction are kept consistent, unless you have deliberately chosen a weaker serialization level.
- numbsafari 11y ago
  The point of SNAPSHOT ISOLATION is to, essentially, implement optimistic concurrency in a transparent fashion. The fact the the OP recommends using lock manipulation, shows that he's already ceded the point about SNAPSHOT ISOLATION. That is, the whole point of supporting SI is to avoid lock twiddling and the associated complexity and overhead. It's pretty clear that the OP is missing aphyr's point entirely.
tptacek 11y ago
I do not quite like the usage of the word “corrupted” here. For me, the more correct word be to use is “inconsistent”. Aren't we talking about situations in which a database tracking account balances creates money out of thin air, or vaporizes it unexpectedly? I feel like Aphyr is always at pains to talk about the real-world implications of these findings --- not just how bad they are in sensitive applications, but also the kinds of places you can get away with these "inconsistencies".
- j_s 11y ago
  These well, actually[1] moments are great. My favorite ever seen here was yesterday: Not to be pedantic, but I think you are using the word accurate when you mean precise.[2] Communicating with just the written word is tough enough; fighting over which near-synonym to use is always fun to watch but rarely productive. [1] http://tirania.org/blog/archive/2011/Feb-17.html http://tirania.org/blog/archive/2011/Feb-17.html [2] https://news.ycombinator.com/item?id=10237199 https://news.ycombinator.com/item?id=10237199
- noja 11y ago
  It is an inconsistency, and inconsistencies are very bad. Calling it something else won't help.
- michaelt 11y ago
  A database maker would say it isn't the database that created the money out of thin air, but the faulty application code that didn't select the right isolation level. Of course, whether it's wise for a database to default to anything except Serializable isolation is another matter.
  - pricechild 11y ago
    > SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE doesn't get much clearer?
  - lhc- 11y ago
    The database maker's docs have said that their database provides Snapshot Isolation though, which suggests the user should be able to do what Aphyr did without a problem. That's pretty clearly not true, which means it isn't faulty application code. It is the database maker misleading the user.
sagichmal 11y ago
> Following that conclusion is using Galera cluster may result in “corrupted” data. I do not quite like the usage of the word “corrupted” here. For me, the more correct word be to use is “inconsistent”. But Aphyr never once uses the term "corrupted data", or the word "corrupted". If you're going to quote an article, it's important to be precise. This response feels panicked, or at least rushed. And it really misses the point. You can't, or shouldn't, try to explain away these types of findings as irrelevant, or just an issue of semantics. Instead, I'd hope by now that technical folks on the receiving end of a Call Me Maybe analysis would have learned that there's precisely one correct way to respond: acknowledge the faults, clarify relevant documentation, and file (and link to) issues in public issue-trackers that will address the problems. HashiCorp, CoreOS, and arguably Elastic played it correctly. Aerospike, Mesos, and now Percona, didn't. Shame.
- ldite 11y ago
  Yes; it's got a bunch of typos and a subtext of barely concealed rage. (e.g. This is a good opportunity for Asphyr to start another FUD “Call Me Maybe: InnoDB”).
- mbrutsch 11y ago
  > But Aphyr never once uses the term "corrupted data", or the word "corrupted". If you're going to quote an article, it's important to be precise. As opposed to "pedantic"? Here's a quote from the article: > The probability of data corruption scales with client concurrency, with the duration of transactions, and with the increased probability of intersecting working sets.
- hobofan 11y ago
  Chronos, not Mesos. I still wonder why Chronos was even tested. To break Chronos you don't need a network partition, you just have to let it run for a day.
AlisdairO 11y ago
Edit: ignore this, it's addressing a completely different situation, and I clearly didn't read the article well enough. The code in the article writes all of the locations it reads, so true SI ought to keep you safe. My apologies! Yet another edit: huh, it seems that InnoDB in RR doesn't rollback when you write to a row that's been written since you started the transaction. TIL. -------- It's worth noting here that (AFAIR) Oracle's 'SERIALIZABLE' (actually SI) level suffers from this exact write skew vulnerability, so MySQL/MariaDB is not alone in this issue. As pointed out, SELECT FOR UPDATE is a commonly used remedy. What it comes down to is that while re-reading data in a given transaction under SI will give you the same result, it doesn't guarantee that the data in the DB itself has stayed constant. If you want to guarantee that the data won't change, you need to lock it. IIRC this also applies to PostgreSQL's REPEATABLE READ level.
- inversionOf 11y ago
  it doesn't guarantee that the data in the DB itself has stayed constant By definition, snapshot isolation is supposed to guarantee that all reads are the consistent, committed data as of the begin transaction, and the commit will fail and rollback if any data altered within a snapshot isolation transaction was already changed. The response by Percona, unless I am reading it wrong, actually agrees that they are not truly implementing snapshot isolation. They then argue that the tester should have tested something totally different, but that doesn't address the snapshot isolation not actually being snapshot isolation. As to locks, snapshot isolation/MVCC are there to avoid a sea of locks. The solution to a broken snapshot isolation level isn't simply to manually and programmatically demand locks. While that may work, it completely undermines the whole reason for SI.
  - AlisdairO 11y ago
    EDIT: per grandparent comment edit, this comment is based on a misreading of the article and should be ignored. My apologies! ------- > By definition, snapshot isolation is supposed to guarantee that all reads are the consistent, committed data as of the begin transaction, and the commit will fail and rollback if any data altered within a snapshot isolation transaction was already changed Absolutely. What it doesn't guarantee is that data that you read and then use to update a different location has remained constant - which is what write skew is in the first place. What InnoDB and PostgreSQL call 'REPEATABLE READ' and Oracle calls SERIALIZABLE are in fact snapshot isolation. edit: to clarify, Aphyr's original article describes RR as preventing basic write skew, which mainstream MVCC-based RR implementations simply don't do. Postgres does prevent it when using SSI (the SERIALIZABLE level), and lock-based implementations (RR on DB2 and SQL Server) do also. You can argue this both ways: The MVCC-based RR implementations do conform to the (fuzzy) letter of the ANSI SQL standard law. They don't conform to Adya's formal definitions, but in fairness many of them existed before those formalisations did :-).
moomin 11y ago
I think he's trying to define "corrupted" data as completely inaccessible data (because of file corruption, presumably). However, if I zip up a picture of PaulGraham.jpg and when I unzip it I get one of Iron Man, I think I'd be within my rights to call the data corrupt.
MichaelGG 11y ago
Am I missing something or does this not address the main issue the original article raised: The documentation is simply incorrect. It claims to support SNAPSHOT ISOLATION but does not. The company knows this and even this article says the behaviour "is totally expected". Seems like the first response should be to fix the docs and not claim capabilities beyond what's implemented. (Also it was pretty clear from the original article that corrupted data meant the balances were incorrect, not that the file was corrupted like a bad checksum.)
- numbsafari 11y ago
  Yeah, it's pretty clear that OP missed the point, especially in that his recommended solution is to use lock hints. They really should just update their documentation.
- fipar 11y ago
  Disclaimer: I work for Percona. The docs are on the galeracluster.com page, which is owned and maintained by another company, so there's no way we could fix those. A staff member from this company (And one of the Galera authors) replied on the original 'Call me maybe' post indicating they would fix the docs, though. I think the 'corruption vs inconsistency' debate could seem as nitpicking, but anybody who has been working long enough on databases has a very specific concept for each word, and, given transaction processing (distributed or otherwise) is such a complex topic, it does not help to use the wrong terminology.
  - MichaelGG 11y ago
    Perhaps update the article to make a note that you don't control that site's docs? It wasn't immediately apparent that there's yet another party involved. I'd suggest that the tone of the article be very clear that the problem described is accurate and isn't trying to be downplayed. Suggesting that Aphyr write another "FUD" article comes across as dismissive of the issue. And that'll make people read more into the corruption/inconsistent comment. (Unfair as it may be.)
  - sergiosgc 11y ago
infinotize 11y ago
Besides being a super well-written and interesting series technically, the Call Me Maybe blogs have been very revealing as far as different organizations' response to criticism. Especially considering all of the target applications are open source, the project maintainers should be profusely thankful someone has taken the time for such thorough analysis, presumably much deeper than the maintainers themselves appear to have done at least on consistency behavior, to reveal bugs which should ultimately make it that much stronger.
- dbarlett 11y ago
  I really like jerf's take on it [1]: How a project performs today tells you the zeroth derivative of its location. Looking at the commit log tells you about the first derivative. How people react to Call Me Maybe when its about their product gives you a lot of information about the second derivative. [1] https://news.ycombinator.com/item?id=10082099 https://news.ycombinator.com/item?id=10082099 Edit: username
  - seiji 11y ago
    A common response to these has been: "But you're using it wrong!!!!" If your product is so complex or so ill-specified that full time testers can't make it work correctly, it's probably difficult for your developers to even understand or fix the problems. We ignore how much of software development has become "it works for me under all of my default assumptions as the developer of the product—ship it," especially when faced with business deadlines and business management focused around business objectives (and maybe not so much around software quality or correctness).
  - jerf 11y ago
    DoubleTakeException: Unexpected pointer dereference in dbartlett's post on line 0: got "Jeremy Bowers'", expected "jerf's" Man that's a weird thing to see unexpectedly....
  - cwyers 11y ago
    Something of a quibble here -- MariaDB with Galera Cluster is not Percona's product. Galera Cluster is from Codership and MariaDB is the MariaDB Foundation. Percona does resell Galera Cluster, so they're not an impartial third party, but it's funny to me to see people in this thread asking why Percona isn't fixing the documentation for Codership's product.
PeterZaitsev 11y ago
There are many approaches to data consistency, including for example Eventual consistency, not only in databases but in any parallel programming. Different CPU architectures for example for years provided different memory consistency models with trade-offs of performance and being usable for application programmers. It is important however behavior in this case is clearly documented. I think there is decent documentation about Innodb describing how transactions work in Innodb http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-model.html http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-mo... Galera would benefit having more clear documentation about what data consistency model exactly it provides. At the same time Percona XtraDB Cluster, MariaDB Cluster can be used to built reliable applications assuming you're writing to their consistency model correctly.
deleted 11y ago
[deleted]
rushi_agrawal 11y ago
Below are the tweets by Aphyr for the same thing (with language slightly toned down). I never thought I won't find their mention here on HN. :) Nevertheless, precise, to the point: Buncha people giving me <filth> for calling data written through an invariant violation "corrupted state", like somehow it's not garbage. If TCP checksums don't work right we don't call the packet "inconsistent." We call it corrupt. If a disk shuffles your file's bits? Corrupt. I use the word corrupt to emphasize that not only has the system <messed> up, but you have no way to know your data is now <messed> up.
- dsp1234 11y ago
  https://twitter.com/aphyr/status/644828700366647296 https://twitter.com/aphyr/status/644828700366647296
- inyourtenement 11y ago
  I don't think you need to censor quotes on HN.
eis 11y ago
A lot of defensive talking around technical terms, mixed with a bunch of typos and topped off with unfair attacks. Not very classy. And the point of Aphyr still stands I think. In default mode it is easy to get corrupt data with Galera Cluster. That InnoDB on a single instance can have the same problem makes it all the more troubling and I'm glad I moved away from MySQL a long time ago.

62 points by crivabene 11y ago

bakhy 11y ago

doesn't this "solution" here actually still leave wide open the possibility of reading inconsistent data?

wglb 11y ago

How is inconsistent different than just plain wrong? This may be another example of reactions to Aphyr's reports telling about the mental model of the developers.

creshal 11y ago

> If you use this in a real life, the more obvious way to write these transactions is: Is it? Do ORMs really do that, or is it one of those "SQL was designed to be used this way, but nobody using SQL read the design documents" cases?

bsaul 11y ago
Good point, but imho anyone that relies on transactionnal properties to ensure validity of its operation should really not be using ORMs ( at least not for the sensitive operations). You shouldn't need to read an ORM documentation to understand what kind of locking is happening at a given time. It should all be there right in your code.
- ealexhudson 11y ago
  This approach only works if your logic is codable in SQL format in the way this trivial balance change can be. If the change is happening in app business logic, this is always going to be a problem. The fact the balance+=25 type approach works is kind of a hack, and I don't think Aphyr is wrong to call it "corrupted" (in this specific example, in one transaction, you could say that it is inconsistent - but you only need a few inconsistent operations in the same place and suddenly your data is complete rubbish). This feels a bit like handwaving the problem away, to me. Yes, InnoDB has the problem, but with that backend you can set SERIALIZABLE isolation. You can't do that in Galera. So if you don't have the option of rewriting the SQL (either because it's not your SQL, or there is no SQL alternative available) then you're screwed.
  - bsaul 11y ago
    Sure, my remark wasn't meant to dismiss aphyr blog. I completely agree that there is a big problem, and calling it corrupted seems valid to me as well. It was more of a sidenote about orms and transactions in general.
deleted 11y ago
[deleted]
viraptor 11y ago
They don't do that automatically. In sqlalchemy for example you still have to call an extra function explicitly. (http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_09.html#new-for-update-support-on-select-query http://docs.sqlalchemy.org/en/rel_0_9/changelog/migration_09...) I know of only one developer who knew about "for update", but then again they just read the mysql book with chapter dedicated to locking behaviour.
needusername 11y ago
> Do ORMs really do that I haven't seen any, but they do use optimistic locking.
- otis_inf 11y ago
  (Disclaimer: I write ORMs for a living) ORMs might issue lock hints but in general they simply start the transaction with the desired isolation level and let the RDBMS take care of the locking of affected rows/tables by statements executed. Lock hints are sometimes used by naive ORMs who think you can implement optimistic concurrency using row locking. Their users will find out eventually that this whole row-locking doesn't help anyone as it simply reverses the tables on 'first write wins' vs. 'last write wins': there's still someone who loses, so the locking doesn't solve the problem of no-one losing their changes while it does give slower performance (and on SQL Server even the risk of deadlocks). What developers often overlook is that there are two types of transactions: business transactions and DB transactions. A business transaction can span multiple DB transactions and a DB transaction can span multiple SQL statements. They're not the same, seeing a business transaction as equal to a DB transaction makes things get messed up and often gives food to the need of explicit lock hints for some queries to e.g. get the false sense of being able to implement optimistic concurrency. If you consider a business transaction, locking doesn't make any sense: it can take some time to complete it, so you have to deal with the side effects of stale data: it immediately becomes apparent that e.g. optimistic concurrency has no place here, one needs other ways to avoid people overwriting work of each other. TFA re-orders statements to get the desired locks in place, and it IS possible to do so, e.g. some ORMs offer when to start a transaction or implement Unit of works which allow you to specify which batches to execute first (e.g. first deletes, then inserts). However the developer using the ORM isn't working at that abstraction level, as the ORM offers an abstraction level above all that; so re-ordering statements to get the desired read locks or avoid dirty reads within a transaction is abusing knowledge of how the abstraction offered by the ORM internally works. IMHO one shouldn't do that: the code utilizing the ORM is db agnostic: expecting certain DB behavior in DB agnostic code is going to give unpleasant side effects when the DB agnostic code will be used to e.g. utilize another RDBMS instead of the current one.
dack 11y ago
I think in general when writing applications, you should assume that any data you keep in memory between SQL queries could become stale and change before the next query/update. Anytime you have to update multiple individual records and rely on calculating state from both of them at once, alarm bells should start going off. Yes, you can start dropping into special transactions but there's also a potential opportunity for a better design. Unfortunately, all that feels like a huge accidental complexity.
- acveilleux 11y ago
  To some extent, isn't that the whole point of specifying the transaction isolation level you need? So you can make these assumptions?
  - dack 11y ago
    Sure, but there are tradeoffs in performance then. I'm not saying transactions are bad (although maybe my strong wording in the parent implies that), just that, especially if you're using an ORM, you shouldn't make those assumptions by default.
    - rcthompson 11y ago
      The only assumption that was made is that Galera cluster's SNAPSHOT ISOLATION actually implements SNAPSHOT ISOLATION. How is this not an assumption that should be made by default? If the database user has specified a specific isolation level, then they have already selected a specific trade-off in performance vs consistency. Giving them a weaker isolation level in the name of performance is like giving someone a car when they asked for a train ticket.
- bboreham 11y ago
  > Yes, you can start dropping into special transactions Transactions aren't "special" in SQL. You expect that reads and writes within a transaction are kept consistent, unless you have deliberately chosen a weaker serialization level.
numbsafari 11y ago
The point of SNAPSHOT ISOLATION is to, essentially, implement optimistic concurrency in a transparent fashion. The fact the the OP recommends using lock manipulation, shows that he's already ceded the point about SNAPSHOT ISOLATION. That is, the whole point of supporting SI is to avoid lock twiddling and the associated complexity and overhead. It's pretty clear that the OP is missing aphyr's point entirely.

tptacek 11y ago

I do not quite like the usage of the word “corrupted” here. For me, the more correct word be to use is “inconsistent”. Aren't we talking about situations in which a database tracking account balances creates money out of thin air, or vaporizes it unexpectedly? I feel like Aphyr is always at pains to talk about the real-world implications of these findings --- not just how bad they are in sensitive applications, but also the kinds of places you can get away with these "inconsistencies".

j_s 11y ago
These well, actually[1] moments are great. My favorite ever seen here was yesterday: Not to be pedantic, but I think you are using the word accurate when you mean precise.[2] Communicating with just the written word is tough enough; fighting over which near-synonym to use is always fun to watch but rarely productive. [1] http://tirania.org/blog/archive/2011/Feb-17.html http://tirania.org/blog/archive/2011/Feb-17.html [2] https://news.ycombinator.com/item?id=10237199 https://news.ycombinator.com/item?id=10237199
noja 11y ago
It is an inconsistency, and inconsistencies are very bad. Calling it something else won't help.
michaelt 11y ago
A database maker would say it isn't the database that created the money out of thin air, but the faulty application code that didn't select the right isolation level. Of course, whether it's wise for a database to default to anything except Serializable isolation is another matter.
- pricechild 11y ago
  > SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE doesn't get much clearer?
- lhc- 11y ago
  The database maker's docs have said that their database provides Snapshot Isolation though, which suggests the user should be able to do what Aphyr did without a problem. That's pretty clearly not true, which means it isn't faulty application code. It is the database maker misleading the user.

sagichmal 11y ago

> Following that conclusion is using Galera cluster may result in “corrupted” data. I do not quite like the usage of the word “corrupted” here. For me, the more correct word be to use is “inconsistent”. But Aphyr never once uses the term "corrupted data", or the word "corrupted". If you're going to quote an article, it's important to be precise. This response feels panicked, or at least rushed. And it really misses the point. You can't, or shouldn't, try to explain away these types of findings as irrelevant, or just an issue of semantics. Instead, I'd hope by now that technical folks on the receiving end of a Call Me Maybe analysis would have learned that there's precisely one correct way to respond: acknowledge the faults, clarify relevant documentation, and file (and link to) issues in public issue-trackers that will address the problems. HashiCorp, CoreOS, and arguably Elastic played it correctly. Aerospike, Mesos, and now Percona, didn't. Shame.

ldite 11y ago
Yes; it's got a bunch of typos and a subtext of barely concealed rage. (e.g. This is a good opportunity for Asphyr to start another FUD “Call Me Maybe: InnoDB”).
mbrutsch 11y ago
> But Aphyr never once uses the term "corrupted data", or the word "corrupted". If you're going to quote an article, it's important to be precise. As opposed to "pedantic"? Here's a quote from the article: > The probability of data corruption scales with client concurrency, with the duration of transactions, and with the increased probability of intersecting working sets.
hobofan 11y ago
Chronos, not Mesos. I still wonder why Chronos was even tested. To break Chronos you don't need a network partition, you just have to let it run for a day.

AlisdairO 11y ago

Edit: ignore this, it's addressing a completely different situation, and I clearly didn't read the article well enough. The code in the article writes all of the locations it reads, so true SI ought to keep you safe. My apologies! Yet another edit: huh, it seems that InnoDB in RR doesn't rollback when you write to a row that's been written since you started the transaction. TIL. -------- It's worth noting here that (AFAIR) Oracle's 'SERIALIZABLE' (actually SI) level suffers from this exact write skew vulnerability, so MySQL/MariaDB is not alone in this issue. As pointed out, SELECT FOR UPDATE is a commonly used remedy. What it comes down to is that while re-reading data in a given transaction under SI will give you the same result, it doesn't guarantee that the data in the DB itself has stayed constant. If you want to guarantee that the data won't change, you need to lock it. IIRC this also applies to PostgreSQL's REPEATABLE READ level.

inversionOf 11y ago
it doesn't guarantee that the data in the DB itself has stayed constant By definition, snapshot isolation is supposed to guarantee that all reads are the consistent, committed data as of the begin transaction, and the commit will fail and rollback if any data altered within a snapshot isolation transaction was already changed. The response by Percona, unless I am reading it wrong, actually agrees that they are not truly implementing snapshot isolation. They then argue that the tester should have tested something totally different, but that doesn't address the snapshot isolation not actually being snapshot isolation. As to locks, snapshot isolation/MVCC are there to avoid a sea of locks. The solution to a broken snapshot isolation level isn't simply to manually and programmatically demand locks. While that may work, it completely undermines the whole reason for SI.
- AlisdairO 11y ago
  EDIT: per grandparent comment edit, this comment is based on a misreading of the article and should be ignored. My apologies! ------- > By definition, snapshot isolation is supposed to guarantee that all reads are the consistent, committed data as of the begin transaction, and the commit will fail and rollback if any data altered within a snapshot isolation transaction was already changed Absolutely. What it doesn't guarantee is that data that you read and then use to update a different location has remained constant - which is what write skew is in the first place. What InnoDB and PostgreSQL call 'REPEATABLE READ' and Oracle calls SERIALIZABLE are in fact snapshot isolation. edit: to clarify, Aphyr's original article describes RR as preventing basic write skew, which mainstream MVCC-based RR implementations simply don't do. Postgres does prevent it when using SSI (the SERIALIZABLE level), and lock-based implementations (RR on DB2 and SQL Server) do also. You can argue this both ways: The MVCC-based RR implementations do conform to the (fuzzy) letter of the ANSI SQL standard law. They don't conform to Adya's formal definitions, but in fairness many of them existed before those formalisations did :-).

moomin 11y ago

I think he's trying to define "corrupted" data as completely inaccessible data (because of file corruption, presumably). However, if I zip up a picture of PaulGraham.jpg and when I unzip it I get one of Iron Man, I think I'd be within my rights to call the data corrupt.

MichaelGG 11y ago

Am I missing something or does this not address the main issue the original article raised: The documentation is simply incorrect. It claims to support SNAPSHOT ISOLATION but does not. The company knows this and even this article says the behaviour "is totally expected". Seems like the first response should be to fix the docs and not claim capabilities beyond what's implemented. (Also it was pretty clear from the original article that corrupted data meant the balances were incorrect, not that the file was corrupted like a bad checksum.)

numbsafari 11y ago
Yeah, it's pretty clear that OP missed the point, especially in that his recommended solution is to use lock hints. They really should just update their documentation.
fipar 11y ago
Disclaimer: I work for Percona. The docs are on the galeracluster.com page, which is owned and maintained by another company, so there's no way we could fix those. A staff member from this company (And one of the Galera authors) replied on the original 'Call me maybe' post indicating they would fix the docs, though. I think the 'corruption vs inconsistency' debate could seem as nitpicking, but anybody who has been working long enough on databases has a very specific concept for each word, and, given transaction processing (distributed or otherwise) is such a complex topic, it does not help to use the wrong terminology.
- MichaelGG 11y ago
  Perhaps update the article to make a note that you don't control that site's docs? It wasn't immediately apparent that there's yet another party involved. I'd suggest that the tone of the article be very clear that the problem described is accurate and isn't trying to be downplayed. Suggesting that Aphyr write another "FUD" article comes across as dismissive of the issue. And that'll make people read more into the corruption/inconsistent comment. (Unfair as it may be.)
- sergiosgc 11y ago

infinotize 11y ago

Besides being a super well-written and interesting series technically, the Call Me Maybe blogs have been very revealing as far as different organizations' response to criticism. Especially considering all of the target applications are open source, the project maintainers should be profusely thankful someone has taken the time for such thorough analysis, presumably much deeper than the maintainers themselves appear to have done at least on consistency behavior, to reveal bugs which should ultimately make it that much stronger.

dbarlett 11y ago
I really like jerf's take on it [1]: How a project performs today tells you the zeroth derivative of its location. Looking at the commit log tells you about the first derivative. How people react to Call Me Maybe when its about their product gives you a lot of information about the second derivative. [1] https://news.ycombinator.com/item?id=10082099 https://news.ycombinator.com/item?id=10082099 Edit: username
- seiji 11y ago
  A common response to these has been: "But you're using it wrong!!!!" If your product is so complex or so ill-specified that full time testers can't make it work correctly, it's probably difficult for your developers to even understand or fix the problems. We ignore how much of software development has become "it works for me under all of my default assumptions as the developer of the product—ship it," especially when faced with business deadlines and business management focused around business objectives (and maybe not so much around software quality or correctness).
- jerf 11y ago
  DoubleTakeException: Unexpected pointer dereference in dbartlett's post on line 0: got "Jeremy Bowers'", expected "jerf's" Man that's a weird thing to see unexpectedly....
- cwyers 11y ago
  Something of a quibble here -- MariaDB with Galera Cluster is not Percona's product. Galera Cluster is from Codership and MariaDB is the MariaDB Foundation. Percona does resell Galera Cluster, so they're not an impartial third party, but it's funny to me to see people in this thread asking why Percona isn't fixing the documentation for Codership's product.

PeterZaitsev 11y ago

There are many approaches to data consistency, including for example Eventual consistency, not only in databases but in any parallel programming. Different CPU architectures for example for years provided different memory consistency models with trade-offs of performance and being usable for application programmers. It is important however behavior in this case is clearly documented. I think there is decent documentation about Innodb describing how transactions work in Innodb http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-model.html http://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-mo... Galera would benefit having more clear documentation about what data consistency model exactly it provides. At the same time Percona XtraDB Cluster, MariaDB Cluster can be used to built reliable applications assuming you're writing to their consistency model correctly.

deleted 11y ago

[deleted]

rushi_agrawal 11y ago

Below are the tweets by Aphyr for the same thing (with language slightly toned down). I never thought I won't find their mention here on HN. :) Nevertheless, precise, to the point: Buncha people giving me <filth> for calling data written through an invariant violation "corrupted state", like somehow it's not garbage. If TCP checksums don't work right we don't call the packet "inconsistent." We call it corrupt. If a disk shuffles your file's bits? Corrupt. I use the word corrupt to emphasize that not only has the system <messed> up, but you have no way to know your data is now <messed> up.

dsp1234 11y ago
https://twitter.com/aphyr/status/644828700366647296 https://twitter.com/aphyr/status/644828700366647296
inyourtenement 11y ago
I don't think you need to censor quotes on HN.

eis 11y ago

A lot of defensive talking around technical terms, mixed with a bunch of typos and topped off with unfair attacks. Not very classy. And the point of Aphyr still stands I think. In default mode it is easy to get corrupt data with Galera Cluster. That InnoDB on a single instance can have the same problem makes it all the more troubling and I'm glad I moved away from MySQL a long time ago.