8 ms·

NoSQL Meets Bitcoin and Brings Down Two Exchanges

103 points by silenteh 13y ago

bobx11 13y ago
This article was only written to promote something called HyperDex which he is in some way involved.
adambard 13y ago
TLDR: Exactly what you might have guessed: the exchanges backed their financial operations with a datastore without transactions, tragedy ensues. This can't be viewed as an indictment of "NoSQL" though; there are plenty of databases featuring ACID transactions that qualify as "NoSQL" (FoundationDB and OrientDB come to mind). That said, none of the popular NoSQL contenders have them, and the new guys with transactions tend to be commercial, so if you need transactions in an free and open-source datastore, just use PostgreSQL.
nemothekid 13y ago
While I do recognize this is a marketing piece for HyperDex, I'm disappointed HyperDex only chooses to compare itself with MongoDB and Cassandra. It would be different if HyperDex was fully open source and free, but its not. The "transaction" feature is only available with a license, which in most cases will price you out of any developer who is looking to start a bitcoin exchange. So unfortunately it comes across as "If only you had spent $x/mo, buying our product, then you wouldn't have had this problem!" Now that isn't bad, and its the definition of marketing, but then it goes on to compare itself against Cassandra (which has a widely different use case in the wild) and Mongo (which is already the whipping boy of the database community). However it doesn't help the use cases are also extremely well served by something like Postgres, which doesn't need a Warp addon, and has a huge developer community. What I would be more interested in, is how does HyperDex compare with the new "NoSQL" databases (FoundationDB, OrientDB, Spanner-like Databases) or even the traditional commercial SQL like Oracle. In short, it doesn't help to test and compare unsuccessful and unpopular use cases with these technologies. Most competent people, the same people who might pay for HyperDex, already know Cassandra is bad at transactions. What I would like to see is tell me why someone like Netflix, or Ooyala (tried, tested and popular use cases) would move from Cassandra to HyperDex.
- threeseed 13y ago
  Cassandra's transaction support has been improving pretty rapidly. http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0 http://www.datastax.com/dev/blog/lightweight-transactions-in...
  - grizzles 13y ago
    For what I can only guess are strategic reasons, Datastax is moving away from the schema-less design that made Cassandra popular with early adopters. There are workarounds to this deprecation such as using the old Thrift API, but it doesn't really inspire confidence in the product unless the new schema oriented way of doing things appeals to you. PS those aren't transactions.
    - teacup50 13y ago
      "Schema-less" is a hack that appeals to people that don't realize that there is always schema. It's just a question of whether you want your data model to be ambiguous and poorly defined, or explicit.
      - nevi-me 13y ago
        Schema-less also means that you can afford to have deviations in your schemas not hinder your development needs. I agree with you, but I'd venture to say that a schema-less database means what it does, but that a collection of data always has schemas.
        zAy0LfpBZLC8mAC 13y ago
        No, the schema is always a property of the system as a whole, data alone doesn't have an implicit schema. The schema is the set of type constraints that data has to conform to so that it can be processed by the system. Thus, a DB being "schema-less" simply means that it won't check/enforce those constraints - the schema still is there, of course, simply because a given piece of software that does anything meaningful will not sensibly process any random piece of garbage you feed it.
teawithcarl 13y ago
Cornell CS professor, superb work. See also -- http://hackingdistributed.com/2014/03/01/what-did-not-happen-at-mtgox/ http://hackingdistributed.com/2014/03/01/what-did-not-happen... "What may or may not have happened at Mt. Gox"
patio11 13y ago
That's because banks employ systems that guard against this kind of elementary error. It's called transactions with ACID guarantees. Bank systems are often not, in fact, ACID. Think of how many billions of transactions are processed in end-of-the-day batch runs, for example. Overdrafting is not only not impossible, it is a ludicrously profitable feature of the system. How is this secure? Velocity limits, sophisticated anti-fraud systems, acceptable losses, and, as a bit of a by-the-way, the fact that people who make a habit of defrauding banks generally have a few weeks prior to being woken up by very serious men with guns.
- rdtsc 13y ago
  It is interesting because the transaction log inconsistencies can often be resolved technically and because of the regulatory framework. Overdrafts can be "solved" in business sense. Now imagine something like an bitcoin exchange and bitcoin transaction log. Transaction errors cannot be fixed easily. So overdrafting an account means never getting the money back. There a single account would need to have ACID properties in way. It seems ability to deal with and tolerate data conflicts and inconsistencies is different for each business. There was some interest in CRTDs (data types that can automatically converge after a conflict occurs, like imagine a max() function or a set union operation) but still means shortly there could be an inconsistency in the system and sometimes that can be exploited and is just not an option.
- Karellen 13y ago
  Bank systems are ACID. They're often not immediate, using batch runs as you say, but the databases the batch runs work on are definitely ACID. The problem is not necessarily with overdrafts, as you suggest, but - as the article says - with race conditions between multiple readers/writers causing two near-simultaneous withdrawls to only make a single debit to the source account, while crediting both destination accounts. Note that this could lead to unexpected overdrafts being noticed later, if the exchange also wrote a consistent log of all transactions, and attempted to reconcile the log with the real-time balances periodically (e.g. on a daily basis), but it's not clear if they even did that. (Although I might be missing something)
  - 13y ago
aardvark179 13y ago
"Historically, Bitcoin exchanges that suffered significant losses turned into fractional reserve banks, only to fold later." That is not what fractional reserve banking is and I wish bitcoin articles would stop misusing the term.
- mikeash 13y ago
  The continued misuse of that term seems telling. If they really think this is how normal banks work, it helps explain why they set up these bitcoin systems in such crazy ways.
rdtsc 13y ago
Please don't be MongoDB, please don't be MongoDB ... Sigh > a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks. Here is the thing, distributed systems are not standard academic courses. Just like networking they are often optional. That is really sad and it needs to change. The CAP theorem is not something fringe and exotic anymore it should be standard taught to everyone. MongoDB has really messed up. They promoted and marketed a shitty product wrapped in nice marketing. It created 2 things -- low barrier to entry and a trap. Both wrapped in one nice shiny package. "You should have read the docs to enable actually writing data disk when the user asks them to" doesn't jive with me. A really shitty default setup for something calling itself a database. Oh well, they reaped the benefits of their strategy, both the short term (success) and long term (failure) - it became the butt of jokes and ridicule.
- threeseed 13y ago
  Have you actually used the database since if you did you would know that the write concern was set to FSYNC_SAFE by every single driver (before it was the default) and given the lack of configurability should have been known by every developer. It never ceases to amaze me how few technical people actually "get it" even for products designed for themselves. Many, many people like things to be simple and easy. MongoDB fits this criteria perfectly and no matter how many times you call it a shitty product isn't going to dampen its popularity.
- zorlem 13y ago
  I, too, don't like the way MongoDB works and is marketed, but I would not bet too strongly on them failing. Only 10 years or so ago MySQL did pretty much the same, but they managed to succeed, despite there being better alternatives at the time. Even today, after all the development effort put into MySQL and its derivatives, there are still better and more capable databases, but that doesn't prevent MySQL from being one of the top three RDBMS-es (and we can bet it is the most popular FOSS RDBMS).
ChuckMcM 13y ago
Its unfortunate to use the ATM example because just such an exploit was used by carders [1] to get more money out of accounts than the accounts actually had. [1] http://time.com/48344/hackers-target-atms-for-unlimited-withdrawals/ http://time.com/48344/hackers-target-atms-for-unlimited-with...
- edwintorok 13y ago
  ATMs aren't as smart as the article says either: "the real code will check to see if there are sufficient funds". I noticed that (some?) ATMs around here tend to implement this flawed algorithm, paraphrasing the article: 0. database.begin() 1. mybalance = database.read("account-number") 2. newbalance = mybalance - amount 3. database.write("account-number", newbalance) 4. database.commit() 5. dispense_cash(amount) At first sight this might seem ACID, but its not. Consider what happens if the ATM is out of cash: step#5 fails, but the money was already withdrawn from your account! It doesn't automatically revert the transaction either: you have to go to a branch office and ask for the transaction to be reverted. So much for atomic transactions at ATMs.
  - dnmmnd 13y ago
    Whilst that maybe the way your bank has chosen to operate, it is most certainly possible for ATMs to report a step#5 failure to the banks host which will reverse the transaction.
  - ams6110 13y ago
    Wow I've never seen this happen. What happened to step -1: if (cash_in_machine < amount) abort
    - threeseed 13y ago
      He is talking about a failure to dispense the cash. I've had this happen twice to me whilst overseas.
  - jasomill 13y ago
    Interesting. In one particularly ADHD-inspired case, I've also seen 5. Dispense cash 6. Wait a minute or so 7. Retract cash where 7 did not lead to a credit on my account until several days after I reported the problem to my bank. That's probably by design, however, since otherwise the machine would require automated defenses against surreptitious replacement of dispensed bills with counterfeits. Your example could be a similar failsafe against cases where the machine fails to properly dispense cash in a manner that nevertheless allows the cash to be retrieved. Otherwise you have to carefully separate errors that can't possibly lead to cash being dispensed (e.g., hopper empty) vs. those that might (e.g., bills jammed somewhere in the dispenser mechanism).
hibikir 13y ago
The first problem to be had when considering distributed systems is the fact that there is no such thing as a general purpose distributed system. Some problems require doing 50x more reads than writes. Others have more writes than reads. Sometimes losing a few inserts is not a big deal, but a small delay is terrible. Other systems just can't tolerate data loss, and would pay any price to keep it that way. You might have a lot of small pieces of data, searchable in very simple ways, or you might have multi GB documents that require so much indexing than Solr and Elastic seem inadequate. In the old days, you just bought a bigger box for the RDBMS, and you just were covered by knowing one tech. Now we need to go through a wide variety of tools that will rarely do everything you want, and have to write a bunch of code to compensate for the limitations of the tools. And then your DB of choice decides that they will only support their self hosted product, then they get bought by IBM, and you wonder if you'll be forced to retool your entire backend (Hello Cloudant!) Until there is some clarity in the market, and we get to see distributed stacks that are general purpose, we'll see issues like this popping up all the time, as there is a lack of people that both have good knowledge of all the available options and the skill to put them together into something that will solve your specific problem.
camus2 13y ago
Question! Do transactional NoSQL(others than HyperDex) databases exist? or do transactions make horizontal scaling difficult?
nacs 13y ago
Sounds like the author of this 'article' just wanted to make a thinly veiled plug for the "HyperDex" product. Blaming all of this on MongoDB is silly. You can do the kind of idiocy displayed by these exchanges even on generic DBs by doing things like not using transactions and/or not waiting for write confirmation. The root problem is in the (lack of) design and programming of these systems not their data store.
tlarkworthy 13y ago
you can put ACID like transactions on top of NoSQL databases, I did it here, but it took a static analyser to help get the logic right: https://github.com/tomlarkworthy/firesafe/wiki/Send-Item https://github.com/tomlarkworthy/firesafe/wiki/Send-Item
rubyfan 13y ago
This article lacks a fact base on what actually happened and seems to totally rely on conjecture in order to compare Hyperdex against leading NoSQL solutions.
dhendo 13y ago
Mongo does have document-level atomic update operators, such as $inc: http://docs.mongodb.org/manual/reference/operator/update/inc/#up._S_inc http://docs.mongodb.org/manual/reference/operator/update/inc...
- thom_nic 13y ago
  Yes this was completely ignored by the article. This along with a 'where' clause can guarantee the operation will only succeed iff there is sufficient funds. I don't know if other NoSQL DBs offer these sort of atomic operators. Which makes this statement seem particularly unfair: "The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB." He seems to call out Mongo by name a couple times but doesn't say that the exchange was actually using Mongo.
danbruc 13y ago
It's the same story over and over again. The young wild guys arrive, full of energy and ambition to make the world a better place. They throw all the ancient stuff overboard because it seems ugly and useless. Then they painfully rediscover why the ugly stuff is there and not useless at all. And after a lot of pain they finally arrive at a slightly improved version of what they threw overboard long ago, at least if they did not surrender before.
- PhantomGremlin 13y ago
  > The young wild guys arrive That's exactly the problem. The 20-something hipsters who start these companies don't yet have the life experience to realize how ignorant they are. So of course they don't pay any attention to the lessons of history. And so they, very painfully, re-learn them. If the hipsters had some humility they would have also hired a few grizzled, experienced old fogies. Fortunately not all startups assume that anyone over the age of 30 isn't worth hiring. For a non-technical example, look at how Google hired Eric Schmidt to provide adult supervision.
jokoon 13y ago
What are the advantages of nosql btw ?
- threeseed 13y ago
  Basically developers came for the scalability and stayed because they are so easy to use/manage.
- nikatwork 13y ago
  NoSQL is fantastic for backing item-orientated webapps, it removes the transactional and relational overheads. Conversely I would not use NoSQL for anything concerning FIRE or complex inter-related data - those safeguards become important. Scalability is something of a red herring with NoSQL. Yes it's easier to scale, but my primary criteria for choosing it would be the nature of the data, not the intended scale.
- Consultant32452 13y ago
  A more "use-case" example than your other responses.. I believe Amazon uses a NoSQL database for their product data. It's tons of reads and relatively few writes. They don't give a crap about transaction quality for product descriptions, comments, etc. However, when you get to the checkout page and they start doing things like taking your money and updating their inventory... that's and ACID compliant database. When you're dealing with money you want transactions, rollbacks, etc.
late2part 13y ago
But, it is webscale!
bkirwi 13y ago
A bit irritated at how the author complains about other databases astroturfing on Hacker News, then writes a glowing post about HyperDex without mentioning that he's one of the authors. That said, I agree with the thesis that current-generation distributed systems provide a terrible API for users who actually want to get concurrency right. I'm optimistic about projects like Summingbird[0] and Bloom[1], which run very high-level declarative programs with simple consistency guarantees. [0] https://github.com/twitter/summingbird https://github.com/twitter/summingbird [1] http://www.bloom-lang.net/ http://www.bloom-lang.net/
atmosx 13y ago
> It points to a social failure: a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks. So, will the OP do an exchange and show us how it's done using his database of and approach of choice. Because writing articles on other's failures is kinda of... Easy. I'm a little sick of this attitude. Nothing personal with - what I thought was Sinan Eren at first, confusing the names after so many years - the OP here, but given the fact that he is deliberately calling out amateurs and poor designers people who (apparently) deserved it, he might very well create a secure bitcoin exchange in the real world, using his technologies.. There is a huge market for that, today more than ever.
- sillysaurus3 13y ago
  The attitude is warranted when they're losing other people's money.
  - atmosx 13y ago
    Totally get that, but his approach seems a little snooty to me and runs throughout the article that's why I had this reaction/comment.
DigitalSea 13y ago
As soon as I saw the term NoSQL and "Brings Down" in the title of the submission, I knew it was going to be MongoDB. Firstly, I don't understand the hatred towards MongoDB. Like most NoSQL datastores, it doesn't offer things like transactions and ACID compliance: this is well-known thing. There are some things that MongoDB does very well, currently I am using it to store clicks for an internal application which then displays those clicks on a page overlay, it handles the task very well. I would never use it for MongoDB is not to blame for bringing down the exchanges that decided to use it, the stupidity of the developers behind said exchanges are the sole reason. You can open up a can of tinned pineapples with an axe, but that doesn't make it the right tool for the job. This post was one big marketing piece for HyperDex and while it is touted as open source, but that isn't entirely true. If you want support for transactions, you have to by a licence. So basically what this post was saying is, "If you spent X amount of money on a HyperDex licence, you wouldn't have had these issues" most people that start these exchanges start them from nothing and probably can't afford or wouldn't warrant spending money on a licence. If you replaced MongoDB with pretty much every other NoSQL database out there, the result would have been the same. Mongo just has a better marketing team behind them in the form of 10Gen getting people to buy into the hype. However, as far as I am aware of, 10Gen aren't telling people to use MongoDB for exchange platforms and anything involving transactions (I could be wrong though). This is a classic case of seeing people talking about how NoSQL is the future and others getting caught up in the hype. If you want a database that you can rely on, especially when dealing with transactions: use something like Postgresql, not MongoDB and or insert over-hyped NoSQL database here
- hackerboos 13y ago
  > it doesn't offer things like transactions and ACID compliance: this is well-known thing Evidently not. Which is why people are still using it for financial based systems.
- kalleboo 13y ago
  > the stupidity of the developers behind said exchanges are the sole reason Exactly. The fact that they picked MongoDB for this application in the first place tells me that even if they used an ACID-compliant database, they still wouldn't have known to use transactions, and would be vulnerable to the same attack.
kylemaxwell 13y ago
Not quite understanding the focus on NoSQL for this use case set. I love MongoDB in the right place and Redis in others. But when you have a bunch of requirements that lend themselves to traditional DBs, choosing MongoDB or HyperDex feels like choosing a tech because it's cool rather than because it is the proper engineering choice.
cranklin 13y ago
"They did what anyone would do after reading one too many astroturf articles on Hacker News. Sure, their system failed, but in a sense, the overall system failed them." The technology didn't fail them. On the contrary, the technology did exactly what it was supposed to do. The devs could have (should have) considered these race condition scenarios.
ibyon 13y ago
I don't think that MongoDB or any nosql is right choice for financial application. You can build for yelp but not for banking.
platform 13y ago
Marklogic supports ACID transactions. being that it is also a horizontally scalable system, it allows those transactions not just to spawn multiple objects, but also across multiple hosts. http://www.marklogic.com/blog/can-you-pass-the-acid-test/ http://www.marklogic.com/blog/can-you-pass-the-acid-test/ They also support data tiering into HDFS, and now most recently RDF with SPARQL They have been doing ACID for a number of years, not sure for how long. They are not free (unless your data set is small)
kofejnik 13y ago
Why wouldn't they use a plain old ACID database? Was their load something that Oracle couldn't handle?

103 points by silenteh 13y ago

This article was only written to promote something called HyperDex which he is in some way involved.

TLDR: Exactly what you might have guessed: the exchanges backed their financial operations with a datastore without transactions, tragedy ensues. This can't be viewed as an indictment of "NoSQL" though; there are plenty of databases featuring ACID transactions that qualify as "NoSQL" (FoundationDB and OrientDB come to mind). That said, none of the popular NoSQL contenders have them, and the new guys with transactions tend to be commercial, so if you need transactions in an free and open-source datastore, just use PostgreSQL.

nemothekid 13y ago

While I do recognize this is a marketing piece for HyperDex, I'm disappointed HyperDex only chooses to compare itself with MongoDB and Cassandra. It would be different if HyperDex was fully open source and free, but its not. The "transaction" feature is only available with a license, which in most cases will price you out of any developer who is looking to start a bitcoin exchange. So unfortunately it comes across as "If only you had spent $x/mo, buying our product, then you wouldn't have had this problem!" Now that isn't bad, and its the definition of marketing, but then it goes on to compare itself against Cassandra (which has a widely different use case in the wild) and Mongo (which is already the whipping boy of the database community). However it doesn't help the use cases are also extremely well served by something like Postgres, which doesn't need a Warp addon, and has a huge developer community. What I would be more interested in, is how does HyperDex compare with the new "NoSQL" databases (FoundationDB, OrientDB, Spanner-like Databases) or even the traditional commercial SQL like Oracle. In short, it doesn't help to test and compare unsuccessful and unpopular use cases with these technologies. Most competent people, the same people who might pay for HyperDex, already know Cassandra is bad at transactions. What I would like to see is tell me why someone like Netflix, or Ooyala (tried, tested and popular use cases) would move from Cassandra to HyperDex.

threeseed 13y ago
Cassandra's transaction support has been improving pretty rapidly. http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0 http://www.datastax.com/dev/blog/lightweight-transactions-in...
- grizzles 13y ago
  For what I can only guess are strategic reasons, Datastax is moving away from the schema-less design that made Cassandra popular with early adopters. There are workarounds to this deprecation such as using the old Thrift API, but it doesn't really inspire confidence in the product unless the new schema oriented way of doing things appeals to you. PS those aren't transactions.
  - teacup50 13y ago
    "Schema-less" is a hack that appeals to people that don't realize that there is always schema. It's just a question of whether you want your data model to be ambiguous and poorly defined, or explicit.
    - nevi-me 13y ago
      Schema-less also means that you can afford to have deviations in your schemas not hinder your development needs. I agree with you, but I'd venture to say that a schema-less database means what it does, but that a collection of data always has schemas.
      - zAy0LfpBZLC8mAC 13y ago
        No, the schema is always a property of the system as a whole, data alone doesn't have an implicit schema. The schema is the set of type constraints that data has to conform to so that it can be processed by the system. Thus, a DB being "schema-less" simply means that it won't check/enforce those constraints - the schema still is there, of course, simply because a given piece of software that does anything meaningful will not sensibly process any random piece of garbage you feed it.

teawithcarl 13y ago

Cornell CS professor, superb work. See also -- http://hackingdistributed.com/2014/03/01/what-did-not-happen-at-mtgox/ http://hackingdistributed.com/2014/03/01/what-did-not-happen... "What may or may not have happened at Mt. Gox"

patio11 13y ago

That's because banks employ systems that guard against this kind of elementary error. It's called transactions with ACID guarantees. Bank systems are often not, in fact, ACID. Think of how many billions of transactions are processed in end-of-the-day batch runs, for example. Overdrafting is not only not impossible, it is a ludicrously profitable feature of the system. How is this secure? Velocity limits, sophisticated anti-fraud systems, acceptable losses, and, as a bit of a by-the-way, the fact that people who make a habit of defrauding banks generally have a few weeks prior to being woken up by very serious men with guns.

rdtsc 13y ago
It is interesting because the transaction log inconsistencies can often be resolved technically and because of the regulatory framework. Overdrafts can be "solved" in business sense. Now imagine something like an bitcoin exchange and bitcoin transaction log. Transaction errors cannot be fixed easily. So overdrafting an account means never getting the money back. There a single account would need to have ACID properties in way. It seems ability to deal with and tolerate data conflicts and inconsistencies is different for each business. There was some interest in CRTDs (data types that can automatically converge after a conflict occurs, like imagine a max() function or a set union operation) but still means shortly there could be an inconsistency in the system and sometimes that can be exploited and is just not an option.
Karellen 13y ago
Bank systems are ACID. They're often not immediate, using batch runs as you say, but the databases the batch runs work on are definitely ACID. The problem is not necessarily with overdrafts, as you suggest, but - as the article says - with race conditions between multiple readers/writers causing two near-simultaneous withdrawls to only make a single debit to the source account, while crediting both destination accounts. Note that this could lead to unexpected overdrafts being noticed later, if the exchange also wrote a consistent log of all transactions, and attempted to reconcile the log with the real-time balances periodically (e.g. on a daily basis), but it's not clear if they even did that. (Although I might be missing something)
- 13y ago

aardvark179 13y ago

"Historically, Bitcoin exchanges that suffered significant losses turned into fractional reserve banks, only to fold later." That is not what fractional reserve banking is and I wish bitcoin articles would stop misusing the term.

mikeash 13y ago
The continued misuse of that term seems telling. If they really think this is how normal banks work, it helps explain why they set up these bitcoin systems in such crazy ways.

rdtsc 13y ago

Please don't be MongoDB, please don't be MongoDB ... Sigh > a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks. Here is the thing, distributed systems are not standard academic courses. Just like networking they are often optional. That is really sad and it needs to change. The CAP theorem is not something fringe and exotic anymore it should be standard taught to everyone. MongoDB has really messed up. They promoted and marketed a shitty product wrapped in nice marketing. It created 2 things -- low barrier to entry and a trap. Both wrapped in one nice shiny package. "You should have read the docs to enable actually writing data disk when the user asks them to" doesn't jive with me. A really shitty default setup for something calling itself a database. Oh well, they reaped the benefits of their strategy, both the short term (success) and long term (failure) - it became the butt of jokes and ridicule.

threeseed 13y ago
Have you actually used the database since if you did you would know that the write concern was set to FSYNC_SAFE by every single driver (before it was the default) and given the lack of configurability should have been known by every developer. It never ceases to amaze me how few technical people actually "get it" even for products designed for themselves. Many, many people like things to be simple and easy. MongoDB fits this criteria perfectly and no matter how many times you call it a shitty product isn't going to dampen its popularity.
zorlem 13y ago
I, too, don't like the way MongoDB works and is marketed, but I would not bet too strongly on them failing. Only 10 years or so ago MySQL did pretty much the same, but they managed to succeed, despite there being better alternatives at the time. Even today, after all the development effort put into MySQL and its derivatives, there are still better and more capable databases, but that doesn't prevent MySQL from being one of the top three RDBMS-es (and we can bet it is the most popular FOSS RDBMS).

ChuckMcM 13y ago

Its unfortunate to use the ATM example because just such an exploit was used by carders [1] to get more money out of accounts than the accounts actually had. [1] http://time.com/48344/hackers-target-atms-for-unlimited-withdrawals/ http://time.com/48344/hackers-target-atms-for-unlimited-with...

edwintorok 13y ago
ATMs aren't as smart as the article says either: "the real code will check to see if there are sufficient funds". I noticed that (some?) ATMs around here tend to implement this flawed algorithm, paraphrasing the article: 0. database.begin() 1. mybalance = database.read("account-number") 2. newbalance = mybalance - amount 3. database.write("account-number", newbalance) 4. database.commit() 5. dispense_cash(amount) At first sight this might seem ACID, but its not. Consider what happens if the ATM is out of cash: step#5 fails, but the money was already withdrawn from your account! It doesn't automatically revert the transaction either: you have to go to a branch office and ask for the transaction to be reverted. So much for atomic transactions at ATMs.
- dnmmnd 13y ago
  Whilst that maybe the way your bank has chosen to operate, it is most certainly possible for ATMs to report a step#5 failure to the banks host which will reverse the transaction.
- ams6110 13y ago
  Wow I've never seen this happen. What happened to step -1: if (cash_in_machine < amount) abort
  - threeseed 13y ago
    He is talking about a failure to dispense the cash. I've had this happen twice to me whilst overseas.
- jasomill 13y ago
  Interesting. In one particularly ADHD-inspired case, I've also seen 5. Dispense cash 6. Wait a minute or so 7. Retract cash where 7 did not lead to a credit on my account until several days after I reported the problem to my bank. That's probably by design, however, since otherwise the machine would require automated defenses against surreptitious replacement of dispensed bills with counterfeits. Your example could be a similar failsafe against cases where the machine fails to properly dispense cash in a manner that nevertheless allows the cash to be retrieved. Otherwise you have to carefully separate errors that can't possibly lead to cash being dispensed (e.g., hopper empty) vs. those that might (e.g., bills jammed somewhere in the dispenser mechanism).

hibikir 13y ago

The first problem to be had when considering distributed systems is the fact that there is no such thing as a general purpose distributed system. Some problems require doing 50x more reads than writes. Others have more writes than reads. Sometimes losing a few inserts is not a big deal, but a small delay is terrible. Other systems just can't tolerate data loss, and would pay any price to keep it that way. You might have a lot of small pieces of data, searchable in very simple ways, or you might have multi GB documents that require so much indexing than Solr and Elastic seem inadequate. In the old days, you just bought a bigger box for the RDBMS, and you just were covered by knowing one tech. Now we need to go through a wide variety of tools that will rarely do everything you want, and have to write a bunch of code to compensate for the limitations of the tools. And then your DB of choice decides that they will only support their self hosted product, then they get bought by IBM, and you wonder if you'll be forced to retool your entire backend (Hello Cloudant!) Until there is some clarity in the market, and we get to see distributed stacks that are general purpose, we'll see issues like this popping up all the time, as there is a lack of people that both have good knowledge of all the available options and the skill to put them together into something that will solve your specific problem.

camus2 13y ago

Question! Do transactional NoSQL(others than HyperDex) databases exist? or do transactions make horizontal scaling difficult?

nacs 13y ago

Sounds like the author of this 'article' just wanted to make a thinly veiled plug for the "HyperDex" product. Blaming all of this on MongoDB is silly. You can do the kind of idiocy displayed by these exchanges even on generic DBs by doing things like not using transactions and/or not waiting for write confirmation. The root problem is in the (lack of) design and programming of these systems not their data store.

tlarkworthy 13y ago

you can put ACID like transactions on top of NoSQL databases, I did it here, but it took a static analyser to help get the logic right: https://github.com/tomlarkworthy/firesafe/wiki/Send-Item https://github.com/tomlarkworthy/firesafe/wiki/Send-Item

rubyfan 13y ago

This article lacks a fact base on what actually happened and seems to totally rely on conjecture in order to compare Hyperdex against leading NoSQL solutions.

dhendo 13y ago

Mongo does have document-level atomic update operators, such as $inc: http://docs.mongodb.org/manual/reference/operator/update/inc/#up._S_inc http://docs.mongodb.org/manual/reference/operator/update/inc...

thom_nic 13y ago
Yes this was completely ignored by the article. This along with a 'where' clause can guarantee the operation will only succeed iff there is sufficient funds. I don't know if other NoSQL DBs offer these sort of atomic operators. Which makes this statement seem particularly unfair: "The problem here stemmed from the broken-by-design interface and semantics offered by MongoDB." He seems to call out Mongo by name a couple times but doesn't say that the exchange was actually using Mongo.

danbruc 13y ago

It's the same story over and over again. The young wild guys arrive, full of energy and ambition to make the world a better place. They throw all the ancient stuff overboard because it seems ugly and useless. Then they painfully rediscover why the ugly stuff is there and not useless at all. And after a lot of pain they finally arrive at a slightly improved version of what they threw overboard long ago, at least if they did not surrender before.

PhantomGremlin 13y ago
> The young wild guys arrive That's exactly the problem. The 20-something hipsters who start these companies don't yet have the life experience to realize how ignorant they are. So of course they don't pay any attention to the lessons of history. And so they, very painfully, re-learn them. If the hipsters had some humility they would have also hired a few grizzled, experienced old fogies. Fortunately not all startups assume that anyone over the age of 30 isn't worth hiring. For a non-technical example, look at how Google hired Eric Schmidt to provide adult supervision.

jokoon 13y ago

What are the advantages of nosql btw ?

threeseed 13y ago
Basically developers came for the scalability and stayed because they are so easy to use/manage.
nikatwork 13y ago
NoSQL is fantastic for backing item-orientated webapps, it removes the transactional and relational overheads. Conversely I would not use NoSQL for anything concerning FIRE or complex inter-related data - those safeguards become important. Scalability is something of a red herring with NoSQL. Yes it's easier to scale, but my primary criteria for choosing it would be the nature of the data, not the intended scale.
Consultant32452 13y ago
A more "use-case" example than your other responses.. I believe Amazon uses a NoSQL database for their product data. It's tons of reads and relatively few writes. They don't give a crap about transaction quality for product descriptions, comments, etc. However, when you get to the checkout page and they start doing things like taking your money and updating their inventory... that's and ACID compliant database. When you're dealing with money you want transactions, rollbacks, etc.

late2part 13y ago

But, it is webscale!

bkirwi 13y ago

A bit irritated at how the author complains about other databases astroturfing on Hacker News, then writes a glowing post about HyperDex without mentioning that he's one of the authors. That said, I agree with the thesis that current-generation distributed systems provide a terrible API for users who actually want to get concurrency right. I'm optimistic about projects like Summingbird[0] and Bloom[1], which run very high-level declarative programs with simple consistency guarantees. [0] https://github.com/twitter/summingbird https://github.com/twitter/summingbird [1] http://www.bloom-lang.net/ http://www.bloom-lang.net/

atmosx 13y ago

> It points to a social failure: a failure of distributed systems academics to educate developers and to equip them with clear-thinking frameworks. So, will the OP do an exchange and show us how it's done using his database of and approach of choice. Because writing articles on other's failures is kinda of... Easy. I'm a little sick of this attitude. Nothing personal with - what I thought was Sinan Eren at first, confusing the names after so many years - the OP here, but given the fact that he is deliberately calling out amateurs and poor designers people who (apparently) deserved it, he might very well create a secure bitcoin exchange in the real world, using his technologies.. There is a huge market for that, today more than ever.

sillysaurus3 13y ago
The attitude is warranted when they're losing other people's money.
- atmosx 13y ago
  Totally get that, but his approach seems a little snooty to me and runs throughout the article that's why I had this reaction/comment.

DigitalSea 13y ago

As soon as I saw the term NoSQL and "Brings Down" in the title of the submission, I knew it was going to be MongoDB. Firstly, I don't understand the hatred towards MongoDB. Like most NoSQL datastores, it doesn't offer things like transactions and ACID compliance: this is well-known thing. There are some things that MongoDB does very well, currently I am using it to store clicks for an internal application which then displays those clicks on a page overlay, it handles the task very well. I would never use it for MongoDB is not to blame for bringing down the exchanges that decided to use it, the stupidity of the developers behind said exchanges are the sole reason. You can open up a can of tinned pineapples with an axe, but that doesn't make it the right tool for the job. This post was one big marketing piece for HyperDex and while it is touted as open source, but that isn't entirely true. If you want support for transactions, you have to by a licence. So basically what this post was saying is, "If you spent X amount of money on a HyperDex licence, you wouldn't have had these issues" most people that start these exchanges start them from nothing and probably can't afford or wouldn't warrant spending money on a licence. If you replaced MongoDB with pretty much every other NoSQL database out there, the result would have been the same. Mongo just has a better marketing team behind them in the form of 10Gen getting people to buy into the hype. However, as far as I am aware of, 10Gen aren't telling people to use MongoDB for exchange platforms and anything involving transactions (I could be wrong though). This is a classic case of seeing people talking about how NoSQL is the future and others getting caught up in the hype. If you want a database that you can rely on, especially when dealing with transactions: use something like Postgresql, not MongoDB and or insert over-hyped NoSQL database here

hackerboos 13y ago
> it doesn't offer things like transactions and ACID compliance: this is well-known thing Evidently not. Which is why people are still using it for financial based systems.
kalleboo 13y ago
> the stupidity of the developers behind said exchanges are the sole reason Exactly. The fact that they picked MongoDB for this application in the first place tells me that even if they used an ACID-compliant database, they still wouldn't have known to use transactions, and would be vulnerable to the same attack.

kylemaxwell 13y ago

Not quite understanding the focus on NoSQL for this use case set. I love MongoDB in the right place and Redis in others. But when you have a bunch of requirements that lend themselves to traditional DBs, choosing MongoDB or HyperDex feels like choosing a tech because it's cool rather than because it is the proper engineering choice.

cranklin 13y ago

"They did what anyone would do after reading one too many astroturf articles on Hacker News. Sure, their system failed, but in a sense, the overall system failed them." The technology didn't fail them. On the contrary, the technology did exactly what it was supposed to do. The devs could have (should have) considered these race condition scenarios.

ibyon 13y ago

I don't think that MongoDB or any nosql is right choice for financial application. You can build for yelp but not for banking.

platform 13y ago

Marklogic supports ACID transactions. being that it is also a horizontally scalable system, it allows those transactions not just to spawn multiple objects, but also across multiple hosts. http://www.marklogic.com/blog/can-you-pass-the-acid-test/ http://www.marklogic.com/blog/can-you-pass-the-acid-test/ They also support data tiering into HDFS, and now most recently RDF with SPARQL They have been doing ACID for a number of years, not sure for how long. They are not free (unless your data set is small)

kofejnik 13y ago

Why wouldn't they use a plain old ACID database? Was their load something that Oracle couldn't handle?