7 ms·

The comment was referencing an anonymous whitehat talking about having hacked an exchange for 100BTC due to way the service was doing their accounting. They'd e

by nwh 13y ago

The comment was referencing an anonymous whitehat talking about having hacked an exchange for 100BTC due to way the service was doing their accounting. They'd essentially found a race condition that allowed them to overwrite a large withdrawal with the result of a smaller one, affording them unlimited Bitcoin. It was inferred that this was coinbase, though not directly stated I don't think. http://www.reddit.com/r/Bitcoin/comments/1wtbiu/how_i_stole_roughly_100_btc_from_an_exchange_and/ http://www.reddit.com/r/Bitcoin/comments/1wtbiu/how_i_stole_... Why is MongoDB bad? It's not ACID compliant. It's hard to determine the state of the database with concurrent processes attempting to modify its data. Given the explanation it's fairly clear that it's Coinbase being talked about. http://en.wikipedia.org/wiki/ACID http://en.wikipedia.org/wiki/ACID

fennecfoxen 13y ago
From the world of NoSQL, let me nitpick a bit: You don't need to run ACID software to do safe processing of your transactions (durability and clever business-logic constraints can serve to limit your exposure to losses to finite knowable values, which may be worth it because you can achieve availability and partition-tolerance)... but man, ACID sure makes things a heck of a lot easier, and will save you from many classes of implementation error. Think twice before giving it up. Think more than twice. (Also, MongoDB is like the worst NoSQL data store out there for a task like this.)
- BlackDeath3 13y ago
  Not to start a flame war here, but from the perspective of someone with limited database experience, what is the best database out there? I realize that this question is very vague, but is there a best general-use database?
  - georgemcbay 13y ago
    "Best" is a strong word to use universally, but IMO when in doubt use PostgreSQL. It has a very solid traditional ACID/SQL core and it has also been introducing "NoSQL" style json document storage on top of that for projects that want to go that route. I suspect PostgreSQL will do the the NoSQL type stuff well a a lot sooner than NoSQL databases will do the consistence/durability thing well. Having said all of that on my personal site I actually use RethinkDB because it is delightfully easy to setup and code against (I just persist my Go structs to the db and don't worry about a relational schema to maintain) and I don't really care about data integrity there, so I don't mind that RethinkDB is relatively new and untested. So one size doesn't fit all, but PostgreSQL is still, IMO, generally the "safest bet" if you do care about data integrity and don't want to spend hundreds of thousands of dollars on licensing to scale it up.
    - BlackDeath3 13y ago
      Out of curiosity, is there anything to this pairing of languages with databases presented in the link below? Does each database have its own optimal programming language? https://www.gandi.net/hosting/simple https://www.gandi.net/hosting/simple
      - exelius 13y ago
        Generally not; though there are some exceptions given the maturity level of the database drivers in any given language. For the most part though, any well-supported programming language should have a mature driver for any well-supported database.
      - epsylon 13y ago
        The pairing on Gandi's offer has more to do with which database programmers of a given language are usually more familiar with. Hopefully you can use pgSQL in all three cases :-)
  - bri3d 13y ago
    I would use Postgres. It's not particularly likely to eat your data or kill you as you get started, and the ecosystem is relatively unified which means consistent and generally good best practices documentation. Becoming a Postgres expert could take a lifetime as the database offers an astonishing depth of features, but it won't bite you if you just use the basics. Plus the source is quite good if you ever find the urge to dig deep. A couple of gotchas might bother you. You'll need to implement your own strategy for upserts if you need those. But you do get robust native support for popular data types commonly seen in today's apps: geographic data as well as document data (via HStore or JSON). The MySQL/MariaDB family is well-proven as well, but with tens of available storage engines, two core forks, and a bit of non-standard SQL magic sprinkled throughout it's harder for a beginner to learn the best practices IMO. The challenge comes once you exceed what one machine can do and want to move beyond the simple client(s) <-> (possibly replicated) server model. Keep in mind that depending on your project, this might not ever happen to you. Certain types of mid-sized sites like Basecamp and Stack Exchange have gotten away with throwing bigger hardware at their database systems. If or when you reach the point where you need a distributed database, you have to choose between consistency and availability. I'll let the experts explain some of those concepts by linking to this excellent article by Coda Hale: http://codahale.com/you-cant-sacrifice-partition-tolerance/ http://codahale.com/you-cant-sacrifice-partition-tolerance/ . That's when you'll find yourself in the NoSQL world - many NoSQL databases are useful in that they try to trade consistent for available in some way or another. While they generally fail completely to actually do so, as illustrated in Kyle Kingsbury's excellent "Call Me Maybe" series, the largest -scale systems are all based on a "NoSQL" data store in some way or another.
  - dbpatterson 13y ago
    Unless you really know what you're doing, just use a relational database. Opinions differ, but Postgresql is a pretty safe bet.
  - justinsb 13y ago
    You will likely be happy with PostgreSQL or MySQL. If you don't know why you're not using a SQL database, likely you should be. If you do want to experiment with NoSQL databases, MongoDB is probably the easiest to get going with (which is why it is so widely used). However, you seem statistically unlikely to be in the group that is happy with MongoDB in production :-) HBase has a much brighter future for the NoSQL use cases (and is even getting SQL support!)
  - jmnicolas 13y ago
    +1 for PostgreSQL but only if you learn relational database design (it's not that hard). Else you're better off with NoSQL (I found CouchDB amazingly easy).
  - fennecfoxen 13y ago
    SQL databases: Postgres unless you have a reason not to. From the point of view of NoSQL data stores, it'll really depend on your application needs, and I can't give you a full overview, buuuut.... Amazon did a solid job with its DynamoDB database, and Apache Cassandra did a solid job cloning it. I believe I've also heard decent things about the sanity level of Redis and Hypderdex.
    - catwell 13y ago
      DynamoDB was released in 2012, Cassandra in 2008, so it is definitely not a clone of the former :) It is loosely based on the Dynamo paper like most modern distributed databases, but Dynamo and DynamoDB are very different things. Riak could be another interesting option in that space. Redis is more adapted for small datasets with high workloads and low durability expectations (i.e. do not host a Bitcoin exchange purely on it unless you really know what you are doing :p).
- sdkmvx 13y ago
  I just want to add that this is obviously true because ACID can be implemented on the filesystem, which is not ACID. So to create a reliable transaction processing system on MongoDB (or any non-ACID system), you would need to build ACID on top of it. And if you're interested in Bitcoins and not database design, that's probably not the right way to go.
  - hack3r_n3ws 13y ago
    And if you build ACID on top of mongodb, then you lose the benefits of NoSQL. Not to mention the immense difficulty of building ACID into your business logic... It makes sense to use mongodb for commenting system where nobody cares if you lose a comment or certain comments are written before others. But no competent architect would use mongodb for financial/currency transactions. If coinbase did use mongodb, it will be inevitable they will have transactional issues. The more popular coinbase gets, the more transactions they process, the more issues they'll have. I don't envy the poor souls that have to track down these kinds of issues...
    - nawitus 13y ago
      Well, NoSQL doesn't simply mean "not ACID". There's also other stuff like documents instead of tables and native support for JSON.
      - notwedtm 13y ago
        On some of them. I have come to call anything that stores data in a non-SQL way NoSQL. Accurate? Probably not, but it sure does seem to bring in those other ones that everyone calls NoSQL for some reason (Redis for example).
    - VintageCool 13y ago
      FYI: TokuMX builds ACID underneath MongoDB. They ripped out Mongo's storage engine and replaced it with their own. It seems to be faster and it supports ACID transactions.
deleted 13y ago
[deleted]
aioprisan 13y ago
You can do 2-step commits in MongoDB to make your transactions ACID: http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/ http://docs.mongodb.org/manual/tutorial/perform-two-phase-co...
- justinsb 13y ago
  That procedure is ridiculous, and they even admit that it is broken e.g. other callers can see intermediate results. If you are even thinking about following any of the advice on that page, you are no longer in the MongoDB use case: you plainly need transactions.
ericingram 13y ago
TokuMX is a drop in replacement for MongoDB and provides ACID transactions, better indexing (fractal tree) and data compression (up to 90%). All without sacrificing performance, flexibility or tool support. Tokutek engineers are top notch.
- justin66 13y ago
  And their almost-GPL database code is covered by multiple patents, so don't read it (or even use it? I'm not sure what's safest there) if you ever plan on doing database development yourself. Remember: willful infringement = treble damages.
  - ericingram 13y ago
    You may have a good point however the parent thread is about using MongoDB without ACID support and you don't need to read the source code to get over that hurdle with TokuMX.
poutine 13y ago
MongoDB isn't the right tool here of course, but the rumoured attack could have been avoided using MongoDB's findAndModify command which gives you the atomicity you need to update a ledger. The problem here is that developers not taking the time to understand their application and how their tools work. Yes, MongoDB gives you enough rope to hang yourself and I certainly wouldn't suggest using it for a financial ledger, but you could have a durable application based on it if you really wanted to. However if developers of a financial system cant grasp the need for atomicity in the transaction path then they're going to be buggering things up all over the place.

by nwh 13y ago

fennecfoxen 13y ago

From the world of NoSQL, let me nitpick a bit: You don't need to run ACID software to do safe processing of your transactions (durability and clever business-logic constraints can serve to limit your exposure to losses to finite knowable values, which may be worth it because you can achieve availability and partition-tolerance)... but man, ACID sure makes things a heck of a lot easier, and will save you from many classes of implementation error. Think twice before giving it up. Think more than twice. (Also, MongoDB is like the worst NoSQL data store out there for a task like this.)

BlackDeath3 13y ago
Not to start a flame war here, but from the perspective of someone with limited database experience, what is the best database out there? I realize that this question is very vague, but is there a best general-use database?
- georgemcbay 13y ago
  "Best" is a strong word to use universally, but IMO when in doubt use PostgreSQL. It has a very solid traditional ACID/SQL core and it has also been introducing "NoSQL" style json document storage on top of that for projects that want to go that route. I suspect PostgreSQL will do the the NoSQL type stuff well a a lot sooner than NoSQL databases will do the consistence/durability thing well. Having said all of that on my personal site I actually use RethinkDB because it is delightfully easy to setup and code against (I just persist my Go structs to the db and don't worry about a relational schema to maintain) and I don't really care about data integrity there, so I don't mind that RethinkDB is relatively new and untested. So one size doesn't fit all, but PostgreSQL is still, IMO, generally the "safest bet" if you do care about data integrity and don't want to spend hundreds of thousands of dollars on licensing to scale it up.
  - BlackDeath3 13y ago
    Out of curiosity, is there anything to this pairing of languages with databases presented in the link below? Does each database have its own optimal programming language? https://www.gandi.net/hosting/simple https://www.gandi.net/hosting/simple
    - exelius 13y ago
      Generally not; though there are some exceptions given the maturity level of the database drivers in any given language. For the most part though, any well-supported programming language should have a mature driver for any well-supported database.
    - epsylon 13y ago
      The pairing on Gandi's offer has more to do with which database programmers of a given language are usually more familiar with. Hopefully you can use pgSQL in all three cases :-)
- bri3d 13y ago
  I would use Postgres. It's not particularly likely to eat your data or kill you as you get started, and the ecosystem is relatively unified which means consistent and generally good best practices documentation. Becoming a Postgres expert could take a lifetime as the database offers an astonishing depth of features, but it won't bite you if you just use the basics. Plus the source is quite good if you ever find the urge to dig deep. A couple of gotchas might bother you. You'll need to implement your own strategy for upserts if you need those. But you do get robust native support for popular data types commonly seen in today's apps: geographic data as well as document data (via HStore or JSON). The MySQL/MariaDB family is well-proven as well, but with tens of available storage engines, two core forks, and a bit of non-standard SQL magic sprinkled throughout it's harder for a beginner to learn the best practices IMO. The challenge comes once you exceed what one machine can do and want to move beyond the simple client(s) <-> (possibly replicated) server model. Keep in mind that depending on your project, this might not ever happen to you. Certain types of mid-sized sites like Basecamp and Stack Exchange have gotten away with throwing bigger hardware at their database systems. If or when you reach the point where you need a distributed database, you have to choose between consistency and availability. I'll let the experts explain some of those concepts by linking to this excellent article by Coda Hale: http://codahale.com/you-cant-sacrifice-partition-tolerance/ http://codahale.com/you-cant-sacrifice-partition-tolerance/ . That's when you'll find yourself in the NoSQL world - many NoSQL databases are useful in that they try to trade consistent for available in some way or another. While they generally fail completely to actually do so, as illustrated in Kyle Kingsbury's excellent "Call Me Maybe" series, the largest -scale systems are all based on a "NoSQL" data store in some way or another.
- dbpatterson 13y ago
  Unless you really know what you're doing, just use a relational database. Opinions differ, but Postgresql is a pretty safe bet.
- justinsb 13y ago
  You will likely be happy with PostgreSQL or MySQL. If you don't know why you're not using a SQL database, likely you should be. If you do want to experiment with NoSQL databases, MongoDB is probably the easiest to get going with (which is why it is so widely used). However, you seem statistically unlikely to be in the group that is happy with MongoDB in production :-) HBase has a much brighter future for the NoSQL use cases (and is even getting SQL support!)
- jmnicolas 13y ago
  +1 for PostgreSQL but only if you learn relational database design (it's not that hard). Else you're better off with NoSQL (I found CouchDB amazingly easy).
- fennecfoxen 13y ago
  SQL databases: Postgres unless you have a reason not to. From the point of view of NoSQL data stores, it'll really depend on your application needs, and I can't give you a full overview, buuuut.... Amazon did a solid job with its DynamoDB database, and Apache Cassandra did a solid job cloning it. I believe I've also heard decent things about the sanity level of Redis and Hypderdex.
  - catwell 13y ago
    DynamoDB was released in 2012, Cassandra in 2008, so it is definitely not a clone of the former :) It is loosely based on the Dynamo paper like most modern distributed databases, but Dynamo and DynamoDB are very different things. Riak could be another interesting option in that space. Redis is more adapted for small datasets with high workloads and low durability expectations (i.e. do not host a Bitcoin exchange purely on it unless you really know what you are doing :p).
sdkmvx 13y ago
I just want to add that this is obviously true because ACID can be implemented on the filesystem, which is not ACID. So to create a reliable transaction processing system on MongoDB (or any non-ACID system), you would need to build ACID on top of it. And if you're interested in Bitcoins and not database design, that's probably not the right way to go.
- hack3r_n3ws 13y ago
  And if you build ACID on top of mongodb, then you lose the benefits of NoSQL. Not to mention the immense difficulty of building ACID into your business logic... It makes sense to use mongodb for commenting system where nobody cares if you lose a comment or certain comments are written before others. But no competent architect would use mongodb for financial/currency transactions. If coinbase did use mongodb, it will be inevitable they will have transactional issues. The more popular coinbase gets, the more transactions they process, the more issues they'll have. I don't envy the poor souls that have to track down these kinds of issues...
  - nawitus 13y ago
    Well, NoSQL doesn't simply mean "not ACID". There's also other stuff like documents instead of tables and native support for JSON.
    - notwedtm 13y ago
      On some of them. I have come to call anything that stores data in a non-SQL way NoSQL. Accurate? Probably not, but it sure does seem to bring in those other ones that everyone calls NoSQL for some reason (Redis for example).
  - VintageCool 13y ago
    FYI: TokuMX builds ACID underneath MongoDB. They ripped out Mongo's storage engine and replaced it with their own. It seems to be faster and it supports ACID transactions.

deleted 13y ago

[deleted]

aioprisan 13y ago

You can do 2-step commits in MongoDB to make your transactions ACID: http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/ http://docs.mongodb.org/manual/tutorial/perform-two-phase-co...

justinsb 13y ago
That procedure is ridiculous, and they even admit that it is broken e.g. other callers can see intermediate results. If you are even thinking about following any of the advice on that page, you are no longer in the MongoDB use case: you plainly need transactions.

ericingram 13y ago

TokuMX is a drop in replacement for MongoDB and provides ACID transactions, better indexing (fractal tree) and data compression (up to 90%). All without sacrificing performance, flexibility or tool support. Tokutek engineers are top notch.

justin66 13y ago
And their almost-GPL database code is covered by multiple patents, so don't read it (or even use it? I'm not sure what's safest there) if you ever plan on doing database development yourself. Remember: willful infringement = treble damages.
- ericingram 13y ago
  You may have a good point however the parent thread is about using MongoDB without ACID support and you don't need to read the source code to get over that hurdle with TokuMX.

poutine 13y ago

MongoDB isn't the right tool here of course, but the rumoured attack could have been avoided using MongoDB's findAndModify command which gives you the atomicity you need to update a ledger. The problem here is that developers not taking the time to understand their application and how their tools work. Yes, MongoDB gives you enough rope to hang yourself and I certainly wouldn't suggest using it for a financial ledger, but you could have a durable application based on it if you really wanted to. However if developers of a financial system cant grasp the need for atomicity in the transaction path then they're going to be buggering things up all over the place.