7 ms·

I agree with you, per our experience Mongo is not scalable. We use Mongo in our company (60GB per db.stats().fileSize), both for relational data, and for what w

by deepsun 12y ago

I agree with you, per our experience Mongo is not scalable. We use Mongo in our company (60GB per db.stats().fileSize), both for relational data, and for what we call "fat data" (just to avoid calling these mere gigabytes "big data" -- those are just lots of statistical numbers, more like OLAP cubes). Over many problems and sad surprises, we came to the point that we'd better have used PostgreSQL (or any other SQL) for relational data, and some NoSQL for "fat data". Or just single PostgreSQL would do better, because Mongo's feature set is a subset of PostgreSQL's feature set. Just Mongo renamed them and pretends it's something new.

donw 12y ago
This fits with my experience as well, with (possibly) a larger data set. MongoDB is actually pretty good as a document store, where you can accept soft commits, and are dealing with non-relational data that doesn't need to be aggregated. When you start needing more than just a loose pile of documents, or live in a world where you really need ACID, Mongo has this nasty habit of falling down on the floor and twitching. Those problems are solveable, but it doesn't just happen "out-of-the-box".
- e12e 12y ago
  I wonder, what do you get over exporting a file system over webdav, if you only do "document storage" of json documents? After all filesystems are made for storing a hierarchy of files... some indexes for searching? More convenient rest api?
  - donw 12y ago
    The API is much more convenient, and last time I checked, WebDAV didn't provide the same wealth of query options or multi-node replication. Plus, I can get solid commercial support for MongoDB, which matters a lot if you're using it to power a business.
    - e12e 12y ago
      Multi-node replication for webdav would be done at the file system level (eg: gfs or whatnot). By query options, to you mean some kind of join, for example? Obviously anything beyond plain id-based get/set (by file path/url) isn't possible with "just" webdav. But if you already limit yourself to looking at single documents (ie: json-files) -- you could always fetch doc1, find the id/path of doc2 in doc1 and fetch that. I'm not saying this sounds like a sane db strategy, I'm just not convinced doing the same thing, with a little easier api sounds like a sane strategy either?
nl 12y ago
Or just single PostgreSQL would do better, because Mongo's feature set is a subset of PostgreSQL's feature set. Just a slight warning on this, because I was mildly burnt by hearing this and assuming it was true. For background, I'm a long time happy Postgres user. Recently I decided to use it for a new project instead of Mongo and to utilise the new JSON support. Turns out it is great for CRD apps, not so much for CRUD. The current release version of Postgresql (9.3) has no capability to do updates to parts of JSON stored as the JSON datatype (ie, you have to read the entire JSON blob, change it and write it back)[1]. Updates within JSON fields are coming in 9.4. [1] http://dba.stackexchange.com/questions/54663/how-can-i-update-a-json-element-in-postgresql-json-datatype http://dba.stackexchange.com/questions/54663/how-can-i-updat...
- jbellis 12y ago
  How sure are you that mongodb doesn't rewrite the entire document with each update? I couldn't find a definitive answer without source diving.
  - tracker1 12y ago
    But, it's in the box API functionality... since it's a single record, there's minimal locking involved. Depending on your load that can be very important. For that matter, imho doing a locked partial update via something like _.assign would be fine imho in postgres. It depends on how you really need to use your data... and how it fits into that. If you have a lot of recursion in your data, it may be better suited towards sql... if you have a lot of data gathered around a group of objects/documents a document db like mongo/rethink/elasticsearch may be best... if you really need key/value lookups, then cassandra is hard to beat. For that matter, having data duplicated/replicated to multiple types of db servers is entirely reasonable. You management UI can interact with an SQL datastore, and on save, you also save to Mongo. That was the interim step I chose in migrating our data structures.. the queries that run against mongo work great, there's three servers in the replica set, for a relatively small data set, and it is really nice.
  - nl 12y ago
    I'm more familiar with CouchDB, and I know Couch doesn't have to do that. Even if MongoDB does at least you don't have to write the functionality yourself in client code.
cal2014 12y ago
Like all technologies MongoDB is a tool and you need to pick the right tool for the right job or in this case the right DB for the right work load. We run 100's of TB of MongoDB with 100's of millions of Mongo Ops. Does it have challenges for sure, does mean you can't scale it no. Here's a talk about scaling MongoDB specifically. http://engineering.objectrocket.com/2014/03/10/scaling-mongodb-going-multi-terabyte/ http://engineering.objectrocket.com/2014/03/10/scaling-mongo...
- colordrops 12y ago
  You can scale the file system too but it doesn't mean that it's well suited to the job.
- deepsun 12y ago
  Right, but we're disappointed that MongoDB markets itself to fit into NoSQL niche, while it doesn't. If it honestly declared its shortcomings, we'd have nothing against, we'd love it. For example, it doesn't support crucial for NoSQL "compare-and-set" functionality to avoid transactions/locks. Their best suggestion is to use findAndUpdate() with the full object for "find". And it works (though slow) when you have static schema. But over time you'll want to change your objects, and findAndUpdate() won't find them anymore. Grief. Also, order of fields in nested JSON object matters for findAndUpdate(), so have a happy time debugging why it doesn't find some objects anymore.
StillBored 12y ago
"fat data" (just to avoid calling these mere gigabytes "big data" Mere GB's of data is just a normal database. I have a rule that should be more widespread: If you can put the database in RAM on a x86 server, its not "big data" by any stretch. Then beyond that, it becomes more complex, but for starters lets consider whether the indexes fit in RAM. If the indexes/hashes for your data cannot fit in RAM on a commodity x86 then you probably can consider that you have "big data". So, currently its possible to buy supermicro systems that take 6TB of RAM (just a normal QPI link) without getting into any of the exotic SSI systems (like the SGI UV 2000). We should also avoid talking about the physical plant requirements for "big data" as well, since as its possible to put over 350TB of storage in 4U with products from Nexsan, JetStor, etc. That is over 3PB per rack... So, you can call your data set "big data" if the indexes are > 6TB or the actual data set is > 3PB. These numbers will change next year when new machines/storage arrive.
- techdragon 12y ago
  I find it easier to think about it from a data "usage" angle, "if you can query anything you like while running the 'database' on a single machine and get your answer within 24hrs your data is not 'big'" I used to see 24hr OLAP cube runs and no one ever called that 'big data'. It's entirely a question of scale in my mind, because these days you can buy truly gigantic servers, and they are phenomenally powerful but if your data needs multiple servers dividing the load in order to perform queries in a timely manner then you start talking about big data, it's a question of scale.
- deepsun 12y ago
  Yes, we call it "fat data" because of different approach to it, "bigdatey". E.g. it's not relational -- we don't JOIN it with anything (we intentionally designed it so that we stay scalable while growing), and it would perfectly fit a NoSQL datastorage (just don't confuse NoSQL with MongoDB :) After working in Google I (with another teammate) kinda "feel" what's big data: it's more about approach, mindset and toolset to work with it. I agree with you and @techdragon that if you can fit the data into one machine it's probably not really big. But one can also work with 1GB of data using bigdata approach, what we call "fat data". When we grow out of a single machine we won't need to rewrite our project. Nevertheless, this all doesn't prevent our sales team to say "big data" and "cloud" in every their phrase :)

by deepsun 12y ago

donw 12y ago

This fits with my experience as well, with (possibly) a larger data set. MongoDB is actually pretty good as a document store, where you can accept soft commits, and are dealing with non-relational data that doesn't need to be aggregated. When you start needing more than just a loose pile of documents, or live in a world where you really need ACID, Mongo has this nasty habit of falling down on the floor and twitching. Those problems are solveable, but it doesn't just happen "out-of-the-box".

e12e 12y ago
I wonder, what do you get over exporting a file system over webdav, if you only do "document storage" of json documents? After all filesystems are made for storing a hierarchy of files... some indexes for searching? More convenient rest api?
- donw 12y ago
  The API is much more convenient, and last time I checked, WebDAV didn't provide the same wealth of query options or multi-node replication. Plus, I can get solid commercial support for MongoDB, which matters a lot if you're using it to power a business.
  - e12e 12y ago
    Multi-node replication for webdav would be done at the file system level (eg: gfs or whatnot). By query options, to you mean some kind of join, for example? Obviously anything beyond plain id-based get/set (by file path/url) isn't possible with "just" webdav. But if you already limit yourself to looking at single documents (ie: json-files) -- you could always fetch doc1, find the id/path of doc2 in doc1 and fetch that. I'm not saying this sounds like a sane db strategy, I'm just not convinced doing the same thing, with a little easier api sounds like a sane strategy either?

nl 12y ago

Or just single PostgreSQL would do better, because Mongo's feature set is a subset of PostgreSQL's feature set. Just a slight warning on this, because I was mildly burnt by hearing this and assuming it was true. For background, I'm a long time happy Postgres user. Recently I decided to use it for a new project instead of Mongo and to utilise the new JSON support. Turns out it is great for CRD apps, not so much for CRUD. The current release version of Postgresql (9.3) has no capability to do updates to parts of JSON stored as the JSON datatype (ie, you have to read the entire JSON blob, change it and write it back)[1]. Updates within JSON fields are coming in 9.4. [1] http://dba.stackexchange.com/questions/54663/how-can-i-update-a-json-element-in-postgresql-json-datatype http://dba.stackexchange.com/questions/54663/how-can-i-updat...

jbellis 12y ago
How sure are you that mongodb doesn't rewrite the entire document with each update? I couldn't find a definitive answer without source diving.
- tracker1 12y ago
  But, it's in the box API functionality... since it's a single record, there's minimal locking involved. Depending on your load that can be very important. For that matter, imho doing a locked partial update via something like _.assign would be fine imho in postgres. It depends on how you really need to use your data... and how it fits into that. If you have a lot of recursion in your data, it may be better suited towards sql... if you have a lot of data gathered around a group of objects/documents a document db like mongo/rethink/elasticsearch may be best... if you really need key/value lookups, then cassandra is hard to beat. For that matter, having data duplicated/replicated to multiple types of db servers is entirely reasonable. You management UI can interact with an SQL datastore, and on save, you also save to Mongo. That was the interim step I chose in migrating our data structures.. the queries that run against mongo work great, there's three servers in the replica set, for a relatively small data set, and it is really nice.
- nl 12y ago
  I'm more familiar with CouchDB, and I know Couch doesn't have to do that. Even if MongoDB does at least you don't have to write the functionality yourself in client code.

cal2014 12y ago

Like all technologies MongoDB is a tool and you need to pick the right tool for the right job or in this case the right DB for the right work load. We run 100's of TB of MongoDB with 100's of millions of Mongo Ops. Does it have challenges for sure, does mean you can't scale it no. Here's a talk about scaling MongoDB specifically. http://engineering.objectrocket.com/2014/03/10/scaling-mongodb-going-multi-terabyte/ http://engineering.objectrocket.com/2014/03/10/scaling-mongo...

colordrops 12y ago
You can scale the file system too but it doesn't mean that it's well suited to the job.
deepsun 12y ago
Right, but we're disappointed that MongoDB markets itself to fit into NoSQL niche, while it doesn't. If it honestly declared its shortcomings, we'd have nothing against, we'd love it. For example, it doesn't support crucial for NoSQL "compare-and-set" functionality to avoid transactions/locks. Their best suggestion is to use findAndUpdate() with the full object for "find". And it works (though slow) when you have static schema. But over time you'll want to change your objects, and findAndUpdate() won't find them anymore. Grief. Also, order of fields in nested JSON object matters for findAndUpdate(), so have a happy time debugging why it doesn't find some objects anymore.

StillBored 12y ago

"fat data" (just to avoid calling these mere gigabytes "big data" Mere GB's of data is just a normal database. I have a rule that should be more widespread: If you can put the database in RAM on a x86 server, its not "big data" by any stretch. Then beyond that, it becomes more complex, but for starters lets consider whether the indexes fit in RAM. If the indexes/hashes for your data cannot fit in RAM on a commodity x86 then you probably can consider that you have "big data". So, currently its possible to buy supermicro systems that take 6TB of RAM (just a normal QPI link) without getting into any of the exotic SSI systems (like the SGI UV 2000). We should also avoid talking about the physical plant requirements for "big data" as well, since as its possible to put over 350TB of storage in 4U with products from Nexsan, JetStor, etc. That is over 3PB per rack... So, you can call your data set "big data" if the indexes are > 6TB or the actual data set is > 3PB. These numbers will change next year when new machines/storage arrive.

techdragon 12y ago
I find it easier to think about it from a data "usage" angle, "if you can query anything you like while running the 'database' on a single machine and get your answer within 24hrs your data is not 'big'" I used to see 24hr OLAP cube runs and no one ever called that 'big data'. It's entirely a question of scale in my mind, because these days you can buy truly gigantic servers, and they are phenomenally powerful but if your data needs multiple servers dividing the load in order to perform queries in a timely manner then you start talking about big data, it's a question of scale.
deepsun 12y ago
Yes, we call it "fat data" because of different approach to it, "bigdatey". E.g. it's not relational -- we don't JOIN it with anything (we intentionally designed it so that we stay scalable while growing), and it would perfectly fit a NoSQL datastorage (just don't confuse NoSQL with MongoDB :) After working in Google I (with another teammate) kinda "feel" what's big data: it's more about approach, mindset and toolset to work with it. I agree with you and @techdragon that if you can fit the data into one machine it's probably not really big. But one can also work with 1GB of data using bigdata approach, what we call "fat data". When we grow out of a single machine we won't need to rewrite our project. Nevertheless, this all doesn't prevent our sales team to say "big data" and "cloud" in every their phrase :)