7 ms·

MongoDB 1.8 (stable) released

187 points by luigi 16y ago

rbranson 16y ago
Starting to get excited once again about MongoDB. I was kind of down about it after having some issues with real world implementations. Considering journaling is something I would never have thought would have made it in, I wonder if they will come around on the memory mapped I/O like everyone else eventually does. EDIT: Also... does the group commit mean that ALL write transactions will be un-acknowledged to the client until the group commit finishes?
- mathias_10gen 16y ago
  Clients are able to wait for the next group commit by adding fsync:true to getLastError calls (some drivers allow you to add this to WriteConcern). We already have some enhancements to this planned for the 1.9 series.
- dmytton 16y ago
  "You can wait for group commit acknowledgement with the getLastError command. When running with --journal, the fsync:true option returns after the data is physically written to the journal (rather than actually fsync'ing all the data files). Note that the group commit interval (see above) is considerable: you may prefer to call getLastError without fsync, or with a w: parameter instead with replication. In releases after 1.8.0 the delay for commit acknowledgement will be shorter." http://www.mongodb.org/display/DOCS/Journaling http://www.mongodb.org/display/DOCS/Journaling
  - rbranson 16y ago
    RTFM'ing myself in public. While it's a great step forward, the whole thing sounds pretty immature at this point. It'll be a good day when this stuff matures and MongoDB has fsync:true and --journal by default, much to the chagrin of sensationalists everywhere. They'll really need to add some major smarts into the journaling and group commit if they're going to be able to stack up concurrent I/Os to help feed big disk arrays and even to get the best use out of SSDs which make back-and-forth latency on the I/O pipes even more significant.
- tmountain 16y ago
  I'm curious what you're getting at IRT memory mapped I/O. To me, one of Mongo's selling points is the way it lets the OS manage caching for you instead of bothering you with tuning a bunch of buffers and things along those lines. For the sake of disclosure, I'm running 6 servers in a pretty high volume cluster, and while I've had to address some issues here and there, memory management hasn't been one of them.
suhail 16y ago
yay "Tab completion in the shell" and "B-tree index self-compaction" are great. So is --journal.
- rb2k_ 16y ago
  Do you happen to know details about the B-tree index self-compaction? I can't seem to find it in their changelog on jira ( http://jira.mongodb.org/browse/SERVER?report=com.atlassian.jira.plugin.system.project:changelog-panel http://jira.mongodb.org/browse/SERVER?report=com.atlassian.j... )
  - dm_mongodb 16y ago
    consolidation of adjacent btree nodes after key deletions when appropriate. arguably should have already done this, but it's there now!
deleted 16y ago
[deleted]
- baltcode 16y ago
  Even numbers after the period are good, odd are bad. Large numbers after second period are bad.
rb2k_ 16y ago
Can't wait until they start implementing filtered indexes ( http://jira.mongodb.org/browse/SERVER-785 http://jira.mongodb.org/browse/SERVER-785 ). Sparse indexes are a step in the right direction, but filtered ones would be just a bit cooler :) "New map/reduce options for incremental updates" would also be really cool if they had a way to do something like couchDBs incremental views. This would require keeping track of changes or a "trigger" functionality that runs the m/r task after every x inserts
rch 16y ago
This is exciting: db.users.mapReduce(map, reduce, {out: { inline : 1}}); This is Not exciting: "Note that this option is possible only when the result set fits within the 16MB limit of a single document."
- dacort 16y ago
  Not really a big deal to just output to a new collection and query that.
  - rch 16y ago
    Sure - I'm just pointing out something that could be a big deal (streaming results through an ordered structure in memory, that may or may not be backed by a lazy disk store) is actually not a big deal, due to a current implementation constraint. But you're right too, of course.
    - dacort 16y ago
      Yea, makes sense. That limit almost seems to fall along the lines of "Nobody should need more than 16MB for inline map/reduce results." ;)
      - rch 16y ago
        cheers to that.
- rit 16y ago
  I did a writeup, for what it's worth, on the new MapReduce output options: http://blog.evilmonkeylabs.com/2011/01/27/MongoDB-1_8-MapReduce/ http://blog.evilmonkeylabs.com/2011/01/27/MongoDB-1_8-MapRed... (Disclaimer, I work for 10gen / MongoDB)
  - rch 16y ago
    Very nice - worth the reading. I'm particularly interested in seeing how you've implemented 'merge' and 'reduce' output. You guys do seem to be headed in the right direction technically... I just can't bring myself to say "mongo" out loud.
davidw 16y ago
MongoDB is not something I have a good handle on yet. What's its sweet spot? Where should I consider using it instead of Postgres?
- yawniek 16y ago
  some of the sweet spots where it _might_ be better in than postgres are: - horizontal scalability - flexible datastructures - map/reduce
- kchodorow 16y ago
  A lot of people like it because it makes development faster. It's like the scripting language of databases: you can get stuff out the door really fast (with the obvious power/responsibility caveats).
  - DonnyV 16y ago
    This was the main reason why I picked it for a project. Its just sick the amount of code you DON'T have to write to use this database. I literally have 1 class called GenericCRUD with 5 functions that do all my database functions for all my models. Plus not having to worry about stored procedures anymore is a heavy weight lifted off your shoulders.
- andrewjshults 16y ago
  If you are working with location data, Mongo has built in geospatial indexing built in since 1.4 (earlier in the unstable builds) - http://www.mongodb.org/display/DOCS/Geospatial+Indexing http://www.mongodb.org/display/DOCS/Geospatial+Indexing which has been a big draw for a number of people I know using it (it looks like 1.8 brings spherical distances to the stable branch which makes the geo lookups a lot more useful if you need accurate distances and not just near by lookups).
  - igrekel 16y ago
    Interesting I was planning to toy with location data, do you know how it compares with Postgresql extension PostGIS?
gaius 16y ago
But is it web scale?
- gaius 16y ago
  I guess not!
- mrinterweb 16y ago
  I got in so much trouble for a very similar web scale comment. Never shall trite jokes be used on hacker news. You'll get your head bit off. Personally, it gives me a giggle, and I guess that means I am much less mature than most consumers of Hacker News. http://news.ycombinator.com/item?id=2104276 http://news.ycombinator.com/item?id=2104276
  - gaius 16y ago
    The thing you have to understand about the NoSQL crowd is there's no actual thinking here, just religious fervour.
  - kchodorow 16y ago
    I think it's just an old joke at this point. It's funny the first time you see it and the second time, but the 50th?
    - mrinterweb 16y ago
      I did admit the joke is trite.
dstorrs 16y ago
Could someone with Mongo experience help me gut-check this? I want my data store to be durable and unsurprising -- barring a hardware failure or such, if I submit data it should either tell me that it failed to commit or it should be stored durably and without surprises (e.g., it should not truncate a long string to fit). I've read some of the Mongo docco, and it's pretty exciting, but the lack of ACID -- primarily the Durability -- has kept me from really using it. With a WAL journal, it sounds like maybe the durability issue is fixed. Is it? Could I use Mongo with relatively out-of-the-box settings plus --journal and count on a level of durability equivalent to a traditional RDBMS?
- runevault 16y ago
  My understanding is this release is the first stable with single node durability available as an option, which is why I'm considering picking Mongo back up.
- joelhaasnoot 16y ago
  Don't know, I'm going to be testing this tomorrow, but came to work today to a server that was grinding to a halt due to wild running node.js processes. Restarted the server, but that doesn't cleanly shutdown Mongo, and spent an hour repairing everything and setting permissions right (somehow they had gotten reset). This exact problem, so hope the --journal switch in the init file makes the difference.
  - joelhaasnoot 16y ago
    Added "journal = true" to the config and it certainly stops and starts the service nicely now. No more repairs and such, my data is currently query only, so it's mainly "optical", cleaning up locks and such.
- dmytton 16y ago
  Yes, if you combine journaling with safe write operations. This allows you to call the appropriate driver method to confirm that the data has been written. You can also wait for it to be written to n slaves in replication. I believe that "written" means either to the data files or the journal. In the case of the journal, a hard crash would result in the journal being replayed so that the data is then written to disk and you don't get corruption. For example in the PHP driver, calling insert with the safe option. http://www.php.net/manual/en/mongocollection.insert.php http://www.php.net/manual/en/mongocollection.insert.php "If safe is an integer, will replicate the insert to that many machines before returning success (or throw an exception if the replication times out, see wtimeout)." You can then immediately called http://www.php.net/manual/en/mongodb.lasterror.php http://www.php.net/manual/en/mongodb.lasterror.php to confirm the last operation didn't error.
emef 16y ago
I was really hoping for full-text search support, but I understand the team is busy :/
- rabidsnail 16y ago
  All you need for full-text search is to add a multikey of tokens to the documents you want indexed. Tokenizers and stemmers are actually really easy to write, and there are libraries you can use to do that for you.
phsr 16y ago
As a newb to non-relational databases, but planning on learning one soon, what is the advantage of MongoDB vs Redis? I'm planning to use ruby with either, but was interested if there was a reason to pick one over the other.
- A1kmm 16y ago
  I don't think you should just learn one; they are all used for different niches. It depends how your app trades off different things (consistency, reliability of reads, reliability of writes, speed of reads, speed of writes, efficiency of hardware use, and so on). See http://news.ycombinator.com/item?id=2052852 http://news.ycombinator.com/item?id=2052852 for a comparison.
  - phsr 16y ago
    I'm not planning on just learning one, just trying to pick one to learn first
- effkay 16y ago
  That totally depends on what you want to do with Redis/MongoDB. Do you intend to use it as your primary data store? AFAIK One of MongoDB's goals is to be useful for a lot of things you'd normally use a relational database for. Redis on the other hand has a lot of nice things to handle more specialized cases of data.
- tmountain 16y ago
  They're really two different beasts. Mongo stores things in a manner similar to a relational DB (minus the relations). Think of it as a store for JSON that allows indexing and SQL-style queries using a JSON style syntax. Redis data structures are closer to what you'd find in computer science books (lists, sets, hashes, etc.). You should explore both and see what meets your needs.
- kchodorow 16y ago
  (I work on MongoDB, but trying to give a balanced opinion.) Redis is a great key-value store, MongoDB is more of a fully-featured database. Redis has some nice set operations and is pretty easy to learn (all of the commands are here: http://redis.io/commands http://redis.io/commands). MongoDB is also pretty easy to learn (click the "Try it out" button at http://mongodb.org/ http://mongodb.org/), but there are a lot of advanced features to learn about. So, if you need a key-value store, Redis is a great choice. If you want to do something more complex, MongoDB would probably work better.
mike_esspe 16y ago
Warning for those, who want to use MongoDB on FreeBSD: there is a known, unfixed bug, which locks database a lot: http://groups.google.com/group/mongodb-user/browse_thread/thread/95f9386cd57003e4 http://groups.google.com/group/mongodb-user/browse_thread/th... http://jira.mongodb.org/browse/SERVER-663 http://jira.mongodb.org/browse/SERVER-663
vegai 16y ago
The clustrix guy criticized mongodb that it locks the whole database quite often. Can anyone confirm or deny that?
- vegai 16y ago
  Modded to 0 and never answered. I'll take that as a confirmation.
eldenbishop 16y ago
Wow. I knew about the durability changes but I had no idea that sparse and covered indexes where coming. These three changes where the biggest drawbacks to mongo for me.

187 points by luigi 16y ago

Starting to get excited once again about MongoDB. I was kind of down about it after having some issues with real world implementations. Considering journaling is something I would never have thought would have made it in, I wonder if they will come around on the memory mapped I/O like everyone else eventually does. EDIT: Also... does the group commit mean that ALL write transactions will be un-acknowledged to the client until the group commit finishes?

mathias_10gen 16y ago
Clients are able to wait for the next group commit by adding fsync:true to getLastError calls (some drivers allow you to add this to WriteConcern). We already have some enhancements to this planned for the 1.9 series.
dmytton 16y ago
"You can wait for group commit acknowledgement with the getLastError command. When running with --journal, the fsync:true option returns after the data is physically written to the journal (rather than actually fsync'ing all the data files). Note that the group commit interval (see above) is considerable: you may prefer to call getLastError without fsync, or with a w: parameter instead with replication. In releases after 1.8.0 the delay for commit acknowledgement will be shorter." http://www.mongodb.org/display/DOCS/Journaling http://www.mongodb.org/display/DOCS/Journaling
- rbranson 16y ago
  RTFM'ing myself in public. While it's a great step forward, the whole thing sounds pretty immature at this point. It'll be a good day when this stuff matures and MongoDB has fsync:true and --journal by default, much to the chagrin of sensationalists everywhere. They'll really need to add some major smarts into the journaling and group commit if they're going to be able to stack up concurrent I/Os to help feed big disk arrays and even to get the best use out of SSDs which make back-and-forth latency on the I/O pipes even more significant.
tmountain 16y ago
I'm curious what you're getting at IRT memory mapped I/O. To me, one of Mongo's selling points is the way it lets the OS manage caching for you instead of bothering you with tuning a bunch of buffers and things along those lines. For the sake of disclosure, I'm running 6 servers in a pretty high volume cluster, and while I've had to address some issues here and there, memory management hasn't been one of them.

suhail 16y ago

yay "Tab completion in the shell" and "B-tree index self-compaction" are great. So is --journal.

rb2k_ 16y ago
Do you happen to know details about the B-tree index self-compaction? I can't seem to find it in their changelog on jira ( http://jira.mongodb.org/browse/SERVER?report=com.atlassian.jira.plugin.system.project:changelog-panel http://jira.mongodb.org/browse/SERVER?report=com.atlassian.j... )
- dm_mongodb 16y ago
  consolidation of adjacent btree nodes after key deletions when appropriate. arguably should have already done this, but it's there now!

deleted 16y ago

[deleted]

baltcode 16y ago
Even numbers after the period are good, odd are bad. Large numbers after second period are bad.

rb2k_ 16y ago

Can't wait until they start implementing filtered indexes ( http://jira.mongodb.org/browse/SERVER-785 http://jira.mongodb.org/browse/SERVER-785 ). Sparse indexes are a step in the right direction, but filtered ones would be just a bit cooler :) "New map/reduce options for incremental updates" would also be really cool if they had a way to do something like couchDBs incremental views. This would require keeping track of changes or a "trigger" functionality that runs the m/r task after every x inserts

rch 16y ago

This is exciting: db.users.mapReduce(map, reduce, {out: { inline : 1}}); This is Not exciting: "Note that this option is possible only when the result set fits within the 16MB limit of a single document."

dacort 16y ago
Not really a big deal to just output to a new collection and query that.
- rch 16y ago
  Sure - I'm just pointing out something that could be a big deal (streaming results through an ordered structure in memory, that may or may not be backed by a lazy disk store) is actually not a big deal, due to a current implementation constraint. But you're right too, of course.
  - dacort 16y ago
    Yea, makes sense. That limit almost seems to fall along the lines of "Nobody should need more than 16MB for inline map/reduce results." ;)
    - rch 16y ago
      cheers to that.
rit 16y ago
I did a writeup, for what it's worth, on the new MapReduce output options: http://blog.evilmonkeylabs.com/2011/01/27/MongoDB-1_8-MapReduce/ http://blog.evilmonkeylabs.com/2011/01/27/MongoDB-1_8-MapRed... (Disclaimer, I work for 10gen / MongoDB)
- rch 16y ago
  Very nice - worth the reading. I'm particularly interested in seeing how you've implemented 'merge' and 'reduce' output. You guys do seem to be headed in the right direction technically... I just can't bring myself to say "mongo" out loud.

davidw 16y ago

MongoDB is not something I have a good handle on yet. What's its sweet spot? Where should I consider using it instead of Postgres?

yawniek 16y ago
some of the sweet spots where it _might_ be better in than postgres are: - horizontal scalability - flexible datastructures - map/reduce
kchodorow 16y ago
A lot of people like it because it makes development faster. It's like the scripting language of databases: you can get stuff out the door really fast (with the obvious power/responsibility caveats).
- DonnyV 16y ago
  This was the main reason why I picked it for a project. Its just sick the amount of code you DON'T have to write to use this database. I literally have 1 class called GenericCRUD with 5 functions that do all my database functions for all my models. Plus not having to worry about stored procedures anymore is a heavy weight lifted off your shoulders.
andrewjshults 16y ago
If you are working with location data, Mongo has built in geospatial indexing built in since 1.4 (earlier in the unstable builds) - http://www.mongodb.org/display/DOCS/Geospatial+Indexing http://www.mongodb.org/display/DOCS/Geospatial+Indexing which has been a big draw for a number of people I know using it (it looks like 1.8 brings spherical distances to the stable branch which makes the geo lookups a lot more useful if you need accurate distances and not just near by lookups).
- igrekel 16y ago
  Interesting I was planning to toy with location data, do you know how it compares with Postgresql extension PostGIS?

gaius 16y ago

But is it web scale?

gaius 16y ago
I guess not!
mrinterweb 16y ago
I got in so much trouble for a very similar web scale comment. Never shall trite jokes be used on hacker news. You'll get your head bit off. Personally, it gives me a giggle, and I guess that means I am much less mature than most consumers of Hacker News. http://news.ycombinator.com/item?id=2104276 http://news.ycombinator.com/item?id=2104276
- gaius 16y ago
  The thing you have to understand about the NoSQL crowd is there's no actual thinking here, just religious fervour.
- kchodorow 16y ago
  I think it's just an old joke at this point. It's funny the first time you see it and the second time, but the 50th?
  - mrinterweb 16y ago
    I did admit the joke is trite.

dstorrs 16y ago

Could someone with Mongo experience help me gut-check this? I want my data store to be durable and unsurprising -- barring a hardware failure or such, if I submit data it should either tell me that it failed to commit or it should be stored durably and without surprises (e.g., it should not truncate a long string to fit). I've read some of the Mongo docco, and it's pretty exciting, but the lack of ACID -- primarily the Durability -- has kept me from really using it. With a WAL journal, it sounds like maybe the durability issue is fixed. Is it? Could I use Mongo with relatively out-of-the-box settings plus --journal and count on a level of durability equivalent to a traditional RDBMS?

runevault 16y ago
My understanding is this release is the first stable with single node durability available as an option, which is why I'm considering picking Mongo back up.
joelhaasnoot 16y ago
Don't know, I'm going to be testing this tomorrow, but came to work today to a server that was grinding to a halt due to wild running node.js processes. Restarted the server, but that doesn't cleanly shutdown Mongo, and spent an hour repairing everything and setting permissions right (somehow they had gotten reset). This exact problem, so hope the --journal switch in the init file makes the difference.
- joelhaasnoot 16y ago
  Added "journal = true" to the config and it certainly stops and starts the service nicely now. No more repairs and such, my data is currently query only, so it's mainly "optical", cleaning up locks and such.
dmytton 16y ago
Yes, if you combine journaling with safe write operations. This allows you to call the appropriate driver method to confirm that the data has been written. You can also wait for it to be written to n slaves in replication. I believe that "written" means either to the data files or the journal. In the case of the journal, a hard crash would result in the journal being replayed so that the data is then written to disk and you don't get corruption. For example in the PHP driver, calling insert with the safe option. http://www.php.net/manual/en/mongocollection.insert.php http://www.php.net/manual/en/mongocollection.insert.php "If safe is an integer, will replicate the insert to that many machines before returning success (or throw an exception if the replication times out, see wtimeout)." You can then immediately called http://www.php.net/manual/en/mongodb.lasterror.php http://www.php.net/manual/en/mongodb.lasterror.php to confirm the last operation didn't error.

emef 16y ago

I was really hoping for full-text search support, but I understand the team is busy :/

rabidsnail 16y ago
All you need for full-text search is to add a multikey of tokens to the documents you want indexed. Tokenizers and stemmers are actually really easy to write, and there are libraries you can use to do that for you.

phsr 16y ago

As a newb to non-relational databases, but planning on learning one soon, what is the advantage of MongoDB vs Redis? I'm planning to use ruby with either, but was interested if there was a reason to pick one over the other.

A1kmm 16y ago
I don't think you should just learn one; they are all used for different niches. It depends how your app trades off different things (consistency, reliability of reads, reliability of writes, speed of reads, speed of writes, efficiency of hardware use, and so on). See http://news.ycombinator.com/item?id=2052852 http://news.ycombinator.com/item?id=2052852 for a comparison.
- phsr 16y ago
  I'm not planning on just learning one, just trying to pick one to learn first
effkay 16y ago
That totally depends on what you want to do with Redis/MongoDB. Do you intend to use it as your primary data store? AFAIK One of MongoDB's goals is to be useful for a lot of things you'd normally use a relational database for. Redis on the other hand has a lot of nice things to handle more specialized cases of data.
tmountain 16y ago
They're really two different beasts. Mongo stores things in a manner similar to a relational DB (minus the relations). Think of it as a store for JSON that allows indexing and SQL-style queries using a JSON style syntax. Redis data structures are closer to what you'd find in computer science books (lists, sets, hashes, etc.). You should explore both and see what meets your needs.
kchodorow 16y ago
(I work on MongoDB, but trying to give a balanced opinion.) Redis is a great key-value store, MongoDB is more of a fully-featured database. Redis has some nice set operations and is pretty easy to learn (all of the commands are here: http://redis.io/commands http://redis.io/commands). MongoDB is also pretty easy to learn (click the "Try it out" button at http://mongodb.org/ http://mongodb.org/), but there are a lot of advanced features to learn about. So, if you need a key-value store, Redis is a great choice. If you want to do something more complex, MongoDB would probably work better.

mike_esspe 16y ago

Warning for those, who want to use MongoDB on FreeBSD: there is a known, unfixed bug, which locks database a lot: http://groups.google.com/group/mongodb-user/browse_thread/thread/95f9386cd57003e4 http://groups.google.com/group/mongodb-user/browse_thread/th... http://jira.mongodb.org/browse/SERVER-663 http://jira.mongodb.org/browse/SERVER-663

vegai 16y ago

The clustrix guy criticized mongodb that it locks the whole database quite often. Can anyone confirm or deny that?

vegai 16y ago
Modded to 0 and never answered. I'll take that as a confirmation.

eldenbishop 16y ago

Wow. I knew about the durability changes but I had no idea that sparse and covered indexes where coming. These three changes where the biggest drawbacks to mongo for me.