4 ms·

Wait, so MongoDB's MapReduce is so slow that they wrote their own MR framework in Python that pipes the data across the network and back?

mrkurt 15y ago
MongoDB's map reduce is pretty janky, although it's slightly less janky in MongoDB 2.0. I'm not convinced that their home grown m/r is a better option. They mentioned the "global javascript interpreter lock", which may be slightly inaccurate, but it's true that you can only run one m/r job per mongod instance at a time. It's relatively easy to get around that, though, since you can have replica sets and run jobs on the slave members. You can even have small replica set instances that only do m/r. There are a couple of other problems, one is that Mongo writes map output into a temporary collection. There's also entirely too much BSON <-> Javascript converting going on. Mongo 2 addresses some of this with the JS Only flag, but it's limited to something like half a million inputs at a time: http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-jsModeflag http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-jsMo... If I ran into intractable map reduce problems with MongoDB, I'd probably just jump to Riak or something. My guess is that they'll continue to improve things in Mongo land at roughly the same pace as I run into problems, though.
- nosh 15y ago
  There are a few projects in the works to improve the experience for doing aggregation and analysis with data stored in MongoDB: - Using V8 instead of spidermonkey [Will probably lead to better performance for MongoDB's map reduce] https://jira.mongodb.org/browse/SERVER-2407 https://jira.mongodb.org/browse/SERVER-2407 - New aggregation framework [Will make it easy and fast to do simple aggregations] https://jira.mongodb.org/browse/SERVER-447 https://jira.mongodb.org/browse/SERVER-447 There is also Hadoop adapter that read and write directly from/to MongoDB that is in an early stage https://github.com/mongodb/mongo-hadoop https://github.com/mongodb/mongo-hadoop
  - rbranson 15y ago
    Switching to V8 seems like a red herring. Sounds like the current issues are with the architecture, not with the interpreter performance.
    - nosh 15y ago
      true to an extent (about performance), which is why I used the qualifier 'probably'. I should have said may instead! I think Eliot's comment sums it up pretty well: https://jira.mongodb.org/browse/SERVER-2407?focusedCommentId=52733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-52733 https://jira.mongodb.org/browse/SERVER-2407?focusedCommentId... @rbranson - if you have thoughts on ways to improve, please do add them to http://jira.mongodb.org http://jira.mongodb.org :)

mrkurt 15y ago

MongoDB's map reduce is pretty janky, although it's slightly less janky in MongoDB 2.0. I'm not convinced that their home grown m/r is a better option. They mentioned the "global javascript interpreter lock", which may be slightly inaccurate, but it's true that you can only run one m/r job per mongod instance at a time. It's relatively easy to get around that, though, since you can have replica sets and run jobs on the slave members. You can even have small replica set instances that only do m/r. There are a couple of other problems, one is that Mongo writes map output into a temporary collection. There's also entirely too much BSON <-> Javascript converting going on. Mongo 2 addresses some of this with the JS Only flag, but it's limited to something like half a million inputs at a time: http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-jsModeflag http://www.mongodb.org/display/DOCS/MapReduce#MapReduce-jsMo... If I ran into intractable map reduce problems with MongoDB, I'd probably just jump to Riak or something. My guess is that they'll continue to improve things in Mongo land at roughly the same pace as I run into problems, though.

nosh 15y ago
There are a few projects in the works to improve the experience for doing aggregation and analysis with data stored in MongoDB: - Using V8 instead of spidermonkey [Will probably lead to better performance for MongoDB's map reduce] https://jira.mongodb.org/browse/SERVER-2407 https://jira.mongodb.org/browse/SERVER-2407 - New aggregation framework [Will make it easy and fast to do simple aggregations] https://jira.mongodb.org/browse/SERVER-447 https://jira.mongodb.org/browse/SERVER-447 There is also Hadoop adapter that read and write directly from/to MongoDB that is in an early stage https://github.com/mongodb/mongo-hadoop https://github.com/mongodb/mongo-hadoop
- rbranson 15y ago
  Switching to V8 seems like a red herring. Sounds like the current issues are with the architecture, not with the interpreter performance.
  - nosh 15y ago
    true to an extent (about performance), which is why I used the qualifier 'probably'. I should have said may instead! I think Eliot's comment sums it up pretty well: https://jira.mongodb.org/browse/SERVER-2407?focusedCommentId=52733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-52733 https://jira.mongodb.org/browse/SERVER-2407?focusedCommentId... @rbranson - if you have thoughts on ways to improve, please do add them to http://jira.mongodb.org http://jira.mongodb.org :)