4 ms·

and out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it compl

and out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it completely shits the bed. putting hundreds of gigs of ram in a box isn't cheap. are the foursquare folks considering rewriting with a traditional datastore like postgres and some memcached in front of it?

hartror 16y ago
Err you don't . . as the article states their usage and load pattern does. If you want the speed of keeping the DB in RAM you need the RAM available whether it is mongo, MySQL/NDB or memcache. (-1 for not RTFA)
rit 16y ago
They migrated to MongoDB from Postgres. Specifically because they needed distributed sharding which Postgres didn't have. Jorge Ortiz, one of the developers at Foursquare said it best on twitter yesterday: "Baffled by the Mongo haters. If foursquare had stayed on Postgres, we would've had to write our own distributed sharding/balancing/indexing." "Two problems: 1) Our job is to build foursquare, not a database. 2) These things are hard. Odds are we would have had even more downtime." http://twitter.com/#!/jorgeortiz85/status/26563381834 http://twitter.com/#!/jorgeortiz85/status/26563381834 http://twitter.com/#!/jorgeortiz85/status/26563387808 http://twitter.com/#!/jorgeortiz85/status/26563387808 Which sums it up incredibly well. Your comment shows a scary lack of depth - this is the "dirty little secret" of ANY application that needs serious scalable data performance. You need to put your data in RAM where it is quickly accessible. As soon as you go to disk especially on a low I/O cloud box you are going to be in 'extremely slow' territory. This is why, even before the current NoSQL movement, everyone was using Memcached all over the place. Not because it was a fad but because they needed speed. SQL Databases have RAM caches as well which give you a pretty similar behavior. And big surprise - if you throw more memory at them - they run faster! Because they cache things in memory! In this case I believe that what Eliot was highlighting was that Foursquare's performance tolerance requirements were such that going to disk was not an optimal situation for them. There are plenty of applications out there using MongoDB without keeping it all in RAM; I've deployed and maintain several. The more memory you have certainly gives you better performance but it tends to be about a most frequently used cache. It never "Shit the bed" here to use your sophomoric vernacular. As the post states, it started having to go to the disk after it surpassed the memory threshold which slowed things down... nothing "crashed" as far as the description states. But a read/write queue backlog on any database is likely to exhibit the same behavior. Your postgres + memcached solution fixes what exactly? They would still need the same slabs of RAM to solve the problem, lest they go to disk on postgres and slow to a crawl the same way they did with MongoDB.

hartror 16y ago

Err you don't . . as the article states their usage and load pattern does. If you want the speed of keeping the DB in RAM you need the RAM available whether it is mongo, MySQL/NDB or memcache. (-1 for not RTFA)

rit 16y ago

They migrated to MongoDB from Postgres. Specifically because they needed distributed sharding which Postgres didn't have. Jorge Ortiz, one of the developers at Foursquare said it best on twitter yesterday: "Baffled by the Mongo haters. If foursquare had stayed on Postgres, we would've had to write our own distributed sharding/balancing/indexing." "Two problems: 1) Our job is to build foursquare, not a database. 2) These things are hard. Odds are we would have had even more downtime." http://twitter.com/#!/jorgeortiz85/status/26563381834 http://twitter.com/#!/jorgeortiz85/status/26563381834 http://twitter.com/#!/jorgeortiz85/status/26563387808 http://twitter.com/#!/jorgeortiz85/status/26563387808 Which sums it up incredibly well. Your comment shows a scary lack of depth - this is the "dirty little secret" of ANY application that needs serious scalable data performance. You need to put your data in RAM where it is quickly accessible. As soon as you go to disk especially on a low I/O cloud box you are going to be in 'extremely slow' territory. This is why, even before the current NoSQL movement, everyone was using Memcached all over the place. Not because it was a fad but because they needed speed. SQL Databases have RAM caches as well which give you a pretty similar behavior. And big surprise - if you throw more memory at them - they run faster! Because they cache things in memory! In this case I believe that what Eliot was highlighting was that Foursquare's performance tolerance requirements were such that going to disk was not an optimal situation for them. There are plenty of applications out there using MongoDB without keeping it all in RAM; I've deployed and maintain several. The more memory you have certainly gives you better performance but it tends to be about a most frequently used cache. It never "Shit the bed" here to use your sophomoric vernacular. As the post states, it started having to go to the disk after it surpassed the memory threshold which slowed things down... nothing "crashed" as far as the description states. But a read/write queue backlog on any database is likely to exhibit the same behavior. Your postgres + memcached solution fixes what exactly? They would still need the same slabs of RAM to solve the problem, lest they go to disk on postgres and slow to a crawl the same way they did with MongoDB.