3 ms·

"To be honest, the decision to use MongoDb was an ill-thought out one. Lesson learned - thoroughly research any new technology you introduce into your stack, kn

"To be honest, the decision to use MongoDb was an ill-thought out one. Lesson learned - thoroughly research any new technology you introduce into your stack, know well the strengths and weaknesses thereof and evaluate honestly whether it fits your needs or not - no matter how much hype there is surrounding said technology." I think you are not alone in learning this lesson with this particular technology. Fortunately it's one I learned by proxy from working adjacent to a team that decided to introduce Mongo into their stack...but I still wake up and hear the sceams at night of "You have to put the whole dataset on RAM?"...you weren't there, man...we lost a lot of good guys... You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack".

mgkimsal 14y ago
One of the issues is that if you actually do this - evaluate, look at all sides, and decide 'shiny new tech' IS NOT right for your situation/project, you're branded as something not good. "Not a team player", "stick in the mud", "not able to keep up with the times", etc. I'm not suggesting everyone should be sticking with 1966 COBOL - times change, new tech comes up which makes sense to adopt. But not jumping on the shiny new tech bandwagon can have social consequences you need to be aware of.
stonemetal 14y ago
You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack". Even if you have a clean line, when do you promote something across that line? It is easy for little bits of weirdness to escape detection(Mostly thinking about Cassandra write failures here.) in a fun to play with environment especially if they only come out when you are running a cluster.
trafficlight 14y ago
>> "You have to put the whole dataset on RAM?" I'm pretty new to the whole database thing, but how is MongoDB different from Postgres or Mysql in this respect? In a traditional database, the data is pulled directly from the hard drive. Why does Mongo suffer a performance hit and Mysql doesn't?
- otterley 14y ago
  Because MongoDB mmap(2)s its backing stores into its process memory space. It's a naive approach to persistence - it's very fast and simple, but if you overcommit (i.e. you store more in the database than you have memory available), page-thrashing results. MySQL's InnoDB table engine, on the other hand, uses direct I/O (in the recommended scenario) and manages the buffer pool independently of the kernel. Its buffer pool manager is specifically designed for the typical workloads MySQL is used for (http://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.html http://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.ht...) - as opposed to the naive LRU that most OSes employ for their filessytem buffers.
  - trafficlight 14y ago
    Ah, ok. That makes a lot more sense. Thank you.
  - taligent 14y ago
    It's not naive. It's a well understood and communicated design choice. Remember that completely in memory databases are going to be how we all store our data in the decades to come and are already the standard for those who care about speed.
  - einhverfr 14y ago
    For what it's worth VoltDB uses main memory format as its "naive" approach to persistance too. I would consider VoltDB naive in the approach to persistance because it gives you dramatic gains (2 orders of magnitude) in read/write performance for similar workloads only by redefining concurrency and durability out of the equation. Durability is no longer a property of each machine but is instead a property of the network. Concurrency is handled by executing queries fast and doing them in series. So it's like ACID without the C or D.
  - jeremyjh 14y ago
    Its hardly naive, it is optimized for an application that needs to keep its entire working set in RAM, that is why sharding is so fundamental to the design. Not all apps need that which is once again...
lttlrck 14y ago
"You have to put the whole dataset on RAM?" This is hardly a hidden feature...
lttlrck 14y ago
"You have to put the whole dataset on RAM?" This is hardly a hidden feature...

mgkimsal 14y ago

One of the issues is that if you actually do this - evaluate, look at all sides, and decide 'shiny new tech' IS NOT right for your situation/project, you're branded as something not good. "Not a team player", "stick in the mud", "not able to keep up with the times", etc. I'm not suggesting everyone should be sticking with 1966 COBOL - times change, new tech comes up which makes sense to adopt. But not jumping on the shiny new tech bandwagon can have social consequences you need to be aware of.

stonemetal 14y ago

You have to draw a clean line between "stuff it is really fun and enlightening to play with" and "stuff you introduce into your stack". Even if you have a clean line, when do you promote something across that line? It is easy for little bits of weirdness to escape detection(Mostly thinking about Cassandra write failures here.) in a fun to play with environment especially if they only come out when you are running a cluster.

trafficlight 14y ago

>> "You have to put the whole dataset on RAM?" I'm pretty new to the whole database thing, but how is MongoDB different from Postgres or Mysql in this respect? In a traditional database, the data is pulled directly from the hard drive. Why does Mongo suffer a performance hit and Mysql doesn't?

otterley 14y ago
Because MongoDB mmap(2)s its backing stores into its process memory space. It's a naive approach to persistence - it's very fast and simple, but if you overcommit (i.e. you store more in the database than you have memory available), page-thrashing results. MySQL's InnoDB table engine, on the other hand, uses direct I/O (in the recommended scenario) and manages the buffer pool independently of the kernel. Its buffer pool manager is specifically designed for the typical workloads MySQL is used for (http://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.html http://dev.mysql.com/doc/refman/5.5/en/innodb-buffer-pool.ht...) - as opposed to the naive LRU that most OSes employ for their filessytem buffers.
- trafficlight 14y ago
  Ah, ok. That makes a lot more sense. Thank you.
- taligent 14y ago
  It's not naive. It's a well understood and communicated design choice. Remember that completely in memory databases are going to be how we all store our data in the decades to come and are already the standard for those who care about speed.
- einhverfr 14y ago
  For what it's worth VoltDB uses main memory format as its "naive" approach to persistance too. I would consider VoltDB naive in the approach to persistance because it gives you dramatic gains (2 orders of magnitude) in read/write performance for similar workloads only by redefining concurrency and durability out of the equation. Durability is no longer a property of each machine but is instead a property of the network. Concurrency is handled by executing queries fast and doing them in series. So it's like ACID without the C or D.
- jeremyjh 14y ago
  Its hardly naive, it is optimized for an application that needs to keep its entire working set in RAM, that is why sharding is so fundamental to the design. Not all apps need that which is once again...

lttlrck 14y ago

"You have to put the whole dataset on RAM?" This is hardly a hidden feature...

lttlrck 14y ago

"You have to put the whole dataset on RAM?" This is hardly a hidden feature...