3 ms·

Mongo works well as a straight-up JSON store for store-and-retrieve use cases (with no analytics). It is horizontally scalable, avoids the overhead of relationa

by wenc 8y ago

Mongo works well as a straight-up JSON store for store-and-retrieve use cases (with no analytics). It is horizontally scalable, avoids the overhead of relational databases, has indexing capabilities, and provides a strong consistency model. The big improvement came with WiredTiger, which addressed many of the issues that plagued earlier versions of Mongo. I've seen high-speed machine data stored in Mongo for logging and visualization purposes. It's an improvement over writing csv files to disk. However, if you ever need to perform non-trivial analytics, Mongo's weaknesses quickly become obvious. For machine learning, typically you would want to first ETL the data into a dataframe-like structure (which is a structure native to SQL databases).

threeseed 8y ago
Actually MongoDB is quite popular in the analytics space. It has a unique trick with Spark/Hadoop where its data gets represented as a single wide table. This allows you to use it as an analytical/ML feature store which is not possible with anything other than Cassandra. Also not sure where you get the idea dataframes are unique to SQL databases because that's completely wrong. HBase and Cassandra were even the original big data databases and they aren't relational. And Spark can manifest almost any database as a dataframe.
- nevi-me 8y ago
  I've had lots of fun using Mongo with Spark, I have found it pleasant, though most Spark libraries are good to use, including the JDBC ones. I haven't had much luck with the kind of viz where you don't first do ETL from Mongo to some tabular format.
- wenc 8y ago
  > not sure where you get the idea dataframes are unique to SQL databases I'm not sure I said this. > HBase and Cassandra were even the original big data databases and they aren't relational. They also had trouble doing joins and many other query operations which are common in analytics. Presto addresses this somewhat. > And Spark can manifest almost any database as a dataframe. Which entails a translation layer from whatever non-tabular form that data was (e.g. JSON) into a dataframe-like structure, rather keeping it in its native form, which reinforces my point. You still need to somehow transform data into tabular form. (ETL is just a batch way of doing this transformation; you can have live transformations of course, with accompanying overheads) BI tools also generally require data to be in tabular form, which entails the use of a translation layer. The Mongo BI connector is one such translator. > Actually MongoDB is quite popular in the analytics space. I work in this space, interact regularly with vendors, and monitor the space actively for strategic developments. This does not track with my observations.
davidp 8y ago
It's an improvement over writing csv files to disk. This is not a high bar.
hinkley 8y ago
But why not memcached or Redis? You are describing a situation that those are designed for, instead of something that is pretended for years to be what it never was.
- wenc 8y ago
  > you are describing a situation that those are designed for I'm curious: how so? memcached and Redis are commonly deployed for caching, but not for persistence (Redis has optional persistence). That is not to say you can't use them, but I'm not sure what makes them a necessarily better choice than Mongo in this situation (persisted high-speed machine data). Edit: OK, I see you mean the "for store-and-retrieve use cases" part. You have a point, though Mongo seems to be ok for that use case too.
  - hinkley 8y ago
    My life has been saner when we use the KV store for search or fast lookup and not as the system of record (for that we use a traditional database). It tends not to occur to me that durability is a feature people are looking for in nosql, but if you’re trying to avoid having three to five copies (SQL, nosql, search, reporting, ?). of your data I could understand. But as the team gets bigger it gets harder to maintain that, figuratively and literally.

by wenc 8y ago

threeseed 8y ago

Actually MongoDB is quite popular in the analytics space. It has a unique trick with Spark/Hadoop where its data gets represented as a single wide table. This allows you to use it as an analytical/ML feature store which is not possible with anything other than Cassandra. Also not sure where you get the idea dataframes are unique to SQL databases because that's completely wrong. HBase and Cassandra were even the original big data databases and they aren't relational. And Spark can manifest almost any database as a dataframe.

nevi-me 8y ago
I've had lots of fun using Mongo with Spark, I have found it pleasant, though most Spark libraries are good to use, including the JDBC ones. I haven't had much luck with the kind of viz where you don't first do ETL from Mongo to some tabular format.
wenc 8y ago
> not sure where you get the idea dataframes are unique to SQL databases I'm not sure I said this. > HBase and Cassandra were even the original big data databases and they aren't relational. They also had trouble doing joins and many other query operations which are common in analytics. Presto addresses this somewhat. > And Spark can manifest almost any database as a dataframe. Which entails a translation layer from whatever non-tabular form that data was (e.g. JSON) into a dataframe-like structure, rather keeping it in its native form, which reinforces my point. You still need to somehow transform data into tabular form. (ETL is just a batch way of doing this transformation; you can have live transformations of course, with accompanying overheads) BI tools also generally require data to be in tabular form, which entails the use of a translation layer. The Mongo BI connector is one such translator. > Actually MongoDB is quite popular in the analytics space. I work in this space, interact regularly with vendors, and monitor the space actively for strategic developments. This does not track with my observations.

davidp 8y ago

It's an improvement over writing csv files to disk. This is not a high bar.

hinkley 8y ago

But why not memcached or Redis? You are describing a situation that those are designed for, instead of something that is pretended for years to be what it never was.

wenc 8y ago
> you are describing a situation that those are designed for I'm curious: how so? memcached and Redis are commonly deployed for caching, but not for persistence (Redis has optional persistence). That is not to say you can't use them, but I'm not sure what makes them a necessarily better choice than Mongo in this situation (persisted high-speed machine data). Edit: OK, I see you mean the "for store-and-retrieve use cases" part. You have a point, though Mongo seems to be ok for that use case too.
- hinkley 8y ago
  My life has been saner when we use the KV store for search or fast lookup and not as the system of record (for that we use a traditional database). It tends not to occur to me that durability is a feature people are looking for in nosql, but if you’re trying to avoid having three to five copies (SQL, nosql, search, reporting, ?). of your data I could understand. But as the team gets bigger it gets harder to maintain that, figuratively and literally.