4 ms·

As with anything, it depends on the project. I’m working on an internal service that uses Mongo as a single merged cache for a lot of mostly unchanging data fro

As with anything, it depends on the project. I’m working on an internal service that uses Mongo as a single merged cache for a lot of mostly unchanging data from various data stores with different credentials for each, which are distributed around the world, that we otherwise have to fetch through multiple comparatively slow API calls. For this, Mongo is perfect: no messing with schemas as they change, unannounced, from upstream; I can index just the fields I want to search on; Mongo will expire things for me on a TTL; the query API is simpler than the API we’re caching from; and we get results 20-50x faster. We looked into FoundationDB and Postgres, but they require a lot more initial setup. ElasticSearch is the closest solution, but it needs a lot more info about the schema up front and its query language is a nightmare compared to Mongo’s, for no real gain in functionality that I can see. Is Mongo the right tool to build your entire business on top of? Probably not, but it can be the best tool for the right job.

tgtweak 8y ago
You and I must be the only ones on HN using mongo at scale and enjoying it. We use it as an event database which collects over 100M+ semi-structured records daily with about 200 (and growing) different schemas... It keeps 1.5TB of records in the collection which is achieved using the invaluable capped collection function, and we can index the structured fields very easily. We also pipe the data into elastic for quick kibana querying but just that step requires a lot of index partitioning and mapping to work smoothly. I also feel that a significant amount of mongo's power lies in the aggregation pipeline which is often ignored and which can replace entire ETLs, sending the compute to the data can be much more efficient in a lot of use cases. All this runs for us on some relatively cheap (compared to 1.5TB of high i/o RDS) commodity hardware, running 3 replicated nodes. More complex ETL jobs or Hadoop users can pull data from mongo selectively and quickly - spark has a way to partition a query into N threads based on a provided key achieving line rate data pulls even from a single node. Throw in compression on disk and in-flight and you get something that is really compelling. We've all run out of disk on a database before, it's never fun. My one gripe is the learning curve for new developers. I do find that once leveraged, I see a lot more data being stored when it's as simple as nesting an object into your main object and saving it, which data teams like. Depends on your use cases, but really, it's been an invaluable tool so far.
- tinco 8y ago
  My project as well, sharded over 4 nodes, ingesting about a TB every two weeks. Automatic deletion of old data was too slow so we had to work with a scheme that allowed us to drop daily collections, but beside that it ran great. This was 6 to 3 years ago though. Maybe there's a better log stash out there now? I haven't seen one yet.
  - tracker1 8y ago
    ElasticSearch does most of what you are saying in the box, can even have each day read into a different collection and query them on joined aliases. At least IIRC, I'm not an expert. When I first tried using it and Mongo, I had issues with ES geo indexing, but since then have used both with little issue. Just depends on the scenario.
    - tgtweak 8y ago
      Latest ES supports purging expiry based on a time field I believe, but before that you were creating daily/hourly indexes and haggling with elastic curator to delete them. There is also no meaningful way to do it by logical size. Elastic is great in it's own sense, but it's not as flexible as mongo when it comes to schemas.
      - tracker1 8y ago
        Agreed... I didn't know it allowed for time expiry now, been a couple years. I just find that at least half the time where I would consider Mongo today, ES seems to be a better fit. If PG was nearly as easy to setup replication hot/auto fail over, I'd probably favor that 95% of the time. I really wish RethinkDB had been as successful with marketing as Mongo though.
james-mcelwain 8y ago
Elasticsearch can index documents dynamically, and doesn't require a schema to create an index. Dynamic data types for fields may not always produce what you want, but it's possible to define a partial schema for the fields that are important and let Elasticsearch handle the rest. The query language is verbose but I would hesitate to call it a nightmare. You can always search using the Lucene query language, and SQL support is landing sometime soon.
- AznHisoka 8y ago
  The query language can be a bit quirky if you're coming from a SQL background, but you're absolutely right. It is nowhere close to a nightmare.
- pmarreck 8y ago
  My last project required fulltext search and I was going to go with Elasticsearch but being dependency-averse (also complicates the deployment story), I ended up using Postgres' built-in fulltext search capabilities and it actually works really nice, especially after I added a little DSL to take advantage of the full power of `to_tsquery` (so I could avoid the oversimplified `plainto_tsquery`)
duncan-donuts 8y ago
I used mongo years ago with pretty much the exact same use case and it was awesome. I moved companies a few years ago and they were using mongo as well, but for services that were 95% relational. It was awful. We no longer use mongo for anything
godber 8y ago
One benefit to ES is scale. Elasticsearch can scale to at least a couple trillion records (on the order of 1.5 PB) if you spread your data out across enough hardware. You can then search those records in seconds (for simple queries). I'm not sure Mongo can scale quite that far, it was given up on before I started this work. Not saying it can't, just saying others had more success scaling ES and haven't really hit its limits. Bear in mind datasets span multiple clusters and are searches are merged using cross cluster search in these larger cases. Not likely to be an apples to apples comparison.
HillRat 8y ago
I think you’ve got the right use case. Personally I’ve never found NoSQL to be superior, or even equal, to relational databases for general product development, but with Mongo’s recent improvements (fixing the data loss bugs, adding multidoc TXs), it’s become a pretty good platform for consuming system-of-record data and making it rapidly available. Great for customer 360 or CDP scenarios where you need to rapidly analyze personalization or segmentation data and act on it.

tgtweak 8y ago

You and I must be the only ones on HN using mongo at scale and enjoying it. We use it as an event database which collects over 100M+ semi-structured records daily with about 200 (and growing) different schemas... It keeps 1.5TB of records in the collection which is achieved using the invaluable capped collection function, and we can index the structured fields very easily. We also pipe the data into elastic for quick kibana querying but just that step requires a lot of index partitioning and mapping to work smoothly. I also feel that a significant amount of mongo's power lies in the aggregation pipeline which is often ignored and which can replace entire ETLs, sending the compute to the data can be much more efficient in a lot of use cases. All this runs for us on some relatively cheap (compared to 1.5TB of high i/o RDS) commodity hardware, running 3 replicated nodes. More complex ETL jobs or Hadoop users can pull data from mongo selectively and quickly - spark has a way to partition a query into N threads based on a provided key achieving line rate data pulls even from a single node. Throw in compression on disk and in-flight and you get something that is really compelling. We've all run out of disk on a database before, it's never fun. My one gripe is the learning curve for new developers. I do find that once leveraged, I see a lot more data being stored when it's as simple as nesting an object into your main object and saving it, which data teams like. Depends on your use cases, but really, it's been an invaluable tool so far.

tinco 8y ago
My project as well, sharded over 4 nodes, ingesting about a TB every two weeks. Automatic deletion of old data was too slow so we had to work with a scheme that allowed us to drop daily collections, but beside that it ran great. This was 6 to 3 years ago though. Maybe there's a better log stash out there now? I haven't seen one yet.
- tracker1 8y ago
  ElasticSearch does most of what you are saying in the box, can even have each day read into a different collection and query them on joined aliases. At least IIRC, I'm not an expert. When I first tried using it and Mongo, I had issues with ES geo indexing, but since then have used both with little issue. Just depends on the scenario.
  - tgtweak 8y ago
    Latest ES supports purging expiry based on a time field I believe, but before that you were creating daily/hourly indexes and haggling with elastic curator to delete them. There is also no meaningful way to do it by logical size. Elastic is great in it's own sense, but it's not as flexible as mongo when it comes to schemas.
    - tracker1 8y ago
      Agreed... I didn't know it allowed for time expiry now, been a couple years. I just find that at least half the time where I would consider Mongo today, ES seems to be a better fit. If PG was nearly as easy to setup replication hot/auto fail over, I'd probably favor that 95% of the time. I really wish RethinkDB had been as successful with marketing as Mongo though.

james-mcelwain 8y ago

Elasticsearch can index documents dynamically, and doesn't require a schema to create an index. Dynamic data types for fields may not always produce what you want, but it's possible to define a partial schema for the fields that are important and let Elasticsearch handle the rest. The query language is verbose but I would hesitate to call it a nightmare. You can always search using the Lucene query language, and SQL support is landing sometime soon.

AznHisoka 8y ago
The query language can be a bit quirky if you're coming from a SQL background, but you're absolutely right. It is nowhere close to a nightmare.
pmarreck 8y ago
My last project required fulltext search and I was going to go with Elasticsearch but being dependency-averse (also complicates the deployment story), I ended up using Postgres' built-in fulltext search capabilities and it actually works really nice, especially after I added a little DSL to take advantage of the full power of `to_tsquery` (so I could avoid the oversimplified `plainto_tsquery`)

duncan-donuts 8y ago

I used mongo years ago with pretty much the exact same use case and it was awesome. I moved companies a few years ago and they were using mongo as well, but for services that were 95% relational. It was awful. We no longer use mongo for anything

godber 8y ago

One benefit to ES is scale. Elasticsearch can scale to at least a couple trillion records (on the order of 1.5 PB) if you spread your data out across enough hardware. You can then search those records in seconds (for simple queries). I'm not sure Mongo can scale quite that far, it was given up on before I started this work. Not saying it can't, just saying others had more success scaling ES and haven't really hit its limits. Bear in mind datasets span multiple clusters and are searches are merged using cross cluster search in these larger cases. Not likely to be an apples to apples comparison.

HillRat 8y ago

I think you’ve got the right use case. Personally I’ve never found NoSQL to be superior, or even equal, to relational databases for general product development, but with Mongo’s recent improvements (fixing the data loss bugs, adding multidoc TXs), it’s become a pretty good platform for consuming system-of-record data and making it rapidly available. Great for customer 360 or CDP scenarios where you need to rapidly analyze personalization or segmentation data and act on it.