9 ms·
MongoDB’s new query engine
- romanovcode 3y agoSerious question: Why use MongoDB when Postgres supports indexed dynamic json?
- dzogchen 3y agoWhy use a non-FOSS database at all?
- maxloh 3y agoThere is FerretDB. But they are not fully compatible to Mongo yet. https://www.ferretdb.io/ https://www.ferretdb.io/
- romanovcode 3y agoAlso, but main point for MongoDB 10 years ago was the JSON. But now?
- malermeister 3y agoIsn't mongodb open source? https://github.com/mongodb/mongo https://github.com/mongodb/mongo Edit: ah it's source-available, not open source.
- JackMorgan 3y agoMy question exactly. No sense building a house on someone else's land. Too many projects I've worked on were shattered because of technology that stopped being offered or got too expensive to justify.
- nemo44x 3y agoBecause just about no one cares and I want to burn down a backlog quickly and often mongo enables that. Developer productivity is far more valuable that open source purity.
- KronisLV 3y ago> Developer productivity is far more valuable that open source purity. I mean, at least you get access to MongoDB source code, so that is something. I remember working on a project that used Clusterpoint years ago and the problem was that it was basically an abandoned piece of software due to the company behind it no longer releasing new versions. It was running, yes, but getting data out of it and migrating to something else sure took work and was very annoying because a lot of functionality had been built around it. So in regards to something so critical as databases and data storage solutions, I'd err on the side of picking things that are open source, or have compatible alternatives with permissive licenses. If someone were to build their own business around MongoDB (say, offering it as a service) and then SSPL came along, they'd be done for, even if just because of getting caught up in things going on between the org and large cloud vendors or something. Same with how you have to be careful with AGPL, so something like MinIO might be a non-starter in some cases.
- nemo44x 3y agoYeah if your goal is to offer it as a service then you’re out of luck. But if you plan on building anything else then you’re fine. Mongo is a thriving database so I’m not worried about it being abandon.
- KronisLV 3y ago> Mongo is a thriving database so I’m not worried about it being abandon. With this, I can totally agree! And as long as it's not AGPL, you should be fine using it for most regular development stuff, even for closed source projects.
- deleted 3y ago[deleted]
- riku_iki 3y ago> Developer productivity is far more valuable that open source purity. that's until provider starts abusing power: increasing rates, close source completely, etc.
- jamil7 3y agoIf you're only storing and querying JSON, then it's cheaper to run DynamoDB instead of RDS.
- salil999 3y agoThis is completely false and has nothing to do with the article. DynamoDB and MongoDB might both serve JSON but they're used for wildly different things. DynamoDB is a key value store whereas MongoDB is much more than that.
- jamil7 3y agoSorry you're right of course I messed up the service name.
- threeseed 3y agoMongoDB: a) has a proven, supported, easy-to-use horizontal scaling solution. PostgreSQL doesn't. b) is ridiculously faster than PostgreSQL at per-tuple document updates. c) has clients which are tailored for operations and data structures around documents. d) is easier to install, configure and manage.
- coldtea 3y agoI thought all those points died along with the hype ("MongoDB is web scale"), as all were wrong. Those were the marketing points people mentioned in their "honeymoon phase" posts. Then after using it in production, actual benchmarks and comparisons coming in, came the regret and moving on posts. In fact, most of those mentioned moving from Mongo to Postgres, and there was a full blow "yeah, NoSQL was a dumb idea for 99% of use cases, and Mongo even more so" discussion. In the end MongoDB was the butt of a joke, there were whole memes about it. So this comment is like a trip down memory lane, or into an alternate universe, where it's like 2012 and these things never happened.
- isoprophlex 3y agoYou're wrong. MongoDB is web scale. /s
- re-thc 3y agoBut is it PlanetScale?
- romanovcode 3y agoIt's not 2012 anymore ;)
- seedless-sensat 3y agoHow is (a) wrong? Pg doesn't horizontally scale like Mongo does
- re-thc 3y ago
- frankjr 3y agoI gave my answer a few years ago. The summary is that its query and update operations over documents are miles ahead of what PostgreSQL has to offer. https://news.ycombinator.com/item?id=23271085 https://news.ycombinator.com/item?id=23271085
- throwaway2990 3y ago3 year… almost 4 years ago you commented about a few years ago. So not really relevant. For the idiots downvoting. 2018 was version 11. Right when PostgreSQL JSONB support was beginning to get good.
- nemo44x 3y agoBecause the developer productivity is great because the language drivers that interface with the database are seamless with your code. MongoDB lets you burn down a backlog better than any other database. It’s not right for every problem but when it is right it’s the best tool because the language drivers are so good. That’s their secret sauce.
- Hypocritelefty 3y ago[dead]
- dubcanada 3y agoWhy is there no conversation on the topic at hand and instead conversation on why use MongoDB. This authors post has nothing to do with pro/cons of mongodb, and is entirely around a new query engine and computer science. And instead we've devolved into a discussion on the pros and cons of PostgreSQL vs MongoDB.
- christophilus 3y agoSame as it ever was. Someone posts a new JS framework, and the top 50 comments are about how bad JS is, and why don't we do RUST with WASM instead? Someone posts about a new Go release, and the top 50 comments are about how terrible Go is, and why don't we use a real language instead? Mongo... same thing. I get it. People don't like certain tools. But it would be nice for the rest of us to be able to actually discuss the topic at hand without all of the tangential critical noise.
- sambeau 3y agoIt's completely understandable. This is a significant and interesting upgrade to MongoDB that naturally gets people interested in a, "Hmmm… maybe I should take another look at MongoDB" way. Before wasting time, it seems sensible to use this forum full of experts to ask whether it would be worthwhile, and what the advantages or disadvantages would be. I didn't sense any negativity in the question, and it was the exact question I had in my head too. After skim-reading the article I went straight to scrolling down the comments to look for this exact question.
- hot_gril 3y agoThe original question was fine, asking what's the reason to use MongoDB. Like, it got a new query engine, please convince me to try using it / how does the new engine compare to Postgres. The sub-questions about FOSS and font choice are annoying, in fact I removed my replies on them because I didn't want to feed the problem.
- h1fra 3y agoI'm a postgres fan but I have to admit the JSON syntax is not the best and not well supported by all the tooling around this database. Doesn't mean it's bad but the DX could be better. Other than that, I have never seen a proper justification to use schema less db, good for prototyping and get started, not incredible as a long-term solution.
- synthmeat 3y agoSchema-less is a bit wrong term, since you always end up with one but, indeed, schema-on-read vs schema-on-write discussion for individual use cases is far from a settled thing, even though the zeitgeist is that schema-on-write won sometime in the 70s. You say "prototyping", but I would generalize that to "faster evolving" in the long term. Of course, not without tradeoffs. I personally have zero issues writing a custom marshaller when needed for any the schema-less document collections I have. Constraints your application has on the data are a superset of db schema anyways.
- synthmeat 3y agoJust to add to sibling comments, one killer feature for me is ChangeStreams[1]. It's miles ahead of what Postgres[2] offers, and it enables really interesting use cases. Some of my services built around ChangeStreams end up not doing a single query to the DB. Data is right there in the program memory, indexed how I need it to be, and program is immediately reactive to any changes in the DB. [1] https://www.mongodb.com/docs/manual/changeStreams/ https://www.mongodb.com/docs/manual/changeStreams/ [2] https://blog.sequin.io/all-the-ways-to-capture-changes-in-postgres/ https://blog.sequin.io/all-the-ways-to-capture-changes-in-po...
- Lolol00 3y agoI remember adding changestreams a few years back to my project, they were amazing .
- deleted 3y ago[deleted]
- splix 3y agoMongoDB is much easier to setup as a HA cluster
- heipei 3y agoSerious counter-question: How do you run Postgres in a native and hands-off replicated failover setup? With MongoDB you create a three-node replica set and you're done. I've yet to see a simple guide for Postgres. But I might be wrong, so happy to hear how you would achieve the same with Postgres.
- Too 3y agoThis isn't just a server property. It need to be implemented across the whole vertical, including client libraries and how queries are constructed in the application. With mongo it all is, and it's there out of the box. Client libraries are made to do this failover and retry under the hood. Queries are more often made as upserts instead of assuming transactions, eg ObjectId rather than auto increment and so on. Even non upserting operations like $inc are safely retried if handled by queries and not application code. It's magic to see it happen, with no extra code, the application just keeps running even when one server is down. Starting it up again and it reconciles with no hands-on. I would love to see a postgres that does this, because relational data is often more useful than the document model.
- masukomi 3y agoDear Geeks with interesting things to say on your blog: Please, for sake of the people you want to hear your words. DO NOT USE A SANS SERIF MONOSPACED FONT FOR ANYTHING BUT CODE. I get it. We stare at text (code) formatted in this kind of font all day long, and many of us find fonts that we truly enjoy. But, most of our monospace sans-serif fonts are designed to make sure individual characters aren't misread. We don't read prose the same way we read code. There is far more pattern recognition going on than actual parsing of individual letters, and monospaced fonts break that. We can debate the aesthetics of serifs but they actually do help provide context clues to the pattern recognition systems in our brain. Convincing you all to start using serif fonts on prose is not a battle i'm likely to win, but maybe I can convince you to only use monospaced fonts for your terminal, and your code. Please.
- colesantiago 3y agoJust use reader mode? All modern browsers have this.
- deleted 3y ago[deleted]
- weinzierl 3y agoYou exaggerate the issue. Proportional fonts were a necessity when lead, ink and paper were expensive. In lead typesetting they are easy to do. This caused the success of newspaper fonts like Times New Roman. Contemporary fonts have much less variance in the width of characters, except for a couple of outliers like the i. From there to a completely monospaced font is not as big a leap as you make it seem. For me, I'm fine with monospaced fonts for prose.
- TylerE 3y agoDisagree. Reading monospaced prose sucks big time. For those of us with a little bit of vision deficiency the vary character widths are helpful. Also, the reduced character width (as a monospaced font inevitable has to be spaced at what the widest chatacters, such as w, require, means that you have to use a smaller font size to get the same amount of info per scroll/line/page whatever, again compromising readability. Here's a comparison between the posted site, and the same site with the font size bumped up 4px, proportional (just browser default, I didn't cherry pick) and the ridiculously wide line spacing reduced. Even with all that, the easier to read version is quite a bit more compact. Could likely bump the font size 2 more px and still be smaller. https://i.imgur.com/DBFI2RU.png https://i.imgur.com/DBFI2RU.png
- salil999 3y agoThis is great and all but I'm curious on the performance improvements. I'm surprised there are no graphs or charts when show improvement in latency, CPU usage, disk reads, etc
- tjpnz 3y agoDoes MongoDB still lose data?
- salil999 3y agoProviding some context would be nice
- endisneigh 3y ago> Our most recent report on MongoDB 3.6.4 focused on causal consistency and linearizability in sharded collections. We found that sharded clusters appeared to offer linearizable reads, writes, and compare-and-set operations against single documents, so long as users ran with read concern linearizable and write concern majority. However, any weaker level of write concern resulted in the loss of committed writes. MongoDB’s default level of write concern was (and remains) acknowledgement by a single node, which means MongoDB may lose data by default. Although the write concern documentation does not make this clear, the rollback documentation states: https://jepsen.io/analyses/mongodb-4.2.6 https://jepsen.io/analyses/mongodb-4.2.6
- jabradoodle 3y agoAnother case of bad defaults, seems to have changed since 5.0.
- jayd16 3y agoThe backlash at the time was that some felt it wasn't just bad defaults but intentional benchmark juicing. Old news now, though.
- beembeem 3y agoThis was prior to the introduction of transactions years ago. The versions stated in your comment are past EOL and not offered in Atlas today. Not relevant to today's reality.
- 3y ago
- nevi-me 3y agoHeh, I missed that there's a new query engine. From [0] it looks like it was available from 5.1, also interesting is that you can't choose which engine to use, so I suppose it only works on subsets or queries that meet certain conditions. This isn't the first time I hear something to the effect of "LLVM JIT is great, but it introduces a lot of query latency". I wonder if there are other JIT engines more suitable for compiling potentially small/simple queries. [0] https://www.mongodb.com/docs/v7.0/reference/sbe/#std-label-sbe-landing https://www.mongodb.com/docs/v7.0/reference/sbe/#std-label-s...
- __s 3y agore LLVM: Yes, Cranelift was designed to address these issues https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/docs/compare-llvm.md https://github.com/bytecodealliance/wasmtime/blob/main/crane... https://blog.benj.me/2021/02/17/cranelift-codegen-primer https://blog.benj.me/2021/02/17/cranelift-codegen-primer https://jason-williams.co.uk/posts/a-possible-new-backend-for-rust https://jason-williams.co.uk/posts/a-possible-new-backend-fo... https://news.ycombinator.com/item?id=25130528 https://news.ycombinator.com/item?id=25130528 30% faster rust build time
- jandrewrogers 3y agoLLVM has many advantages for JIT but it does not prioritize very low latency, choosing to optimize other properties instead. Consequently, LLVM tends to be more popular for systems targeted at analytical workloads where queries commonly have intrinsically high overhead latency. The latency isn't terrible but it is noticeable if you are running a low-latency "fast-twitch" workload. There are other specialized JIT compilers that are essentially purpose-built to provide very low latency; for systems like MongoDB which are rarely used for serious analytical processing, you'd probably want to use one of these instead.
- matthewcford 3y agocan it sort by id?
- maxbond 3y agoI'm sure it can[1], do you mean can you get documents in insertion order? The default _id is a random right-leaning identifier[2], so sorting by it would only get you an approximate insertion order (except in a deployment with a single server). However as the linked documentation demonstrates you could supply a strictly incrementing ID, if your architecture abides. [1] https://www.mongodb.com/docs/manual/reference/operator/aggregation/sort/ https://www.mongodb.com/docs/manual/reference/operator/aggre... [2] https://www.mongodb.com/docs/manual/reference/method/ObjectId/ https://www.mongodb.com/docs/manual/reference/method/ObjectI...
- winrid 3y agono, that means you can sort on a max of 32 fields per sort request, like fieldOne, fieldTwo, fieldThree... There's no limit on number of sortable documents. Use an index to sort and it'll also be fast and use no extra memory. If you want to do an in memory sort there are memory limits, but you can also tell it to overflow to disk.
- maxbond 3y agoOh gotcha, thanks that makes sense. 32 did seem like a bizarrely low limit for how many documents you can sort, but as a number of fields it's plenty. I've removed that part of my comment. Appreciate the correction.
- bit_flipper 3y agoThis article seems to have inspired others to look at MongoDB again, so I'll give my thoughts after using it recently. MongoDB Atlas is a surprisingly good managed database product. I'm not a huge fan of someone else running my databases, but I think it might be the best one you can run across any cloud. If you like MongoDB (and, ignore the memes, there is a lot to like nowadays), and are OK paying a bit more to have someone run your database, I'd strongly consider Atlas.
- winrid 3y agoThe problem is when you grow. They can be really tough to work with on pricing. Also, their licensing does not allow servers past a certain size. Can you imagine Oracle telling the CIA they can't use servers with more than 256gb of ram? Just silly.
- bit_flipper 3y agoI'm not sure what experience you have, but I've run both their Enterprise licensed database on prem as well as migrated to Atlas and there have never been any licensing issues preventing vertical scaling of databases. One of our clusters on Atlas right now has machines larger than 256GB of RAM -- you're more limited by what your cloud vendor has available than Atlas.
- winrid 3y agoActually yeah for Atlas, I guess they automatically bill you as if it was 2-3x Enterprise Advanced licenses so there's no discussion. I thought it was the same as Enterprise Advanced but I guess not. With EA each unit above 256gb is billed as an additional license. See [0] [1] [2]. [0] https://www.mongodb.com/community/forums/t/for-mongodb-enterprise-advanced-subscriptions-the-unit-of-ram-for-a-server-is-up-to-256gb/238244 https://www.mongodb.com/community/forums/t/for-mongodb-enter... [1] https://www.mongodb.com/community/forums/t/for-mongodb-enterprise-advance-licensing/238537 https://www.mongodb.com/community/forums/t/for-mongodb-enter... [2] https://www.linkedin.com/pulse/mongodb-sizing-guide-sepp-renfer/ https://www.linkedin.com/pulse/mongodb-sizing-guide-sepp-ren...
- oron 3y agoI love Postgres and use it for different projects but for inboxes.com which has a very high insert rate coupled with auto delete by time stamp Mongo up until now has been very kind to me. We sometimes have 1000 incoming emails per second and high usage of our API and it just works.
- pphysch 3y agoDid you actually migrate from PostgreSQL to MongoDB for inboxes.com prod?