5 ms·

The project isn't about performance, it's about providing an embedded version of MongoDB - it clearly states that if you grow too large, then you can easily mig

The project isn't about performance, it's about providing an embedded version of MongoDB - it clearly states that if you grow too large, then you can easily migrate to the full MongoDB. It also clearly states not to use it if you want a relational database.

remram 6y ago
Then the comparison with SQLite is totally wrong. SQLite is a production-grade DBMS: though it has fewer features compared to e.g. PostgreSQL, if those feature suffice, you can throw very sizable workloads at it.
- tracker1 6y ago
  Came to say largely the same... especially considering this is really Python only, where SQLite is available pretty much anywhere you run code.
- deleted 6y ago
  [deleted]
- nerdponx 6y ago
  The comparison is apt in that it's a library, not a separate server, and it can store data in memory, which is normally only available in Mongo enterprise.
scottrogowski 6y ago
Author here. That's exactly it. I think it would be very difficult to completely beat SQLite with a Python library. My goal wasn't to beat it but to have performance that's within an order of magnitude which I think I've achieved. In my opinion, the MongoDB interface has a lot of advantages over SQL that make sense in a lot of use cases. Certainly, there are times when a traditional relational database is the right choice. But I do think Mongita fills a niche. Separately, the JSON1 extension, which the top-level comment refers to, is nice technically but has a challenging interface IMHO https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html
- nerdponx 6y ago
  Everyone is so overly critical in this thread. This is an excellent idea and your library is proof of concept. Surely one day performance can be improved in the future, and the core logic can be rewritten in a compiled language that can be used from other runtimes.
  - bachmeier 6y ago
    > Everyone is so overly critical in this thread. The main criticism seems to be the flawed connection to sqlite. As useful as Mongita might be, and I don't think anyone is denying that this is good work, it doesn't make sense to compare it to sqlite.
    - pbreit 6y ago
      Comparison seems entirely reasonable to me.
- kevincox 6y ago
  Have you considered using SQLite (and probably JSON1) as the backend and just making an API that amps from the Mongo API to SQLite? It seems like this would probably have better read performance and the battle-tested reliability that we love from SQLite.
  - scottrogowski 6y ago
    Yup. Considered it. I initially settled on no because it might compromise the design goal of 1:1 MongoDB/PyMongo compatibility. Now the idea is being revived as an option alongside the two existing backends. I am concerned there might be an issue with some square-peg high level MongoDB operations simply not fitting into the SQLite round-hole but it's certainly something that's being investigated.
saurik 6y ago
I am not even sure what is going on now, as I thought the only reason people had ever cared about MongoDB in the first place was in an attempt to get performance they thought was impossible using a relational database (notably without realizing all of the tradeoffs... some inexcusable, such as simply coming with defaults that didn't call fsync, leading to a ton of memes about "database administrators running with scissors" and years of having to convince people that MongoDB was only a project you use if you don't care about data). The irony, of course, was that it turned out that if you used PostgreSQL and simply stored json in it, you could get better performance without giving up on a relational database... and even more damning was that the translation from MongoDB's query syntax to PostgreSQL's query syntax was trivial, leading to people building adapter layers that were drop-in compatible (and yet still faster and safer than MongoDB). So I guess there was something fitting about seeing someone post a project that is trying to the SQLite of MongoDB, but with benchmarks right off the bar showing it slower than SQLite ;P. I thereby continue to feel that if you want to use MongoDB, but want a library version of it, it would seem--based on this project's own benchmarks, unless you are doing a read-heavy workload of random documents by index--that what you probably want is a query translator to go from MongoDB's syntax to SQL (working with the SQLite json API), and then store your workload in SQLite... which is (critically) a battle-tested production-grade database engine used as a foundational storage layer for lots of projects. That said, it isn't like SQLite was a drop-in replacement for other database engines: when I think of "X is to SQLite as MongoDB is to PostgreSQL" I picture something that is attempting to being benefits over SQLite -- in way of performance or scalability -- at the cost of losing the power of being a full relational database (and, because of Mongo's legacy, probably a lot of safety and stability guarantees ;P). (FWIW, I remembered there's being a project like this: UnQLite-- https://news.ycombinator.com/item?id=18101689 https://news.ycombinator.com/item?id=18101689 --where people ironically seem to have wanted to get it benchmarked against SQLite with its json API ;P.) It sounds like maybe this project is just trying to provide MongoDB's query layer? You simply don't use SQLite until you "migrate to the full PostgreSQL"... they are designed for different scenarios, and while SQLite is good enough you might be able to use it in a place where PostgreSQL "was called for", a migration might be brutal (as the syntax and type system are different). The project tagline is thereby leading to the wrong mental space--as seemingly multiple other people have since now mentioned on this thread--particularly given that it is written in Python.
jbverschoor 6y ago
Then why not just create a library which uses sqlite as a proven storage and query engine?

remram 6y ago

Then the comparison with SQLite is totally wrong. SQLite is a production-grade DBMS: though it has fewer features compared to e.g. PostgreSQL, if those feature suffice, you can throw very sizable workloads at it.

tracker1 6y ago
Came to say largely the same... especially considering this is really Python only, where SQLite is available pretty much anywhere you run code.
deleted 6y ago
[deleted]
nerdponx 6y ago
The comparison is apt in that it's a library, not a separate server, and it can store data in memory, which is normally only available in Mongo enterprise.

scottrogowski 6y ago

Author here. That's exactly it. I think it would be very difficult to completely beat SQLite with a Python library. My goal wasn't to beat it but to have performance that's within an order of magnitude which I think I've achieved. In my opinion, the MongoDB interface has a lot of advantages over SQL that make sense in a lot of use cases. Certainly, there are times when a traditional relational database is the right choice. But I do think Mongita fills a niche. Separately, the JSON1 extension, which the top-level comment refers to, is nice technically but has a challenging interface IMHO https://www.sqlite.org/json1.html https://www.sqlite.org/json1.html

nerdponx 6y ago
Everyone is so overly critical in this thread. This is an excellent idea and your library is proof of concept. Surely one day performance can be improved in the future, and the core logic can be rewritten in a compiled language that can be used from other runtimes.
- bachmeier 6y ago
  > Everyone is so overly critical in this thread. The main criticism seems to be the flawed connection to sqlite. As useful as Mongita might be, and I don't think anyone is denying that this is good work, it doesn't make sense to compare it to sqlite.
  - pbreit 6y ago
    Comparison seems entirely reasonable to me.
kevincox 6y ago
Have you considered using SQLite (and probably JSON1) as the backend and just making an API that amps from the Mongo API to SQLite? It seems like this would probably have better read performance and the battle-tested reliability that we love from SQLite.
- scottrogowski 6y ago
  Yup. Considered it. I initially settled on no because it might compromise the design goal of 1:1 MongoDB/PyMongo compatibility. Now the idea is being revived as an option alongside the two existing backends. I am concerned there might be an issue with some square-peg high level MongoDB operations simply not fitting into the SQLite round-hole but it's certainly something that's being investigated.

saurik 6y ago

I am not even sure what is going on now, as I thought the only reason people had ever cared about MongoDB in the first place was in an attempt to get performance they thought was impossible using a relational database (notably without realizing all of the tradeoffs... some inexcusable, such as simply coming with defaults that didn't call fsync, leading to a ton of memes about "database administrators running with scissors" and years of having to convince people that MongoDB was only a project you use if you don't care about data). The irony, of course, was that it turned out that if you used PostgreSQL and simply stored json in it, you could get better performance without giving up on a relational database... and even more damning was that the translation from MongoDB's query syntax to PostgreSQL's query syntax was trivial, leading to people building adapter layers that were drop-in compatible (and yet still faster and safer than MongoDB). So I guess there was something fitting about seeing someone post a project that is trying to the SQLite of MongoDB, but with benchmarks right off the bar showing it slower than SQLite ;P. I thereby continue to feel that if you want to use MongoDB, but want a library version of it, it would seem--based on this project's own benchmarks, unless you are doing a read-heavy workload of random documents by index--that what you probably want is a query translator to go from MongoDB's syntax to SQL (working with the SQLite json API), and then store your workload in SQLite... which is (critically) a battle-tested production-grade database engine used as a foundational storage layer for lots of projects. That said, it isn't like SQLite was a drop-in replacement for other database engines: when I think of "X is to SQLite as MongoDB is to PostgreSQL" I picture something that is attempting to being benefits over SQLite -- in way of performance or scalability -- at the cost of losing the power of being a full relational database (and, because of Mongo's legacy, probably a lot of safety and stability guarantees ;P). (FWIW, I remembered there's being a project like this: UnQLite-- https://news.ycombinator.com/item?id=18101689 https://news.ycombinator.com/item?id=18101689 --where people ironically seem to have wanted to get it benchmarked against SQLite with its json API ;P.) It sounds like maybe this project is just trying to provide MongoDB's query layer? You simply don't use SQLite until you "migrate to the full PostgreSQL"... they are designed for different scenarios, and while SQLite is good enough you might be able to use it in a place where PostgreSQL "was called for", a migration might be brutal (as the syntax and type system are different). The project tagline is thereby leading to the wrong mental space--as seemingly multiple other people have since now mentioned on this thread--particularly given that it is written in Python.

jbverschoor 6y ago

Then why not just create a library which uses sqlite as a proven storage and query engine?