4 ms·

Cool project but I have to admit I sure wish this was written on top of SQLite, rather than just mentioned it, implementing a query language shim for MongoDB on

Cool project but I have to admit I sure wish this was written on top of SQLite, rather than just mentioned it, implementing a query language shim for MongoDB on SQLite would be an amazing project. In the absence of such a project though, this is a pretty great alternative to have.

scottrogowski 6y ago
Thank you! I actually did consider doing exactly what you said. Early on, I decided one my goals for the project would be to make it easy to swap back and forth between Mongita and PyMongo/MongoDB. For me, that meant getting as close as possible to their implementation and using things like BSON, ObjectIds, etc. For that, I sacrificed some performance but I think most people who would use something like Mongita would prefer a faithful reproduction over a few clock cycles. I could be wrong on that though.
- hardwaresofton 6y ago
  Oh thanks for the insight, that makes a lot of sense -- pretty sure mongita is one of the first projects I've seen even attempt this, and it makes sense to target the platform that fits your use case (and obviously everyone who uses PyMongo) the tightest. Your project looks very high quality -- benchmarks, tests, and comparisons are basically an indicator in my mind. Looking through the code you've also already left me (or someone else) space to try doing the SQLite as long as we implement it as an Engine[0] -- am I understanding that right? If I trace the code from database.py to engines/.py it looks like that. I really like the balance you've picked between pragmatism and space for expansion/modification. A couple questions: - How do you feel about type annotations in* the code (as opposed to just the comments, as far as I can see) - PyPy? I wonder if you'd get a ~free speedup [0]: https://github.com/scottrogowski/mongita/blob/master/mongita/engines/disk_engine.py https://github.com/scottrogowski/mongita/blob/master/mongita...
  - scottrogowski 6y ago
    Thanks for the kind words :) I like your idea. As I understand it, since SQLite performs better on most benchmarks anyway, why not have an engine subclass that utilizes SQLite itself? This would be a little trickier than your envisioning. The engine class is the lowest layer but doesn't handle the slow bits like the finds, managing the indicies, etc. So to really get the benefit of SQLite you would have to pull those slow bits in. I do really like the idea of offering a third option so that you have - Memory (fastest) - SQLite (almost as fast but compromises on perfect MongoDB reproduction) - Disk (slower but is faithful to MongoDB) Happy to discuss it more if you want to email me. > How do you feel about type annotations in* the code (as opposed to just the comments, as far as I can see) I used type annotations for a while when they came out but didn't find them to offer advantages over docstrings. The world might have moved over to type annotations though and I've not been aware. > PyPy? I wonder if you'd get a ~free speedup I'm getting a lot of suspicion right now from other commenters who are convinced I have my thumb on the benchmarks and I don't want to give them any more excuses :tears-of-joy:. But joking aside, you're probably right and it's not something I had considered.
WA9ACE 6y ago
Funny enough I started something like that as a ruby library and stopped when I got to the query language shim because I just ran out of time and motivation. https://github.com/wa9ace/squongo https://github.com/wa9ace/squongo
- hardwaresofton 6y ago
  Thanks for sharing -- yeah honestly I think the shim is the huge chunk of work here -- you would basically become a mongo expert/core comitter at the end of it. The same can be said of SQL but it feels like SQL is a bit less ad hoc (because it is, if only for the amount of committees involved), and there are some libraries that already do the parsing and stuff.
thomas4019 6y ago
I created something like this for PostgreSQL. It's a Node.js service that implements the MongoDB wire protocol and converts the query to use jsonb. It works pretty well, but it also got increasingly hard to make it compatible because MongoDB has a lot of quirks.
- hardwaresofton 6y ago
  As a F/OSS freeloader, I would very much like you a link to this project... Feels like something I'd contribute to or do something crazy like try to start a business (2ndquadrant/edb style) on.
  - thomas4019 6y ago
    Thanks! I didn't realize I put the link. Here it is, http://github.com/thomas4019/pgmongo http://github.com/thomas4019/pgmongo. I'd love to hear your thoughts on it and am happy to answer any questions.

scottrogowski 6y ago

Thank you! I actually did consider doing exactly what you said. Early on, I decided one my goals for the project would be to make it easy to swap back and forth between Mongita and PyMongo/MongoDB. For me, that meant getting as close as possible to their implementation and using things like BSON, ObjectIds, etc. For that, I sacrificed some performance but I think most people who would use something like Mongita would prefer a faithful reproduction over a few clock cycles. I could be wrong on that though.

hardwaresofton 6y ago
Oh thanks for the insight, that makes a lot of sense -- pretty sure mongita is one of the first projects I've seen even attempt this, and it makes sense to target the platform that fits your use case (and obviously everyone who uses PyMongo) the tightest. Your project looks very high quality -- benchmarks, tests, and comparisons are basically an indicator in my mind. Looking through the code you've also already left me (or someone else) space to try doing the SQLite as long as we implement it as an Engine[0] -- am I understanding that right? If I trace the code from database.py to engines/.py it looks like that. I really like the balance you've picked between pragmatism and space for expansion/modification. A couple questions: - How do you feel about type annotations in* the code (as opposed to just the comments, as far as I can see) - PyPy? I wonder if you'd get a ~free speedup [0]: https://github.com/scottrogowski/mongita/blob/master/mongita/engines/disk_engine.py https://github.com/scottrogowski/mongita/blob/master/mongita...
- scottrogowski 6y ago
  Thanks for the kind words :) I like your idea. As I understand it, since SQLite performs better on most benchmarks anyway, why not have an engine subclass that utilizes SQLite itself? This would be a little trickier than your envisioning. The engine class is the lowest layer but doesn't handle the slow bits like the finds, managing the indicies, etc. So to really get the benefit of SQLite you would have to pull those slow bits in. I do really like the idea of offering a third option so that you have - Memory (fastest) - SQLite (almost as fast but compromises on perfect MongoDB reproduction) - Disk (slower but is faithful to MongoDB) Happy to discuss it more if you want to email me. > How do you feel about type annotations in* the code (as opposed to just the comments, as far as I can see) I used type annotations for a while when they came out but didn't find them to offer advantages over docstrings. The world might have moved over to type annotations though and I've not been aware. > PyPy? I wonder if you'd get a ~free speedup I'm getting a lot of suspicion right now from other commenters who are convinced I have my thumb on the benchmarks and I don't want to give them any more excuses :tears-of-joy:. But joking aside, you're probably right and it's not something I had considered.

WA9ACE 6y ago

Funny enough I started something like that as a ruby library and stopped when I got to the query language shim because I just ran out of time and motivation. https://github.com/wa9ace/squongo https://github.com/wa9ace/squongo

hardwaresofton 6y ago
Thanks for sharing -- yeah honestly I think the shim is the huge chunk of work here -- you would basically become a mongo expert/core comitter at the end of it. The same can be said of SQL but it feels like SQL is a bit less ad hoc (because it is, if only for the amount of committees involved), and there are some libraries that already do the parsing and stuff.

thomas4019 6y ago

I created something like this for PostgreSQL. It's a Node.js service that implements the MongoDB wire protocol and converts the query to use jsonb. It works pretty well, but it also got increasingly hard to make it compatible because MongoDB has a lot of quirks.

hardwaresofton 6y ago
As a F/OSS freeloader, I would very much like you a link to this project... Feels like something I'd contribute to or do something crazy like try to start a business (2ndquadrant/edb style) on.
- thomas4019 6y ago
  Thanks! I didn't realize I put the link. Here it is, http://github.com/thomas4019/pgmongo http://github.com/thomas4019/pgmongo. I'd love to hear your thoughts on it and am happy to answer any questions.