15 ms·
Mongo but on Postgres and with strong consistency benefits
- ramchip 2y agoHave you tried it with CockroachDB?
- oskar_dudycz 2y agoI did not, but I'm not using any fancy syntax so far besides JSONB operators. If it won't work, then I'm happy to adjust it to make it compliant.
- salomonk_mur 2y agoWhat would be the advantage of using this instead of simple jsonb columns?
- imnotjames 2y agoLooks like it natches the mongo node API
- joshmanders 2y agoIt uses JSONb under the hood. Just gives you a very "mongo" feel to using PostgreSQL. Not sure how I feel about it. CREATE TABLE IF NOT EXISTS %I (_id UUID PRIMARY KEY, data JSONB)
- wood_spirit 2y agoCan they make it use uuid7 for ids for better insert_becomes_append performance?
- lgas 2y agoYes
- oskar_dudycz 2y agoYes, I'm using JSONB underneath and translating the MongoDB syntax to native queries. As they're not super pleasant to deal with, then I thought that it'd be nice to use some familiar to many MongoDB API. Regarding IDs, you can use any UUID-compliant format.
- lopatin 2y agojsonb isn't web scale. Mongo is web scale.
- digger495 2y agoI see what you did there
- zulban 2y agoNeat. When I migrated a project from mongo to postgres I took a similar approach, except I only implemented the mongo feel I needed within my own project instead of building a proper library as done here. I was surprised how much performance improved despite using a hacky wrapper. https://blog.stuartspence.ca/2023-05-goodbye-mongo.html https://blog.stuartspence.ca/2023-05-goodbye-mongo.html Personally tho, I plan to just drop all similarity to mongo in future projects.
- oskar_dudycz 2y agoYup, I might not reach full compliance, but I will try to follow the Pareto principle. Thanks for the link and kind feedback!
- jrochkind1 2y ago> I was surprised how much performance improved Are you saying you got better performance from postgres jsonb than from mongodb itself?
- zo1 2y agoFrom the article, they mention this alongside a neat graph: "API calls generally take 8 ms now, not 150 ms." His endpoints went from 150ms (with Mongo) to 8ms after moving to Postgres.
- deleted 2y ago[deleted]
- jrochkind1 2y agoThanks. How embarressing for mongo, woah.
- Thaxll 2y agoI remember your post back then and it did not made sense at all, many pointed out it was lacking information and you probably did something wrong with mongo. All the stuff under Mongo Problems is garbage, sorry.
- joeyagreco 2y agoGood work! I would like to see a section on the README outlining the benefits of Pongo
- oskar_dudycz 2y agoThanks, I'll try to cover that, good call!
- Squarex 2y agoHow does it compare with FerretDB[0]? [0] https://www.ferretdb.com/ https://www.ferretdb.com/
- aleksi 2y ago(I'm FerretDB co-founder) As far as I can tell, Pongo provides an API similar to the MongoDB driver for Node that uses PostgreSQL under the hood. FerretDB operates on a different layer – it implements MongoDB network protocol, allowing it to work with any drivers and applications that use MongoDB without modifications.
- Keyframe 2y agoEven monstache?
- aleksi 2y agoThat was the first time I heard about that project. Someone could check it using our guide: https://docs.ferretdb.io/migration/premigration-testing/ https://docs.ferretdb.io/migration/premigration-testing/ Or we will check it ourselves later: https://github.com/FerretDB/FerretDB/issues/4429 https://github.com/FerretDB/FerretDB/issues/4429
- Sytten 2y agoI dont want to sound rude, but as a bootstrap founder it kinda boggles my mind how much money people can raise for a product like ferretdb. I just don't see how it can make VC level return without at the very least changing licenses which seems to ne the premise behind creating this MongoDB proxy. I am sure there is a narrative for it though so best of luck! Also check you managed service links on GitHub, half are dead.
- aleksi 2y agoIf bait-and-switch were our strategy, we would have chosen a different license from the beginning. The Apache license allows everyone to fork FerretDB away and do whatever they like with it. It is unlike MongoDB with their initial AGPL that, in theory, allows everyone to, say, run MongoDB SaaS, but in practice, has enough strings attached to scare people off. We want to have a piece of a bigger pie, not a bigger piece of an existing pie. Providing alternatives makes the whole market bigger. > Also check you managed service links on GitHub, half are dead. Thank you.
- davidpfarrell 2y ago[flagged]
- deleted 2y ago[deleted]
- pipe_connector 2y agoMongoDB has supported the equivalent of Postgres' serializable isolation for many years now. I'm not sure what "with strong consistency benefits" means.
- throwup238 2y ago> I'm not sure what "with strong consistency benefits" means. "Doesn't use MongoDB" was my first thought.
- deleted 2y ago[deleted]
- Izkata 2y ago> MongoDB has supported the equivalent of Postgres' serializable isolation for many years now. That would be the "I" in ACID > I'm not sure what "with strong consistency benefits" means. Probably the "C" in ACID: Data integrity, such as constraints and foreign keys. https://www.bmc.com/blogs/acid-atomic-consistent-isolated-durable/ https://www.bmc.com/blogs/acid-atomic-consistent-isolated-du...
- lkdfjlkdfjlg 2y ago> Pongo - Mongo but on Postgres and with strong consistency benefits. I don't read this as saying it's "MongoDB but with...". I read it as saying that it's Postgres.
- zihotki 2y agoOr is it? Jepsen reported a number of issues like "read skew, cyclic information flow, duplicate writes, and internal consistency violations. Weak defaults meant that transactions could lose writes and allow dirty reads, even downgrading requested safety levels at the database and collection level. Moreover, the snapshot read concern did not guarantee snapshot unless paired with write concern majority—even for read-only transactions." That report (1) is 4 years old, many things could have changed. But so far any reviewed version was faulty in regards to consistency. 1 - https://jepsen.io/analyses/mongodb-4.2.6 https://jepsen.io/analyses/mongodb-4.2.6
- karmakaze 2y agoWhat makes mongo mongo is its distibruted nature, without it you could just store json(b) in an RDBMS.
- richwater 2y ago> store json(b) in an RDBMS I actually did this for as small HR application and it worked incredible well.jsonb gin indexes are pretty nice once you get the hang of the syntax. And then, you also have all the features of Postgres as a freebie.
- eddd-ddde 2y agoPersonally, I much better like postgres json syntax than whatever mongo invented. Big fan of jsonb columns.
- oskar_dudycz 2y agoI'm planning to add methods for raw JSON path or, in general, raw SQL syntax to enable such fine-tuning and not need to always use MongoDB API. I agree that for many people, this would be better.
- darby_nine 2y agobut then you wouldn't have the joy of using the most awkward query language invented by mankind
- lkdfjlkdfjlg 2y ago> What makes mongo mongo is its distibruted nature, without it you could just store json(b) in an RDBMS. Welllllllll I think that's moving the goalposts. Being distributed might be a thing _now_ but I still remember when it was marketed as the thing to have if you wanted to store unstructured documents. Now that Postgres also does that, you're marketing Mongo as having a different unique feature. Moving the goalposts.
- thfuran 2y ago
- posix_monad 2y agoDoes MongoDB have serious market share compared to DynamoDB (and similar clones from Azure, GCP) at this point?
- dudeinjapan 2y agoTotally. Many of the biggest tech companies are using for core use cases. Stripe uses a modified version: https://stripe.com/blog/how-stripes-document-databases-supported-99.999-uptime-with-zero-downtime-data-migrations https://stripe.com/blog/how-stripes-document-databases-suppo... We use MongoDB’s cloud offering called Atlas as our core DB at TableCheck.
- maxdo 2y agoMongodb and dynamodb are completely different dbs. One is unlimited scale KV but very expensive , another is document nosql db that sells you idea “it just works” for lots of features , indexes on anything , aggregation , time series . Vector DB, sharding , replicas etc . It’s a very powerful db for sure.
- cpursley 2y agoThanks, just added Pongo to the NoSQL section of my "Postgres Is Enough" gist: https://gist.github.com/cpursley/c8fb81fe8a7e5df038158bdfe0f06dbb#nosql https://gist.github.com/cpursley/c8fb81fe8a7e5df038158bdfe0f...
- oskar_dudycz 2y agoThank you!
- pmarreck 2y ago“Scalling” should say “Scaling” Nice list!
- czechdeveloper 2y agoI suggest you add https://github.com/tensorchord/pgvecto.rs https://github.com/tensorchord/pgvecto.rs for vector search as well
- cpursley 2y agoThis is really nice. Your project?
- revskill 2y agoGenius.
- oskar_dudycz 2y ago<3
- hdhshdhshdjd 2y agoI use JSONB columns a lot, it has its place. It can fit certain applications, but it does introduce a lot of extra query complexity and you lose out on some ways to speed up query performance that you could get from a relational approach. Which is to say JSONB is useful, but I wouldn’t throw the relational baby out with the bath water.
- oskar_dudycz 2y agoI'm planning to add possibility to use Generated Columns in the future https://www.postgresql.org/docs/current/ddl-generated-columns.html https://www.postgresql.org/docs/current/ddl-generated-column... to allow more optimisations.
- cypherpunks01 2y agoCan a Generated Column read data from a JSONB column? That'd be really cool, but I'm not familiar enough with generated columns to know.
- jamescrowley 2y agoYep!
- doctor_eval 2y agoI’ve been doing some reasonably serious playing with the idea of using jsonb columns as a kind of front end to relational tables. So basically, external interactions with the database are done using JSON, which gives end users some flexibility, but internally we effectively create a realtime materialised view of just those properties we need from the json. Anyone else tried this approach? Anything I should know about it?
- hdhshdhshdjd 2y agoI do something similar, building a lightweight search index over very large relational datasets. So the tables are much simpler to manage, much more portable, so I can serve search off scalable hardware without disturbing the underlying source of truth. The downside is queries are more complex and slower.
- 314156 2y agohttps://docs.oracle.com/en/database/oracle/mongodb-api/mgapi/overview-oracle-database-api-mongodb.html#GUID-1CF44843-6294-45F0-8065-B9E8034D6CB1 https://docs.oracle.com/en/database/oracle/mongodb-api/mgapi... Oracle database has had a MongoDB compatible API for a few years now.
- cyberpunk 2y agoAnd it only costs 75k a seat per year per developer, with free bi yearly license compliance audits, a million in ops and hardware to get near prod and all the docu is paywalled. What a deal!
- slau 2y agoA client had a DB hosted by Oracle. The client was doing most of their compute on AWS, and wanted to have a synchronised copy made available to them on AWS. Oracle quoted them a cool $600k/year to operate that copy, with a 3 year contract. DMS + Postgres did it for $5k/year.
- cyberpunk 2y agoClient of mine wanted to shift a rac cluster from some aging sparc gear into a VMware or openstack or whatever farm they had on premise; oracle demanded they pay CPU licenses for every single CPU in the cluster as each one could “potentially” run the oracle database, quoted them seven figures. They rewrote the app instead.
- 314156 2y agoMaybe you are unaware of this? https://www.oracle.com/cloud/free/ https://www.oracle.com/cloud/free/
- rework 2y agoLooks sort of like MartenDB but trying to minic mongo api, unsure why anyone would want to do that... mongo api is horrible...
- JanSt 2y agoWouldn't that allow to switch from Mongo to Postgres without having to rewrite all of your app?
- oskar_dudycz 2y agoHint: I'm an ex-Marten maintainer, so the similarity is not accidental ;) As Op said, not needing to rewrite applications or using the muscle memory from using Mongo is beneficial. I'm not planning to be strict and support only MongoDB API; I will extend it when needed (e.g. to support raw SQL or JSON Path). But I plan to keep shim with compliant API for the above reasons. MongoDB API has its quirks but is also pretty powerful and widely used.
- rework 2y agoOh, so you are, then we can rest assured this will end up being a solid project! I personally can't stand mongodb, its given me alot of headaches, joined a company and the same week I joined we lost a ton of data and the twat who set it up resigned in the middle of the outage. Got it back online and spend 6m moving to postgresql.
- oskar_dudycz 2y agoThanks, that's the goal: to bring the solid and verified approach in Marten to Node.js land. The concept is similar, but the feature set will be different.
- Tao3300 2y agoDitch the dalmatian before Disney rips your face off.
- oskar_dudycz 2y agoFair, I'll redraw it ;)
- harel 2y agoI regularly find the hybrid model is a sweet spot. I keep core fields as regular columns and dynamic data structures as JSONB. It brings the best of both worlds together.
- Waterluvian 2y agoI do this too with Postgres and it is just the best of both. A robot is a record. A sensor calibration is a record. A warehouse robot map with tens of thousands of geojson objects is a single record. If I made every map entity its own record, my database would be 10_000x more records and I’d get no value out of it. We’re not doing spatial relational queries.
- hobs 2y agoIt's great when you have no reason EVER to decompose the data. That being said, when you start going "wait why is one record like this? oh no we have a bug and have to fix one of the records that looks like this across all data" and now you get to update 10,000x the data to make one change.
- harel 2y agoSmall price to pay in my opinion. How often will that happen vs how often the database is used. Migrations like that can be done incrementally over time. It's a solved problem.
- Waterluvian 2y agoIt’s also trivial to do. My JSON fields are all backed by JSON Schema. And I just write a data migration that mutates the data in some way and have the migration run by one host in a rate limited manner. It’s not quite as good as a traditional change in schema but it’s such a non-issue.
- hobs 2y agoI am glad it works! I have just been subject to several systems that have grown over time that worked very well until it became a problem (and then a huge one) so I am glad you are taking a disciplined approach.
- willsmith72 2y agoIt's technologically cool, but I would love a "why" section in the README. Is the idea you're a mongo Dev/love the mongo api and want to use it rather than switch to pg apis? Or want to copy some code over from an old project? I'm sure there are use cases, I'm just struggling to grasp them. Especially if it's about reusing queries from other projects, AI is pretty good at that
- megadal 2y ago> AI is pretty good at that Good at what? Rewriting the queries? I think the point of Pongo is you can use the exact same queries for the most part and just change backends. I've worked a job in the past where this would have been useful (they chose Mongo and regretted it).
- willsmith72 2y agoFor sure, but it feels really risky. If it's a small codebase, I would be more confident knowing what queries I was using and just switch them. If it's a large codebase, I'd want some really comprehensive test coverage, including performance tests
- oskar_dudycz 2y agoGood call! I added such a section: https://github.com/event-driven-io/Pongo?tab=readme-ov-file#why-pongo https://github.com/event-driven-io/Pongo?tab=readme-ov-file#.... I'd be interested in your thoughts on it.
- willsmith72 2y agonice!
- aussieguy1234 2y agoI'd like to know why AWS went with Aurora DB for their DocumentDB backend. Did the Mongo license change trigger a rush to build something Mongo compatible, but not quite MongoDB?
- oskar_dudycz 2y agoYou can check more in those sources: - https://techcrunch.com/2019/01/09/aws-gives-open-source-the-middle-finger/ https://techcrunch.com/2019/01/09/aws-gives-open-source-the-... - https://news.ycombinator.com/item?id=18871483 https://news.ycombinator.com/item?id=18871483 - https://www.enterprisedb.com/blog/documentdb-really-postgresql https://www.enterprisedb.com/blog/documentdb-really-postgres... TLDR: AWS didn't want to pay for licenses and rolled out their own thing.
- navbryce 2y agoI made a joke tweet about this in Nov 2023 (even called it "Pongo"). This is definitely a just a funny coincidence, but I'm going to pretend like I can see into the future: https://x.com/navbryce/status/1720580136737894661 https://x.com/navbryce/status/1720580136737894661
- oskar_dudycz 2y ago"Great mind think alike" B-)
- a13n 2y agoDoes Pongo work with mongoose? I would guess most mongo users are using mongoose and supporting that library would drive more adoption.
- oskar_dudycz 2y agoNot yet, but it's a nice idea. I added GH issue to track that: https://github.com/event-driven-io/Pongo/issues/13 https://github.com/event-driven-io/Pongo/issues/13
- deanCommie 2y agoSo DocumentDB? https://news.ycombinator.com/item?id=18870397 https://news.ycombinator.com/item?id=18870397
- oskar_dudycz 2y agoYes, a similar idea, but I don't aim to be 100% MongoDB compliant or full replacement. My goal is to use as many of PostgreSQL features as possible. Having the library level as translation will allow more scenarios like, e.g. sharing connection and using PostgreSQL hosting.
- DonnyV 2y agoWould love a C# version of this. I usually use Mongodb for all of our projects. But we need to use Postgres for a project. This would come in very handy.
- oskar_dudycz 2y agoYou can check Marten, that I was co-maintaining: https://martendb.io/ https://martendb.io/. It doesn't have MongoDB-compliant API, but it's mature, stable and efficient.
- frithsun 2y agoProgrammers would be better served by learning nothing except SQL instead of their current strategy of trying to learn everything except SQL.
- dudeinjapan 2y agoWhile I’d agree that understanding SQL basics is an important fundamental for novices to learn, I started using MongoDB 11 years ago and haven’t looked back.
- dboreham 2y agoThey should learn about b-trees and how indexed queries can be done with them either with or without an explicit query language. Then they can decide what kind of data storage service they need. Understand what's happening inside the black box.
- cryptonector 2y agoYes, for sure, though I'd still start with SQL.
- marcus_holmes 2y agoI tried a similar approach in a previous startup - treat data as documents and store in a JSONB field. Postgres was awesome and handled this brilliantly, but the lack of schema and typing killed it. We just ended up fighting data quality the whole time. We couldn't assume that any document had all the required fields, or that they were in a format that made sense e.g. the Price column sometimes had currency symbols, and sometimes commas-and-periods in UK/US format and sometimes in Euro format - sorting by Price involved some complicated parsing of all the records first. We moved back to relational tables. I won't say I'd never do this again, but I would definitely not just throw JSON documents to a database and expect good things to happen.
- jimmyl02 2y agothe view I now have is that for a relational table, yes you have to suffer through migrations but at least they are in sql. for document based stores, you still have to have migrations, but they are just implemented in code json documents sound great, especially initially, but end up being a maintenance nightmare
- hans_castorp 2y ago> for document based stores, you still have to have migrations, but they are just implemented in code The problem with that - in my experience - is that migrating the structure for thousands (if not millions) of documents is way slower than running a DDL command (as it means reading each document, parsing it, modifying it and writing it back). Many DDL commands are just metadata update to the system catalogs so they are quite fast (e.g. adding a new column with a default value). With documents you wind up with millions of single row updates. This can be mitigated by doing a "lazy" migration when a document with the old structure is first read. But that makes the code much more complicated.
- globular-toast 2y agoOr, to put it another way, yes you have to write and maintain an upfront schema, but a document-based system has a schema too, it's just distributed amongst 50 codebase and 10 people's heads.
- ilius2 2y agoIf I were to start a new project, I would directly use postgres, and possibly add a JSONB column ONLY FOR OPTIONAL fields that you don't query frequently. Throwing everything in a document is just fermenting chaos and pain. That being said, I do love the syntax and structure of Mongo pipelines over SQL.
- stanislavb 2y ago"Throwing everything in a document is just fermenting chaos and pain." - I LOVE THIS.
- throwaway76324 2y agoAt $WORK, we use the same approach for integrations with 3rd party systems. The data that is common for all integrations are stored as columns in a relational table. Data that are specific for each integration are stored in JSONB. This is typically meta data used to manage each integration that varies. It works great and you get the combination of relational safety and no-schema flexibility where it matters.
- deleted 2y ago[deleted]
- vmfunction 2y agoHmmm how does this compare to https://www.ferretdb.com https://www.ferretdb.com ? How does this handle large files? Is it enough to replace GridFS? One of main attraction for MongoDb is it's handling of large files.
- sberder 2y agoThis looks great, I'll definitely give it a try. As many mentioned already, having classic columns and a JSON(B) column seems to be a common solution. How do you handle data validation for the JSON documents? My current project uses Django for metadata. I've been thinking about creating a layer similar to model fields in Django. You would declare a JSON "model" through those fields and assign it to the actual model JSON field.
- throwaway76324 2y agoYou can just specify a model/DTO object and serialize it as JSON when saving. Many frameworks do that automatically so you don't need to think about it. At work we just annotate the field in the model as a json-field, and the framework will handle the json-conversion automatically and store the other fields in the model as regular database columns. pseudo code (to not trigger language wars): class Foo { @Id UUID id; String name; @Json MyCustomModel model; } Adding fields is not an issue, as it will simply be missing a value when de-serializing. Your business logic will need to handle its absence, but that is no different than using MongoDB or "classic" table columns
- sberder 2y agoThat's a very low cost approach, I love it! I still think the Django ecosystem would benefit from a standardized/packaged approach including migrations. I'll ponder a bit more
- oskar_dudycz 2y agoThank you! I'm planning to add support to JSON schema and run the validation upon insert/update operation.
- deleted 2y ago[deleted]
- tracker1 2y agoHas this been tested with CockroachDB or any other databases that use a mostly compatible PostgreSQL wire protocol and query language?
- mvsdiego 2y agoNice. Oracle has a similar library based documents/collections API named SODA, been around for years: https://docs.oracle.com/en/database/oracle/simple-oracle-document-access/ https://docs.oracle.com/en/database/oracle/simple-oracle-doc... There are separate drivers for Java, node.js, python, REST, etc. In addition to that, it has Mongo API, which is fully Mongo compatible - you can use standard Mongo tools/drivers against it, without having to change Mongo application code. Both are for Oracle Database only, and both are free.