7 ms·
Microsoft open sources PostgreSQL extensions
- theshrike79 2y agoI've said it as a joke many times that PostgreSQL can be a better NoSQL database than many actual NoSQL databases just by creating a (text, JSONB) table. You can do actual searches inside the JSON, index the table with JSONB contents etc. Things that became available in MongoDB very very late.
- Thaxll 2y agoIt's very painful to use JSON with regular SQL drivers, MongoDB driver is much better overall than regular SQL from a programmer perspective.
- javaunsafe2019 2y agoWait what? Writing mongo queries vs SQL is a pain in my experience…
- dtech 2y agoWhich language? In Java (/the JVM) it's perfectly fine while Mongo is a PITA
- nonethewiser 2y agoworking with JSONB in Postgres is painful. It's powerful, I love that it's there, but actually using it is always such a chore.
- krashidov 2y agoGenuinely curious, what do you find painful about it? Is it the syntax? Performance? Or does it just encourage bad practices?
- javaunsafe2019 2y agoI also don't get it - it's super easy to write queries with it. Please elaborate...
- nonethewiser 2y agostuff like this SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @> '{"tags": ["qui"]}'; or SELECT jdoc->'guid', jdoc->'name' FROM api WHERE jdoc @? '$.tags[*] ? (@ == "qui")'; which find documents in which the key "tags" contains array element "qui" https://www.postgresql.org/docs/current/datatype-json.html https://www.postgresql.org/docs/current/datatype-json.html
- Tostino 2y agoI had to ask Claude because I've never worked with Mongo, but apparently the equivalent query for MongoDB is: db.api.find( { "tags": "qui" }, { "guid": 1, "name": 1, "_id": 0 } )
- nonethewiser 2y agoIt's fair to compare alternatives but to be clear I'm not saying postgres JSONB is worse than mongo. I'm not very familiar with mongo either. Im saying it can be tedious is absolute terms. I did not assume this to be universal to document dbs but perhaps it could be. Although your example does seem simpler.
- zulban 2y agoTo your point, I replaced MongoDB with postgresql and a naive wrapper to still use JSON docs the same, and to my surprise my services got way faster and more stable. I write about it here: https://blog.stuartspence.ca/2023-05-goodbye-mongo.html https://blog.stuartspence.ca/2023-05-goodbye-mongo.html
- Thaxll 2y agoAnd many told you that you did not tell the whole story or did not understand mongodb but apparently you fail to adress those. Going from 150ms to 8ms and 80% CPU reduction does not make any sense perf wise. I stand my point it's missing a lot of details in your post and probably miss usage.
- smt88 2y agoIn GP’s shoes, why bother to learn Mongo then? What’s the benefit? Postgres has a good-enough (if not better) document DB inside it, but it also has a billion other features and widespread adoption. Mongo needs a huge benefit somewhere to justify using it.
- troupo 2y agoIf that reduction can be achieved by simply replacing Mongo with Postgres... what is the point of using Mongo? Edit: on top of that, very few things that you store in the DB are non-relational. So you will always end up recreating a relational database on top of any NoSQL database. So why bother when you can just go for a RDBMS from the start?
- Thews 2y agoWhile data can be used in a relational way, it doesn't mean that's the best for performance or storage. Important systems usually require compliance (auditing) and need things like soft deletion and versioning. Relational databases come to a crawl with that need. Sure you can implement things to make it better, but it's layers added that balloon the complexity. Most robust systems end up requiring more than one type of database. It is nice to work on projects with a limited scope where RDBMS is good enough.
- koolba 2y ago> I've said it as a joke many times that PostgreSQL can be a better NoSQL database than many actual NoSQL databases just by creating a (text, JSONB) table. Indeed, the tag line for one of the releases, I think 9.4 or 9.5, was “NoSQL on ACID”.
- sangnoir 2y agoI'd like to buy whoever came up with that tagline a beer.
- deleted 2y ago[deleted]
- mplanchard 2y agoOften true, but updates of JSON documents in postgres or inserts into nested arrays or objects are much, much slower, and involve locking the entire row. I think if your use case is insert/read only, it works well, though at scale even that can become an issue due to the GIN fastupdate index logic, which can lead to the occasional 20-30 second insert as the buffer gets flushed and the index updated.
- GordonS 2y ago> it works well, though at scale even that can become an issue due to the GIN fastupdate index logic, which can lead to the occasional 20-30 second insert as the buffer gets flushed and the index updated. Hmm, interesting! I've been having some intermittent issues recently with a Postgres table that uses JSONB, where everything seems to lock up for several minutes at a time, with no useful logs as to why. Do you have any more info about the GIN issue, and how it can be verified as being a problem please?
- joaohaas 2y agoA quick search brought this up: https://www.louisemeta.com/blog/indexes-gin-algorithms/ https://www.louisemeta.com/blog/indexes-gin-algorithms/ I think it illustrates the issue he's mentioning well enough.
- mplanchard 2y agoThis gitlab issue where they disabled it on one of their tables helped me a lot with understanding and debugging the problem: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7840 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7... It’s not a super well-documented feature in postgres, so I also wound up doing some digging in the mailing lists, but I was really appreciative of all the detail in that GL issue and the associated links.
- GordonS 2y agoThis is absolute gold, thanks!
- pjmlp 2y agoNoSQL is one of those cycles where I was quite happy to keep working with RDMS, while ignoring all the hype around it.
- tracker1 2y agoThere are lessons you can take with you all the same to relational rdbms. Especially with decent json support in the box.
- ak217 2y agoIt's not a joke. JSONB blows Mongo out of the water. I've run large production workloads on both. > You can do actual searches inside the JSON, index the table To be clear, you can do those things with Mongo (and the features have been available for a very long time), they just don't work as well, and are less reliable at scale.
- zX41ZdbW 2y agoThere is still a long road to improve database usability on semistructured data. My team has compared PostgreSQL and MongoDB on analytical queries on JSON, and the results so far have been unimpressive: https://github.com/ClickHouse/JSONBench/ https://github.com/ClickHouse/JSONBench/
- boruto 2y agoWe use postgres jsonb and have search built 30 million/~100-500 kb records. Only GIN/GIST indexes are possible, if we want to anything special(Search/send events if that is edited) at a deeper node of jsonb, the team prefers to extract it out of jsonb into a new column, which creates it own set of problems. Probably if someone looking to use jsonb to replace NoSQL DB would need to look deeper.
- tracker1 2y agoThe past few years that is often the first table I create in a database/schema for b configurations to store in the database.
- Eikon 2y agoI really don't see the point of this compared to Postgres' native JSON support. And with things like this in the source https://github.com/microsoft/documentdb/blob/a0c44348fb473daf75b8950d0118201c0b3e6d19/pg_documentdb_core/src/planner/selectivity.c#L24 https://github.com/microsoft/documentdb/blob/a0c44348fb473da... I'd definitely stay away from it.
- Tostino 2y agoAt least they commented it XD. Selectivity is pretty hard to get right though so I can see why they stubbed it out and will deal with an actual implementation later.
- SigmundA 2y agoSince it uses BSON it support more datatypes like date, binary, decimal, etc. Looks like it supports background indexing too. Date and Binary are ones I wish JSONB supported most often.
- jasonthorsness 2y agoThis is the key feature; unless you work with MongoDB you don’t realize the ubiquity of the extra types it supports and how it improves ease of use
- SigmundA 2y agoMy preference would have been for PG to have a hierarchal data type that supported all of its normal data types and then built JSON support on top of that.
- jasonthorsness 2y agoWell at least now that it is open, that sort of thing can be noticed and fixed more easily
- mplanchard 2y agoI guess my question would be how it handles updates, inserts, and deletes on nested components of objects. These operations in postgres’ JSONB can be a huge problem due to needing to lock the entire row.
- ChicagoDave 2y agoMicrosoft silently hates NoSql document data storage. The complexity of maintaining a relational database is at the core of their proprietary business model. For operational systems, documents are just better in every way. Leave the rdbms for analytics and reporting.
- robertlagrant 2y agoIf it hates it that much it shouldn't have made CosmosDB.
- ChicagoDave 2y agoCosmos is behind a weird wall within Azure. It’s not comparable to DynamoDB or even MongoDB. They (Microsoft) really don’t want you using it.
- p_ing 2y agoI used CosmosDb in MongoDb mode with an app written for MongoDb. Cosmos can be a drop-in replacement.
- dmarlow 2y agoThis is just absurdly false. What basis do you have for this claim?
- ChicagoDave 2y agoIt's just my experience. Microsoft wants you to use SQL Server for everything. If you start thinking in Domain-Driven Design terms, you realize the technology should be dictated by the business models. In many (if not most) cases, a document database is more than sufficient for most bounded contexts and its services, events, commands, and data. Spinning up a relational database, managing schema changes, and tuning indexes is a constant drag on productivity. If Microsoft wanted to compete with DynamoDB or MongoDB, they'd make the Cosmos document database a first line service in Azure. But you have to first spin up Cosmos and then identify a type of data storage. There is no technical reason for this setup other than to lump various non-relational data storage types into one service and create confusion and complexity. I've done bake-offs between Microsoft and AWS and when I talk about a standard serverless architecture, the MS people are usually confused and respond, "What do you mean?" and the AWS folks are "Cool, so Lambdas and DynamoDB. We have 17 sample solutions for you." I'm not saying you can't do serverless in Azure. I'm saying the support and advocacy is not there.
- nikita 2y agoThe word is it's a serious effort on the part of Microsoft. It's missing a MongoDB wire protocol which they plan to opensource as well. In the meantime it's possible to use FerretDB for that. I think the biggest use case is big data and dev platform that need application compatibility and wrapping Atlas is less attractive for some reason.
- cyberax 2y agoI wish PostgreSQL would support better syntax for JSON updates. You can use `->>` to navigate the JSON fields, but there's nothing similar for updates. You have to use clumsy functions or tricks with `||` and string casting.
- joaohaas 2y agoIdk, `jsonb_set` and `||` works good enough for most use cases. For reference, you can use `jsonb_set` to do stuff like: jsonb_set('{"a": [{"b": 1}]}', '{a,0,c}', '2') I think a `||` that works with JSON patches would be nice, but you can easily implement that as an extension if you need it.
- cyberax 2y agoIt's super not great. Instead of having something like: "update blah set field->abc = '123'" you end up with "update blah set field = jsonb_set(field, 'abc', '123');". Not the end of the world, but I have to look up the syntax every time I need to write a query.
- zknill 2y ago> A spokesperson at MongoDB said: "The rise of MongoDB imitators proves our document model is the industry standard. But bolting an API onto a relational database isn't innovation – it's just kicking the complexity can down the road. These 'modern alternatives' come with a built-in sequel: the inevitable second migration when performance, scale, and flexibility hit a wall." I think the reason that a there are so many MongoDB wire compatible projects (like this postgres extension from microsoft, and ferretdb) is because people have systems that have MongoDB clients built into their storage layers but don't want to be running on MongoDB anymore, exactly because "performance, scale, and flexibility hit a wall". If you can change the storage engine, but keep the wire protocol, it makes migrating off Mongo an awful lot cheaper.
- katsura 2y agoMongoDB imitators? Wasn't CouchDB before MongoDB? CouchDB also stores JSON documents. They did create an imitation of the Mongo query syntax, but the document model doesn't seem to originate from Mongo, as far as I can tell.
- WorldMaker 2y agoIn some ways it is another reflection of the unix philosophy "worse is better". The CouchDB wire protocol is really complex because A) it starts with an almost full HTTP stack and is "REST-like", and B) prioritized multi-database synchronization and replication first and up front, which is incredibly powerful and useful, but makes compatible implementations a lot harder to write. MongoDB's wire protocol is simple, stupid, relatively efficient (binary encoding of JSON over plaintext JSON encodings), had a query language that specifically wasn't built on REST terms, and was so simple to implement that every database could do it (and have done it). MongoDB's protocol has fewer features (RIP easy synchronization and replication) but worse is better and if the "core" feature of a database it its query language, then the best/worst query language won, the simple, dumb, easy to replicate one.
- luckydonald 2y agoTo be fair, I'd love a plugin/framework/microservice you could simply on top of Postgres for having CouchDB's offline synchronization capabilites. Like the syncing capabilites for mobile and other "sometimes offline" devices are fire. But in my stack, everything else is already running in a Postgres already, and that's the source of truth.
- atombender 2y agoI was hoping this was an implementation of the schemaless indexing [1], which is the foundation for Azure DocumentDB. That design allows arbitrary nested JSON data to be indexed using inverted indexes on top a variation of B-trees called Bw-trees, and seems like a nice way of indexing data automatically in a way that preserves the ability to do both exact and range matching on arbitrarily nested values. [1] https://www.vldb.org/pvldb/vol8/p1668-shukla.pdf https://www.vldb.org/pvldb/vol8/p1668-shukla.pdf
- isignal 2y agoPostgres supports indexing an arbitrary json document. https://www.postgresql.org/docs/current/datatype-json.html https://www.postgresql.org/docs/current/datatype-json.html Not sure if the query capabilities and syntax match azure docdb but the basic functionality should be workable.
- atombender 2y agoOf course it does, but it's limited. GIN does not support range searches (needed for <, <=, >, >=), prefix or wildcard, etc. It also doesn't support index-only scans, last I checked. You cannot efficiently ORDER BY a nested GIN value. I recommend reading the paper.
- 2d8a875f-39a2-4 2y ago[flagged]
- Someone1234 2y agoIt is MIT licensed, and targets PostgresSql which itself has a very liberal license (and is not a Microsoft project). How does the "extend, extinguish" work in this scenario?
- giancarlostoro 2y agoMy biggest gripe with MongoDB is the memory requirements, which did not become immediately obvious to me when I was first using it. If you index a field, Mongo will shove every record from that field into memory, which can add up very quickly, and can catch you off-guard if you're not accounting for it. It is otherwise a fantastic database that I find myself being highly productive with. I just don't always have unlimited memory to waste, so I opt for more memory friendly SQL databases.
- pier25 2y agohow come Microsoft is investing in PG but dotnet doesn't have an official PG driver?
- taftster 2y agoMicrosoft is weird.
- pphysch 2y agoSame reason they invest in Linux rather than just Windows Developers, developers, developers
- fuy 2y agowhat about npgsql? it's pretty much official and is developed by Microsoft people
- pier25 2y agoAFAIK currently only one contributor is working at Microsoft.
- pickle-wizard 2y agoThis looks promising. I am using MongoDB in my application because the data is document oriented. I did take a brief look at using JSONB in Postgres, but I found it a lot harder to work with. I would prefer to use Postgres as my database, so this is worth investigating. Taking a brief look at the github page, it looks like it will be easy to swap it out in my code. So I think I know what I'll be spending my next sprint on.
- fowl2 2y agoA seemingly coordinated release with the nominally independent FerretDb? What’s the relationship here?
- humanlity 2y agois it https://github.com/microsoft/documentdb https://github.com/microsoft/documentdb ?
- aleksi 2y agoYes