14 ms·
Show HN: BSON Extension for Postgres
JSON support in postgres is superb but sometimes you really want decimal, date, and binary types, "carefree" UTF8 string handling (i.e. no escaping), and robust roundtrippability. So I made an extension for BSON.
- hnto_pics 3y agoThat's the first time I hear of BSON. How does this compare to cbor and messagepack?
- miohtama 3y agoIt started as internal MongoDb format, so it has a different history and context. But today comparison might be interesting.
- Joel_Mckay 3y agoFor cross-platform transactional Queues we found bson rather reliable as a message body format in AMQP over SSL, and tag the protocol handler with the user UUID and API session key. The uptime on that system is nearing 4 years, and its the one part of the system that has been low drama. It is not meant for large reporting formats like live charts etc. These parsers can be unstable if they ignore type checks, precedence, and nesting depth. The original libbson author wouldn't give me a donation link on inquiry, as they probably thought it was some sort of scam. The library merged into mongoDB years ago. Would not touch mongoDB now days given its current license, and propensity to implode key structures on upgrades. =)
- eddd-ddde 3y agoThis is really cool. A common use case for me is building JSON objects directly in my query, for example to return a list of json objects. Usually this means date columns lose their type, is there a way of returning bson like with jsonb_build_object that keeps this types?
- pella 3y agoAs I understand it, it was 'Tested using Postgres 14.4'. I'm wondering if there are any plans to support Postgres versions 15 and 16?
- lxe 3y agoWhy BSON and not JSONB, which is already supported in Postgres?
- robertlagrant 3y agoIt says in the posted comment.
- lolinder 3y agoThey have similar names but solve completely different problems. JSONB is just a binary representation of JSON, with the same limited set of data types. It doesn't provide "decimal, date, and binary types", which OP identified as part of the draw to BSON.
- cryptonector 3y agoJSONB is optimized for traversal and compressibility. Is BSON so-optimized too?
- lolinder 3y agoIt originated with MongoDB as their primary storage and interchange format, so I'd assume it at least was meant to be optimized for that purpose. Either way, like I said, JSONB and BSON are solving slightly different problems—you probably would not choose BSON if you don't need the extra data types, and if you do need the extra data types JSONB can't help you any more than JSON can.
- halayli 3y agoJSONB is not optimized for compression.
- mmis1000 3y agoBSON is more like traditional JSON. All collection elements are ended with specific sub element (just like ']' '}' in JSON), you don't need the complete document before output the first byte. So it is streamable and memory efficient when output. But this also affect decoding. Not prefixed with size means the memory size you need to allocate is unknown at the time of decoding. You may need to allocate more than required amount of memory or grow on demand if you don't know the size at the time that decoding starts. Beside this, BSON is also an exchange format. There is full spec and you can use it outside of mongodb. (For example, you need to write an object that contains ArrayBuffer to disk? Just serialize it as BSON and write to disk.) While jsonb in progress is just a DB internal representation, you don't actually see the binary format anywhere.
- ccleve 3y agoThe code here is really well designed. This project can serve as a tutorial on how to build a Postgres extension. Have a look at this: https://github.com/buzzm/postgresbson/blob/main/pgbson--2.0.sql https://github.com/buzzm/postgresbson/blob/main/pgbson--2.0.... and this: https://github.com/buzzm/postgresbson/blob/main/pgbson.c https://github.com/buzzm/postgresbson/blob/main/pgbson.c Really nice stuff.
- vvpan 3y agoPerhaps this is an opportunity to ask somebody who might know about BSON performance. As a POC/stress test for work I added two multi-GB datasets to Postgres (as JSONB) and to Mongo (BSON). While trying to query approximately a hundred megabytes of data (a few hundred documents) from each I found that Postgres executed the query and decoded the JSON data in under a second, while it took Mongo a few seconds. Does this mean that BSON is slow to deserialize? Or perhaps it is not related to serialization? I was quite confused.
- SahAssar 3y agoDid you do a EXPLAIN ANALYZE on the postgresql query (and perhaps mongodb if it has something similar)? It might help to find if it was the actual query or the de-serialization that was the bottleneck.
- dsizzle 3y agoListed status is "experimental" but only two relatively minor commits in the last two years. Maybe it's more stable than that implies or the author is looking to pick it back up?
- hanszarkov 3y agoI've used the BSON Java ObjectId for distributed primary key generation for a long time. Really useful for distributed systems.
- salil999 3y agoIt's specifically advised NOT to use ObjectId for distribution. This data type has a notion of time embedded in it which results in monotonically increasing ObjectIds. Use UUID or a hashed version of the ObjectId