3 ms·

I really want a data format that is effectively binary JSON. What is the subset of all of the features of SQLite that makes either a read-only or an updatable d

I really want a data format that is effectively binary JSON. What is the subset of all of the features of SQLite that makes either a read-only or an updatable data set that is compact. But better searchability than a streaming parser.

jchw 2y ago
If you want to maintain the properties that SQLite has for read use cases, you'll need to replicate a couple of features. At the very least, you'll probably want the format to still be page-based with a BTree structure. You really could get away with just using the SQLite format if you didn't mind the weirdness; a functional SQLite parser that can read tables would not be a significant amount of code. I think, though, that if you want to read the schema as SQLite understands it, you'd need to interpret the CREATE TABLE syntax, which would make it a bit more complex for sure. Otherwise, you can read tables and columns themselves relatively easily, and the values are all stringified.
- hinkley 2y ago
  Yeah if I wasn’t clear I’m talking about a minimal file that SQLite can still open read only without errors, not a third party implementation. Though there might be a few tweaks that would allow SQLite to be a bit more lenient. For instance missing metadata that can be assumed. Maybe b tree nodes exceeding the usual load factor.
Retr0id 2y ago
sqlite itself supports a binary encoding of JSON: https://sqlite.org/jsonb.html https://sqlite.org/jsonb.html
- hinkley 2y ago
  When I said binary JSON I didn’t mean literal JSON. I meant “common denominator interchange format”. It’s too chatty by far and has dismal performance for queries. So you’re better off asking a specific question and getting a larger document that could answer many questions that you do t yet have. For CDNs things like this matter a lot.
  - deleted 2y ago
    [deleted]
cwmma 2y ago
Parquet or some other column oriented data format is probably closest to what you want without getting into indexing your flat files or similar
w10-1 2y ago
MongoDB's BSON?
- hinkley 2y ago
  Mongo sits on a throne of lies and I will never condone anyone using it for any purpose except to make a joke.
79a6ed87 2y ago
Have you tried MessagePack[0]? 0: https://msgpack.org/index.html https://msgpack.org/index.html
- hinkley 2y ago
  I would probably just use bson or gRPC. As o clarified elsewhere, I means JSON as an analogy. I want something that can be scanned and queried cheaply.
deleted 2y ago
[deleted]

jchw 2y ago

If you want to maintain the properties that SQLite has for read use cases, you'll need to replicate a couple of features. At the very least, you'll probably want the format to still be page-based with a BTree structure. You really could get away with just using the SQLite format if you didn't mind the weirdness; a functional SQLite parser that can read tables would not be a significant amount of code. I think, though, that if you want to read the schema as SQLite understands it, you'd need to interpret the CREATE TABLE syntax, which would make it a bit more complex for sure. Otherwise, you can read tables and columns themselves relatively easily, and the values are all stringified.

hinkley 2y ago
Yeah if I wasn’t clear I’m talking about a minimal file that SQLite can still open read only without errors, not a third party implementation. Though there might be a few tweaks that would allow SQLite to be a bit more lenient. For instance missing metadata that can be assumed. Maybe b tree nodes exceeding the usual load factor.

Retr0id 2y ago

sqlite itself supports a binary encoding of JSON: https://sqlite.org/jsonb.html https://sqlite.org/jsonb.html

hinkley 2y ago
When I said binary JSON I didn’t mean literal JSON. I meant “common denominator interchange format”. It’s too chatty by far and has dismal performance for queries. So you’re better off asking a specific question and getting a larger document that could answer many questions that you do t yet have. For CDNs things like this matter a lot.
- deleted 2y ago
  [deleted]

cwmma 2y ago

Parquet or some other column oriented data format is probably closest to what you want without getting into indexing your flat files or similar

w10-1 2y ago

MongoDB's BSON?

hinkley 2y ago
Mongo sits on a throne of lies and I will never condone anyone using it for any purpose except to make a joke.

79a6ed87 2y ago

Have you tried MessagePack[0]? 0: https://msgpack.org/index.html https://msgpack.org/index.html

hinkley 2y ago
I would probably just use bson or gRPC. As o clarified elsewhere, I means JSON as an analogy. I want something that can be scanned and queried cheaply.

deleted 2y ago

[deleted]