4 ms·

Have you considered using a sqlite file? No, seriously: It doesn't require an external schema, supports every kind of indexing and fast random access you could

Have you considered using a sqlite file? No, seriously: It doesn't require an external schema, supports every kind of indexing and fast random access you could want, is supported in like every language, OS, and architecture, has copious tooling, documentation, and community support, and is battle-tested across literally billions of installations worldwide.

atombender 7y ago
No, that would not make any sense. SQLite cannot deal with structured, hierarchical data like the stuff I described in my comment. Also, I am talking about individual documents that already live in a database such as PostgreSQL. I can't store an entire SQLite database in a single column.
- kentonv 7y ago
  > I can't store an entire SQLite database in a single column. Sure you can, it's just a file. :) sqlite scales down nicely to data sets of just a few kilobytes -- if you're worried about parse time of your documents then I assume they are larger than that. That said, if you're already loading the whole blob from a single row in Postgres anyway, then is random access such a big win? Or is the idea that you would selectively read byte ranges out of Postgres? If you're already pulling the bytes into RAM then avoiding the parse isn't that huge of a win. (I say this as the author of Cap'n Proto which is all about zero-copy random access... it's only a big win in certain use cases, like mmap() or shared memory IPC.)
  - atombender 7y ago
    Well, many documents are several megabytes, but many are in the order of 100 bytes. I need a serialization scheme that scales up to large documents and down to tiny ones. I can't imagine that the overhead of initializing an SQLite database from a small byte array in memory is that small, not to mention the overhead of maintaining the table schema. Out of pure curiosity, I glanced at the Go bindings for SQLite, and there's no provision for initializing a database from a byte array, or accessing the raw underlying byte data of a live database. The C API supports implementing your own VFS for custom storage, but that's not supported by the Go bindings, and seems like a lot of work. You're right about loading whole blobs; I was misremembering a little bit. The application in question already pares down the document keys in its queries to avoid sending everything. I'm in the middle of a research project into an alternative backend where the documents are stored as binary data, not JSON, and given a set of keys/keypaths, I want to do a little better than deserializing the whole blob.

atombender 7y ago

No, that would not make any sense. SQLite cannot deal with structured, hierarchical data like the stuff I described in my comment. Also, I am talking about individual documents that already live in a database such as PostgreSQL. I can't store an entire SQLite database in a single column.

kentonv 7y ago
> I can't store an entire SQLite database in a single column. Sure you can, it's just a file. :) sqlite scales down nicely to data sets of just a few kilobytes -- if you're worried about parse time of your documents then I assume they are larger than that. That said, if you're already loading the whole blob from a single row in Postgres anyway, then is random access such a big win? Or is the idea that you would selectively read byte ranges out of Postgres? If you're already pulling the bytes into RAM then avoiding the parse isn't that huge of a win. (I say this as the author of Cap'n Proto which is all about zero-copy random access... it's only a big win in certain use cases, like mmap() or shared memory IPC.)
- atombender 7y ago
  Well, many documents are several megabytes, but many are in the order of 100 bytes. I need a serialization scheme that scales up to large documents and down to tiny ones. I can't imagine that the overhead of initializing an SQLite database from a small byte array in memory is that small, not to mention the overhead of maintaining the table schema. Out of pure curiosity, I glanced at the Go bindings for SQLite, and there's no provision for initializing a database from a byte array, or accessing the raw underlying byte data of a live database. The C API supports implementing your own VFS for custom storage, but that's not supported by the Go bindings, and seems like a lot of work. You're right about loading whole blobs; I was misremembering a little bit. The application in question already pares down the document keys in its queries to avoid sending everything. I'm in the middle of a research project into an alternative backend where the documents are stored as binary data, not JSON, and given a set of keys/keypaths, I want to do a little better than deserializing the whole blob.