3 ms·

I'm more interested in DB's that store protocol buffers, and can expose their innards (like spanner, bigtable, etc.). What's good to use/try on the outside?

by malkia 6y ago

kodablah 6y ago
Protobuf has well-defined JSON semantics, so leveraging the JSON encoding seems like the practical way there and many databases can query/index/manipulate the contents (and if space is a concern, often JSONB and similar types are quite compact).
jeffbee 6y ago
> (like spanner, bigtable, etc.) Neither Spanner nor Bigtable has this feature. If you store a protobuf in a Bigtable column, that's your own business. Bigtable itself won't notice.
- jvolkman 6y ago
  Internal Spanner does support protobuf columns.
  - jeffbee 6y ago
    That's neat. Just it just blow it up into a struct?
    - jvolkman 6y ago
      I'm not sure what the internal representation is. Spanner is aware of the proto type through some mixture of DDL and build rules. Updates to proto types get pushed when updated schemas are pushed. You can use deeply-nested fields in various query clauses, create indexes, etc. But I believe updates must replace the whole object. Much like JSONB columns in other RDBMS. This is from memory. I'm not at Google anymore.
      - malkia 6y ago
        My memory too ;) - Left middle of 2017, here is some article that mentions it - https://www.nextplatform.com/2019/01/15/spanning-the-database-world-with-google/ https://www.nextplatform.com/2019/01/15/spanning-the-databas... also some bare mention here - https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46103.pdf https://static.googleusercontent.com/media/research.google.c... "... The effort to standardize our own implementations both covers gaps in the SQL standard where details are unclear or left to the implementation and covers our own Google specific extensions. Before defining our own SQL dialect, we did consider other options. Using the front end of an open-source SQL processing system, such as Postgres, would have accomplished several of the same goals including filling the gaps left by the SQL standard and ensuring uniform syntax/semantics across our systems. We ruled out this approach because internal systems at Google make heavy use of Protocol Buffers in their applications, and we wanted Protocol Buffer message and enum types to be first class types in our SQL language"
maxmcd 6y ago
FoundationDB's Record Layer?

I'm more interested in DB's that store protocol buffers, and can expose their innards (like spanner, bigtable, etc.). What's good to use/try on the outside?

by malkia 6y ago

I'm more interested in DB's that store protocol buffers, and can expose their innards (like spanner, bigtable, etc.). What's good to use/try on the outside?

kodablah 6y ago

Protobuf has well-defined JSON semantics, so leveraging the JSON encoding seems like the practical way there and many databases can query/index/manipulate the contents (and if space is a concern, often JSONB and similar types are quite compact).

jeffbee 6y ago

> (like spanner, bigtable, etc.) Neither Spanner nor Bigtable has this feature. If you store a protobuf in a Bigtable column, that's your own business. Bigtable itself won't notice.

jvolkman 6y ago
Internal Spanner does support protobuf columns.
- jeffbee 6y ago
  That's neat. Just it just blow it up into a struct?
  - jvolkman 6y ago
    I'm not sure what the internal representation is. Spanner is aware of the proto type through some mixture of DDL and build rules. Updates to proto types get pushed when updated schemas are pushed. You can use deeply-nested fields in various query clauses, create indexes, etc. But I believe updates must replace the whole object. Much like JSONB columns in other RDBMS. This is from memory. I'm not at Google anymore.
    - malkia 6y ago
      My memory too ;) - Left middle of 2017, here is some article that mentions it - https://www.nextplatform.com/2019/01/15/spanning-the-database-world-with-google/ https://www.nextplatform.com/2019/01/15/spanning-the-databas... also some bare mention here - https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46103.pdf https://static.googleusercontent.com/media/research.google.c... "... The effort to standardize our own implementations both covers gaps in the SQL standard where details are unclear or left to the implementation and covers our own Google specific extensions. Before defining our own SQL dialect, we did consider other options. Using the front end of an open-source SQL processing system, such as Postgres, would have accomplished several of the same goals including filling the gaps left by the SQL standard and ensuring uniform syntax/semantics across our systems. We ruled out this approach because internal systems at Google make heavy use of Protocol Buffers in their applications, and we wanted Protocol Buffer message and enum types to be first class types in our SQL language"

maxmcd 6y ago

FoundationDB's Record Layer?