4 ms·

It's something we thought about, but I think it's somewhat orthogonal to the question of serialization - you still need code to go back and forth between the re

It's something we thought about, but I think it's somewhat orthogonal to the question of serialization - you still need code to go back and forth between the representation in the database and the native data structures. See https://github.com/simoncozens/babelfont/issues/10 https://github.com/simoncozens/babelfont/issues/10 for a deeper discussion of issues around a file format for a font editor. One of the biggest current pain points is dealing with merges in version control (usually git). I think there are ways of dealing with that in an sqlite-based file format, but I worry it would require some custom tooling, especially to make sure invariants aren't violated. Another interesting example is LibrePCB (see https://www.youtube.com/watch?v=vu-h5y6tK34 https://www.youtube.com/watch?v=vu-h5y6tK34) who devised their own text based format specifically to make version control more friendly.

pmarin 6y ago
Alternative text base format was very common in the past because the standard binary format used to be machine dependent for speed (almost a direct dump of the structures in memory).
- raphlinus 6y ago
  Right. Dumping internal memory representations is one point in the spectrum, with extremely appealing properties for compile time, code size, and run time (basically no cost), but other serious problems, including portability (32 bits or 64 bits for pointers) and security. Many of the zero-copy serialization formats (Flatbuffers, Cap'n Proto, FIDL) are inspired by this and to some extent have the property that a "plain ol' data" struct might be serialized by a memcpy of the C representation, but try to improve in the other dimensions. That said, I don't think dealing with the byte representation is the real problem. Projects like simdjson show that you can convert in and out of JSON very fast. The challenge to bloat specifically is getting those converted to the data types of the application. Many serialization approaches, including serde, generate a goodly amount of code for each data type, and this adds up.
  - swsieber 6y ago
    Did you ever take a look at miniserde? It's pretty much just an experiment (e.g. no support for enums), but it seems like the general approach it takes would cut down the amount of code per type. [0] https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde
    - raphlinus 6y ago
      Yes. I think the characterization in TFA is still valid.
      - swsieber 6y ago
        Oh, doh. It's been a while since I read this and forgot you mentioned it there.
freeopinion 6y ago
Focusing strictly on compile time and executable size, I'm curious whether sqlite-as-a-file-format would help or hurt. Of course you still have to move from file to native structures. I just wonder how all the overhead of sqlite compares to whatever xml libs you are using. It seems that both xml and sql tables could be modeled very closely to your native structure. So the question is which approach compiles more efficiently.
eximius 6y ago
Could you wrap the crate mentioned into wasm like watt does for proc macros? Obviously not ideal...

pmarin 6y ago

Alternative text base format was very common in the past because the standard binary format used to be machine dependent for speed (almost a direct dump of the structures in memory).

raphlinus 6y ago
Right. Dumping internal memory representations is one point in the spectrum, with extremely appealing properties for compile time, code size, and run time (basically no cost), but other serious problems, including portability (32 bits or 64 bits for pointers) and security. Many of the zero-copy serialization formats (Flatbuffers, Cap'n Proto, FIDL) are inspired by this and to some extent have the property that a "plain ol' data" struct might be serialized by a memcpy of the C representation, but try to improve in the other dimensions. That said, I don't think dealing with the byte representation is the real problem. Projects like simdjson show that you can convert in and out of JSON very fast. The challenge to bloat specifically is getting those converted to the data types of the application. Many serialization approaches, including serde, generate a goodly amount of code for each data type, and this adds up.
- swsieber 6y ago
  Did you ever take a look at miniserde? It's pretty much just an experiment (e.g. no support for enums), but it seems like the general approach it takes would cut down the amount of code per type. [0] https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde
  - raphlinus 6y ago
    Yes. I think the characterization in TFA is still valid.
    - swsieber 6y ago
      Oh, doh. It's been a while since I read this and forgot you mentioned it there.

freeopinion 6y ago

Focusing strictly on compile time and executable size, I'm curious whether sqlite-as-a-file-format would help or hurt. Of course you still have to move from file to native structures. I just wonder how all the overhead of sqlite compares to whatever xml libs you are using. It seems that both xml and sql tables could be modeled very closely to your native structure. So the question is which approach compiles more efficiently.

eximius 6y ago

Could you wrap the crate mentioned into wasm like watt does for proc macros? Obviously not ideal...