6 ms·

I'm not sure if the problem you are pointing out has to do with: a) SQLite the file format - which is Public Domain and so well documented that parsers for it

by bane 3y ago

I'm not sure if the problem you are pointing out has to do with: a) SQLite the file format - which is Public Domain and so well documented that parsers for it exist in numerous other languages even though it's almost pointless because... b) SQLite, the Public Domain (and thus entirely source available) C implementation of the library that can operate on the file format -- and is documented to a level well above what most ISO standards shoot for. It's designed to be used in other software and has bindings for pretty much every major language. c) Some notional OpenDocument stored in a SQLite file that's really just waiting for somebody to make and document. ISO standards are great, but if we had to wait for ISO to define a file format we'd have pitifully little to work with.

joshspankit 3y ago
Sounds like a solution is to use the C implementation to define the standard and have it canonized in to an ISO.
- belenos46 3y ago
  Yeah, a solution in search of a problem.
- cornstalks 3y ago
  That's what Opus did. The RFC[1] has a base-64 encoded libopus.tar.gz appendix (Appendix A), which is the "primary normative part of this [Opus] specification." If the prose and source code disagree, the source code takes priority and "wins" when it comes to which is normative. I have a love-hate relationship with this approach. [1]: https://datatracker.ietf.org/doc/html/rfc6716 https://datatracker.ietf.org/doc/html/rfc6716
  - dfox 3y ago
    That is common for codec standards, the normative part of many MPEG specifications is the parser/decoder in C-like pseudo-code. What is somewhat unique for Xiph is that their normative reference decoders are actually usable.
  - Eduard 3y ago
    funny, the RFC even includes a shell command pipeline to extract the base64 out of the awkward RFC formatting. Using the C source code still leaves room for ambiguities / under-specification, no? After all, the semantics rely on the particular gcc release used for compiling the code.
    - cornstalks 3y ago
      There is still the possibility of a bug or under-specification, but that's always the case in any spec. At least with Opus they document what implementation-defined behavior they require, so assuming there aren't any hidden bugs then you should get consistent output across compilers.
      - Eduard 3y ago
        but the semantics change depending on the build tool version and other factors.
btilly 3y ago
It is possible that the C implementation of SQLite is the single most commonly deployed software library ever. If not, then it is probably the second, after zlib. https://www.sqlite.org/mostdeployed.html https://www.sqlite.org/mostdeployed.html Therefore I consider it a better supported format than most standardized formats.
- lucideer 3y ago
  That page makes the argument for zlib & sqlite, but Daniel Stenberg makes some good points here[0]. My guess would be zlib is still number 1 though, even accounting for Daniel's considerations. [0] https://daniel.haxx.se/blog/2021/10/21/the-most-used-software-components-in-the-world/ https://daniel.haxx.se/blog/2021/10/21/the-most-used-softwar...
- pl4nty 3y ago
  for one, it's bundled with consumer versions of Windows as winsqlite3.dll. not sure when this started though
sdeframond 3y ago
I think this has been discussed before about WebSQL. > The [WebSQL] specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/
- sdeframond 3y ago
  The Chrome blog post about deprecating sqlite-based WebSQL makes an interesing point. I believe it applies to OpenDocument as well. > The Web SQL specification cannot be implemented sustainably, which limits innovation and new functionality. The last version of the standard literally states "User agents must implement the SQL dialect supported by Sqlite 3.6.19". SQLite was not initially designed to run malicious SQL statements, yet implementing Web SQL means browsers have to do exactly this. The need to keep up with security and stability fixes dictates updating SQLite in Chromium. This comes in direct conflict with Web SQL's requirement of behaving exactly as SQLite 3.6.19. https://developer.chrome.com/blog/deprecating-web-sql/ https://developer.chrome.com/blog/deprecating-web-sql/
- bane 3y ago
  In other words, "all the implementors chose a standard, but we're the standard deciders so we're killing the whole idea".
  - dragonwriter 3y ago
    One problem was the standard was bug-for-bug replication of a particular version of SQLite. There’s very good reason for that not to be a standard. (Now, assuming the SQLite documentation is licensed in a way which supports this, copying the documentation of SQLite’s supported SQL as of that version into the standard might have been viable, but no one interested in having WebSQL proposed that or any other resolution. That relates to the cited issue of absence of independent implementations, which would have been a problem even with a spec that supported independent implementations and verification of their compliance independent of a particular reference implementation. though I personally think the spec problem is a bigger real problem (even if not the decisive policy problem) than the “everyone is using the same underlying software to implement the spec” problem is in this case, where the shared implementation is a permissively licensed open source implementation sponsored by several of the browser vendors, among others.
zie 3y ago
100% agree and the Library of Congress loves it: https://www.loc.gov/preservation/digital/formats/fdd/fdd000461.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0004... and https://sqlite.org/locrsf.html https://sqlite.org/locrsf.html

by bane 3y ago

joshspankit 3y ago

Sounds like a solution is to use the C implementation to define the standard and have it canonized in to an ISO.

belenos46 3y ago
Yeah, a solution in search of a problem.
cornstalks 3y ago
That's what Opus did. The RFC[1] has a base-64 encoded libopus.tar.gz appendix (Appendix A), which is the "primary normative part of this [Opus] specification." If the prose and source code disagree, the source code takes priority and "wins" when it comes to which is normative. I have a love-hate relationship with this approach. [1]: https://datatracker.ietf.org/doc/html/rfc6716 https://datatracker.ietf.org/doc/html/rfc6716
- dfox 3y ago
  That is common for codec standards, the normative part of many MPEG specifications is the parser/decoder in C-like pseudo-code. What is somewhat unique for Xiph is that their normative reference decoders are actually usable.
- Eduard 3y ago
  funny, the RFC even includes a shell command pipeline to extract the base64 out of the awkward RFC formatting. Using the C source code still leaves room for ambiguities / under-specification, no? After all, the semantics rely on the particular gcc release used for compiling the code.
  - cornstalks 3y ago
    There is still the possibility of a bug or under-specification, but that's always the case in any spec. At least with Opus they document what implementation-defined behavior they require, so assuming there aren't any hidden bugs then you should get consistent output across compilers.
    - Eduard 3y ago
      but the semantics change depending on the build tool version and other factors.

btilly 3y ago

It is possible that the C implementation of SQLite is the single most commonly deployed software library ever. If not, then it is probably the second, after zlib. https://www.sqlite.org/mostdeployed.html https://www.sqlite.org/mostdeployed.html Therefore I consider it a better supported format than most standardized formats.

lucideer 3y ago
That page makes the argument for zlib & sqlite, but Daniel Stenberg makes some good points here[0]. My guess would be zlib is still number 1 though, even accounting for Daniel's considerations. [0] https://daniel.haxx.se/blog/2021/10/21/the-most-used-software-components-in-the-world/ https://daniel.haxx.se/blog/2021/10/21/the-most-used-softwar...
pl4nty 3y ago
for one, it's bundled with consumer versions of Windows as winsqlite3.dll. not sure when this started though

sdeframond 3y ago

I think this has been discussed before about WebSQL. > The [WebSQL] specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/

sdeframond 3y ago
The Chrome blog post about deprecating sqlite-based WebSQL makes an interesing point. I believe it applies to OpenDocument as well. > The Web SQL specification cannot be implemented sustainably, which limits innovation and new functionality. The last version of the standard literally states "User agents must implement the SQL dialect supported by Sqlite 3.6.19". SQLite was not initially designed to run malicious SQL statements, yet implementing Web SQL means browsers have to do exactly this. The need to keep up with security and stability fixes dictates updating SQLite in Chromium. This comes in direct conflict with Web SQL's requirement of behaving exactly as SQLite 3.6.19. https://developer.chrome.com/blog/deprecating-web-sql/ https://developer.chrome.com/blog/deprecating-web-sql/
bane 3y ago
In other words, "all the implementors chose a standard, but we're the standard deciders so we're killing the whole idea".
- dragonwriter 3y ago
  One problem was the standard was bug-for-bug replication of a particular version of SQLite. There’s very good reason for that not to be a standard. (Now, assuming the SQLite documentation is licensed in a way which supports this, copying the documentation of SQLite’s supported SQL as of that version into the standard might have been viable, but no one interested in having WebSQL proposed that or any other resolution. That relates to the cited issue of absence of independent implementations, which would have been a problem even with a spec that supported independent implementations and verification of their compliance independent of a particular reference implementation. though I personally think the spec problem is a bigger real problem (even if not the decisive policy problem) than the “everyone is using the same underlying software to implement the spec” problem is in this case, where the shared implementation is a permissively licensed open source implementation sponsored by several of the browser vendors, among others.

zie 3y ago

100% agree and the Library of Congress loves it: https://www.loc.gov/preservation/digital/formats/fdd/fdd000461.shtml https://www.loc.gov/preservation/digital/formats/fdd/fdd0004... and https://sqlite.org/locrsf.html https://sqlite.org/locrsf.html