5 ms·

In my personal opinion and usage, the performance doesn't matter. Only one driver is written in pure go, and can be easily statically compiled and/or cross-comp

In my personal opinion and usage, the performance doesn't matter. Only one driver is written in pure go, and can be easily statically compiled and/or cross-compiled. > modernc, modernc.org/sqlite, a pure Go solution. This is a newer library, based on the SQLite C code re-written in Go. Unless I'm mistaken, this is not a re-write in Go. This is a transpilation of the the SQLite C library into go, using https://gitlab.com/cznic/ccgo https://gitlab.com/cznic/ccgo

vbezhenar 3y ago
Sqlite developers go to very great lengths making sure their software works reliably. You can't just introduce some transpiler in-between and expect that it'll work. It's a different software and it should not even be called sqlite IMO. The only proper way to use sqlite is to use FFI.
- jchw 3y ago
  If the transpiler is not breaking any defined behavior in C, there is no reason to believe it cannot be a perfectly reasonable solution. There is nothing inherently wrong with targeting another programming language when compiling code, it does not mean there is anything wrong or broken with the generated code or its semantics. It is a less battle-tested C compiler, sure, but it's a C compiler. IMO, you can't argue that something shouldn't be called sqlite because it contains a transpiled copy of sqlite. To me this isn't really substantially different from transpiling sqlite to JS using Emscripten.
- arp242 3y ago
  > You can't just introduce some transpiler in-between and expect that it'll work Obviously it "works" and they didn't "just" transpile it but spent quite a bit of effort on this. Whether it's free of bugs that are not present in SQLite is a different matter. And while it's not run against the SQLite proprietary tests, it does pass all of the TCL tests IIRC, which are quite comprehensive on their own, so there shouldn't be huge glaring bugs.
- shp0ngle 3y ago
  They run the whole testsuite on the transpiled code. I think. But yeah it does give me a little shiver to think you are transpiling C code to go code, and the go-code is not really "pure go", but has a platform-dependent unsafe operations. Just look at the repo https://gitlab.com/cznic/sqlite/-/tree/master/lib?ref_type=heads https://gitlab.com/cznic/sqlite/-/tree/master/lib?ref_type=h...
  - mook 3y ago
    My understanding is that the full test suite is actually proprietary and it's part of how sqlite is funded — the maintainers can provide better claims. There is a public test suite but it's a subset of what they do. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- shepherdjerred 3y ago
  We use go-sqlite3 at work and have had a positive experience, with the benefit of having fewer system dependencies and a simpler build process.
Aissen 3y ago
The biggest issue of the modernc approach is that it can't use the proprietary sqlite test suites and fuzzer to ensure that it works like the original: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- Aissen 3y ago
  (I just realized that probably goes for libsql as well)
psanford 3y ago
I cross compile mattn/go-sqlite3 a lot. It's about the easiest cgo project to cross compile since it has no external dependencies. Just use zig as your c compiler and it all just works.
- makeworld 3y ago
  Do you have any links for using Zig for this?
  - psanford 3y ago
    export CC="zig cc -target aarch64-linux-musl"
    - el-dude-arino 3y ago
      Well I found my side project for the day
  - kristoff_it 3y ago
    https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic https://zig.news/kristoff/building-sqlite-with-cgo-for-every...
mariusor 3y ago
My only complain so far about the package is that it required transpiled versions of the whole dependency chain for sqlite in order to get it working, from libc to tcl. The whole tree for the latest version is about 2G. This is not a trivial amount of traffic/space to use for every compilation.
shp0ngle 3y ago
> Only one driver is written in pure go, and can be easily statically compiled and/or cross-compiled. Which one do you mean? The WASM one? It includes WASM, it needs to be compiled too. Ah github.com/cvilsmeier/sqinn-go. Which... is made by the same person that made this benchmark...? edit: and that requires some random binary to be pre-installed...? Which is in C anyway? https://github.com/cvilsmeier/sqinn-go https://github.com/cvilsmeier/sqinn-go https://github.com/cvilsmeier/sqinn https://github.com/cvilsmeier/sqinn so I don't see any "actually written in go".
- acatton 3y ago
  The modernc one is the one I was refering to. https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite
  - shp0ngle 3y ago
    But that's not pure go, that's C transpiled to go, but even the transpilation is using OS specific and CPU-specific unsafe code.
- ncruces 3y ago
  For the WASM one (disclaimer: I built it) it's at least reasonably simple to reproduce the build [1], and the resulting blob is fully cross platform. Also, although I do (lightly) patch SQLite which might invalidate results in your view, the SQLite team is producing/testing/releasing WASM builds of SQLite built with much the same toolchain, which, hopefully, smokes out compiler bugs. [1]: https://github.com/ncruces/go-sqlite3/blob/main/.github/workflows/repro.sh https://github.com/ncruces/go-sqlite3/blob/main/.github/work...
zombiezen 3y ago
zombiezen.com/go/sqlite is built on top of modernc.org/sqlite, so also has the same properties.
- alephnan 3y ago
  How come Zombiezen came out faster ? I’d expect the opposite
  - cvilsmeier 3y ago
    > How come Zombiezen came out faster ? I’d expect the opposite (OP here) Honestly, me too. Maybe it has to do with leaving out that database/sql driver layer. I guess it would need another round of pprof to find out the real reason.
    - zombiezen 3y ago
      That would be my guess as well. Please do let me know what you find, and feel free to open any issues or PRs.
tedunangst 3y ago
Have you tested that the cross-compiled pure go sqlite runs, not just compiles, on every platform? (Spoiler: it does not.)

vbezhenar 3y ago

Sqlite developers go to very great lengths making sure their software works reliably. You can't just introduce some transpiler in-between and expect that it'll work. It's a different software and it should not even be called sqlite IMO. The only proper way to use sqlite is to use FFI.

jchw 3y ago
If the transpiler is not breaking any defined behavior in C, there is no reason to believe it cannot be a perfectly reasonable solution. There is nothing inherently wrong with targeting another programming language when compiling code, it does not mean there is anything wrong or broken with the generated code or its semantics. It is a less battle-tested C compiler, sure, but it's a C compiler. IMO, you can't argue that something shouldn't be called sqlite because it contains a transpiled copy of sqlite. To me this isn't really substantially different from transpiling sqlite to JS using Emscripten.
arp242 3y ago
> You can't just introduce some transpiler in-between and expect that it'll work Obviously it "works" and they didn't "just" transpile it but spent quite a bit of effort on this. Whether it's free of bugs that are not present in SQLite is a different matter. And while it's not run against the SQLite proprietary tests, it does pass all of the TCL tests IIRC, which are quite comprehensive on their own, so there shouldn't be huge glaring bugs.
shp0ngle 3y ago
They run the whole testsuite on the transpiled code. I think. But yeah it does give me a little shiver to think you are transpiling C code to go code, and the go-code is not really "pure go", but has a platform-dependent unsafe operations. Just look at the repo https://gitlab.com/cznic/sqlite/-/tree/master/lib?ref_type=heads https://gitlab.com/cznic/sqlite/-/tree/master/lib?ref_type=h...
- mook 3y ago
  My understanding is that the full test suite is actually proprietary and it's part of how sqlite is funded — the maintainers can provide better claims. There is a public test suite but it's a subset of what they do. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
shepherdjerred 3y ago
We use go-sqlite3 at work and have had a positive experience, with the benefit of having fewer system dependencies and a simpler build process.

Aissen 3y ago

The biggest issue of the modernc approach is that it can't use the proprietary sqlite test suites and fuzzer to ensure that it works like the original: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

Aissen 3y ago
(I just realized that probably goes for libsql as well)

psanford 3y ago

I cross compile mattn/go-sqlite3 a lot. It's about the easiest cgo project to cross compile since it has no external dependencies. Just use zig as your c compiler and it all just works.

makeworld 3y ago
Do you have any links for using Zig for this?
- psanford 3y ago
  export CC="zig cc -target aarch64-linux-musl"
  - el-dude-arino 3y ago
    Well I found my side project for the day
- kristoff_it 3y ago
  https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

mariusor 3y ago

My only complain so far about the package is that it required transpiled versions of the whole dependency chain for sqlite in order to get it working, from libc to tcl. The whole tree for the latest version is about 2G. This is not a trivial amount of traffic/space to use for every compilation.

shp0ngle 3y ago

> Only one driver is written in pure go, and can be easily statically compiled and/or cross-compiled. Which one do you mean? The WASM one? It includes WASM, it needs to be compiled too. Ah github.com/cvilsmeier/sqinn-go. Which... is made by the same person that made this benchmark...? edit: and that requires some random binary to be pre-installed...? Which is in C anyway? https://github.com/cvilsmeier/sqinn-go https://github.com/cvilsmeier/sqinn-go https://github.com/cvilsmeier/sqinn https://github.com/cvilsmeier/sqinn so I don't see any "actually written in go".

acatton 3y ago
The modernc one is the one I was refering to. https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite
- shp0ngle 3y ago
  But that's not pure go, that's C transpiled to go, but even the transpilation is using OS specific and CPU-specific unsafe code.
ncruces 3y ago
For the WASM one (disclaimer: I built it) it's at least reasonably simple to reproduce the build [1], and the resulting blob is fully cross platform. Also, although I do (lightly) patch SQLite which might invalidate results in your view, the SQLite team is producing/testing/releasing WASM builds of SQLite built with much the same toolchain, which, hopefully, smokes out compiler bugs. [1]: https://github.com/ncruces/go-sqlite3/blob/main/.github/workflows/repro.sh https://github.com/ncruces/go-sqlite3/blob/main/.github/work...

zombiezen 3y ago

zombiezen.com/go/sqlite is built on top of modernc.org/sqlite, so also has the same properties.

alephnan 3y ago
How come Zombiezen came out faster ? I’d expect the opposite
- cvilsmeier 3y ago
  > How come Zombiezen came out faster ? I’d expect the opposite (OP here) Honestly, me too. Maybe it has to do with leaving out that database/sql driver layer. I guess it would need another round of pprof to find out the real reason.
  - zombiezen 3y ago
    That would be my guess as well. Please do let me know what you find, and feel free to open any issues or PRs.

tedunangst 3y ago

Have you tested that the cross-compiled pure go sqlite runs, not just compiles, on every platform? (Spoiler: it does not.)