8 ms·
Go port of SQLite without CGo
- vrglvrglvrgl 4y ago[dead]
- eatonphil 4y agoIt's a really neat project. I did some simple benchmarks of it and it was between 10% slower to twice as slow. Which depending on your use case isn't that bad! https://datastation.multiprocess.io/blog/2022-05-12-sqlite-in-go-with-and-without-cgo.html https://datastation.multiprocess.io/blog/2022-05-12-sqlite-i...
- philosopher1234 4y agoWhat makes it slower?
- platzhirsch 4y agoMy guess is that over the years many optimizations have gone into SQLite that would take a lot of effort to catch up to.
- cdogl 4y agoIt’s hard for Go to beat C in a straight benchmark - Go is garbage collected and allocations often have extra overhead due to the memory layout of interfaces.
- icholy 4y agoIn addition, a lot of optimizations (like loop unrolling) are left on the table to preserve Go's fast compile times.
- pjmlp 4y agoGccgo is the answer here.
- Patrickmi 4y agoGccgo at its core may be fast but when it comes to how the go frontend interpret it to gcc ir that pales in comparison than just straight up optimizating the Go compiler There’ll more opportunities for more optimizations since they move to SSA ir since 1.5 and in turn compile time will definitely go down
- 59nadir 4y agoI was surprised with the announcement that Go's standard compiler had register allocation for function arguments[0] added to it. This seemed like table stakes for a very long time but it really illustrated just how Go's compiler is much, much simpler than most in many areas. On one hand it makes you wonder how much could be squeezed out of Go and how many basic things they're still leaving out of the compiler. On the other it tells part of the story of why Go compiles fast with the backdrop that despite this lack of work by the compiler people use Go productively all day and deliver services they are happy with from a performance perspective. [0] - https://go.dev/doc/go1.17#compiler https://go.dev/doc/go1.17#compiler (2021-08)
- zimpenfish 4y agoI wonder if they'll ever switch to a "normal = fast compile, low optimisation; release = slow compile, high optimisation" model like Rust. Although I suppose other languages already have the "high optimisation" parts covered (rust, julia, python+numpy, etc.) which might make it a fool's errand.
- Mawr 4y agoIt goes to show the impact of optimizing from top-down instead of bottom-up can have. Go was designed with performance in mind ("mechanical sympathy") as opposed to most other GC languages for which performance was an afterthought. If you get high-level language constructs right, you'll get great performance by default, even with a simple compiler: - structs instead of objects - value types - exposed pointers - straightforward allocation semantics Most compilers out there are busy fighting bad language designs. There's only so much a compiler can do when faced with fat objects hidden behind a spider web of pointers. So they have to do these microoptimizations to claw back at least some performance.
- pjmlp 4y agoIt is a matter of skill, Go can do manual memory management and stack allocation just as well.
- kaba0 4y agoAlso, Go barely optimizes code, that’s how it compiles as fast.
- pkolaczk 4y agoNot only Go compiler is not as good at optimisation as C/C++/Rust/Zig/D compilers, but also Go FFI to C has significant overhead, which means there is often not much point in using low-level Linux APIs for performance advantage. Same problem like with Java. And in a database system you quite likely want a fine degree of control with non-standard stuff.
- deleted 4y ago[deleted]
- yencabulator 4y agoHowever, it is absolutely plausible for Go-calling-Go to beat Go-calling-C-via-CGo, since CGo has plenty of overhead.
- cube2222 4y agoOther than what the other responders said, since it uses a C to Go compiler, I'd imagine the Go code that's generated isn't as fast as handwritten Go code could be.
- Patrickmi 4y ago[dead]
- carbocation 4y agoI raised this question on twitter awhile back and the response that I got was that (for read-only tasks) the connection string might ameliorate this. I haven't been able to test it, though: https://twitter.com/frioux/status/1483235674228596739 https://twitter.com/frioux/status/1483235674228596739
- collinvandyck76 4y agoat segment we benchmarked https://github.com/segmentio/ctlstore https://github.com/segmentio/ctlstore against this driver. We saw about a 50% hit to read performance, so we didn't move forward with it, but the improvements in service build times were really appealing.
- djbusby 4y agoReally tho, how much is build time vs runtime? Maybe is just work small projects but I rarely care about build time. Am I missing something?
- collinvandyck76 4y agoIf you have a lot of people and a lot of different builds it can become increasingly significant. It wasn't in our case, but it felt like it could have become so. And a lot of services wouldn't really notice too much if the occasional read from disk was slower, given that the nature of the data was control data.
- badtuple 4y agoHonestly, on a non-toy project, build times with cgo are _brutal_. I agree with you usually, but when a build time on a beefy computer switches from under a second to >1min you notice it. Linters and IDEs get slow when they check for errors, tests run slow, feedback drags, and all your workflows that took advantage of Go's fast compile times are now long enough that your flow and mental context disappear. I'm way more lenient with other languages since the tooling and ecosystem are built around long build times. Your workflows compensate. But Go's tooling and ecosystem assume it compiles fast and treat things more like a scripting language. When that expectation is violated it hurts and everything feels like it's broken.
- bombolo 4y agoWhy would you have to recompile sqlite every time? I guess you just need to compile the .a once and then just reuse it? If you're rebuilding it every single time, your build is set up wrong.
- emodendroket 4y agoI’d go so far as to say it’s completely acceptable for 90+% of use cases.
- schemescape 4y agoWhat approach is used here? WebAssembly (or something similar)? I see a bunch of huge Go files with architecture names.
- eatonphil 4y agoIt's hard to find actual info on this other than Reddit comments but the author wrote a C to Go compiler and then ran it on the SQLite C code base to get a SQLite Go code base. Which means you don't need CGo to embed SQLite if you use that version. https://gitlab.com/cznic/ccgo https://gitlab.com/cznic/ccgo Edit: Look around the source for "ccgo". Like https://gitlab.com/cznic/sqlite/-/blob/master/doc.go#L202 https://gitlab.com/cznic/sqlite/-/blob/master/doc.go#L202 and look for "ccgo" in the closed issues.
- schemescape 4y agoWow! It consumes C ASTs and produces Go code? That is much more impressive than what I had suspected (and what I would have done).
- anuraaga 4y agoJFYI a Wasm approach here https://github.com/ncruces/go-sqlite3 https://github.com/ncruces/go-sqlite3
- ASinclair 4y agoI was just looking for something like this.
- tptacek 4y agoIf you cross-compile (which everybody using macOS to developer for Linux servers does), it's annoying to introduce SQLite to your project, because immediately the conventional `GOOS= GOARCH= go build` trick stops working. But, in case it's helpful, it's also really easy to set up a full cross-compiling environment, either with Zig or with Filippo's musl-cross: https://words.filippo.io/easy-windows-and-linux-cross-compilers-for-macos/ https://words.filippo.io/easy-windows-and-linux-cross-compil... I debated doing the pure-Go SQLite thing, but musl-cross worked just fine for me, and kept things simpler.
- cynix 4y agoIs there something like this for cross-compiling to FreeBSD?
- zX41ZdbW 4y agoYou can check how ClickHouse is cross-compiled for FreeBSD. 1. https://github.com/ClickHouse/sysroot/ https://github.com/ClickHouse/sysroot/ 2. https://github.com/ClickHouse/ClickHouse/blob/master/cmake/freebsd/toolchain-x86_64.cmake https://github.com/ClickHouse/ClickHouse/blob/master/cmake/f... and the `CMAKE_TOOLCHAIN_FILE` variable.
- psanford 4y agoCross compiling with zig is so easy. For example I use sqlite in Go projects on arm. The incantation is just: export GOARCH=arm64 export CC=zig cc -target aarch64-linux-musl export CXX=zig cc -target aarch64-linux-musl
- pjmlp 4y agoCross compilation is always easy when there are no dependencies to the target platform SDKs. When it goes beyond basic libc stuff, then it is when the fun starts.
- 10000truths 4y ago
- AdeptusAquinas 4y agoI've use this. Slightly slower, but for my use case that was fine (a small blog) and immeasurably better doing away with all the CGO hassle. Especially when I might cross compile to an Arm architecture, or some docker container that uses a different approach to the c libraries (like Alpine vs Ubuntu Server or whatever) it was always a hassle, and thats all gone away now. Always seemed silly that one of the most popular storage technologies couldn't be accessed with pure Go.
- JustSomeNobody 4y ago>Always seemed silly that one of the most popular storage technologies couldn't be accessed with pure Go. I don't know why Google didn't pay someone to do this or do it themselves. Surely they use Go and sqlite together.
- Patrickmi 4y agoYou’re overestimating the influence Google as a cooperation has on Go, you think they’re gonna pay for what they’re not gonna use ?
- SrslyJosh 4y ago> you think they’re gonna pay for what they’re not gonna use This is it. They have internal datastores that they use for nearly everything. They might be using SQLite somewhere for something, but I really doubt this would even land near the bottom of their priority list.
- ikiris 4y agoWhy would someone at google ever use sqlite?
- tomjakubowski 4y agoChrome and Android make heavy use of SQLite. Not that Golang is very relevant to those projects
- JustSomeNobody 4y agoWait, so I can use sqlite and Go on Windows (yeah, I know) without gcc!?
- bradfitz 4y agoYes.
- Patrickmi 4y agoThis for sure isn’t meant for performance critical services, best when it comes to appengine
- tedunangst 4y agoI should mention that if you use this, somebody will eventually discover this line of code and be unhappy. https://gitlab.com/cznic/libc/-/blob/v1.22.2/libc_openbsd_amd64.go#L213 https://gitlab.com/cznic/libc/-/blob/v1.22.2/libc_openbsd_am...
- eschaton 4y agoWhat’s the point of something like this instead of just using SQLite? Seriously, “foo but in bar!” is not exactly the sort of thing that solves real problems for most people.
- d1str0 4y agoThis isn’t reinventing the wheel in another language. This is a SQLite driver/implementation that is an alternative to the main one which relies on CGO which is annoying for development in some cases. Making this pure go makes compilation easier.
- adastra22 4y agoCGO is a real problem for many people. Due to how Go handles its async-by-default concurrency model, every call to an FFI library requires setting up a stack and marshaling data. It ends up being rather expensive. So the "without CGo" stuff is about retaining the fast performance people expect of these highly optimized C libraries.
- pkolaczk 4y agoHow does it really solve the FFI issue? Now instead of an FFI boundary between the app and the database, you have an FFI boundary between the database (now implemented in Go) and the operating system C API. And the database quite likely performs multiple calls into the C API per each user app query, so it might be in fact worse.
- masklinn 4y agoInteractions with the OS are managed by the go runtime, and on Linux it does direct syscalls (except when it tries to use vdso). Every program needs to do syscalls at some point, so that’s built in. It is not “in fact worse”.
- pkolaczk 4y agoThat applies only to the stuff exposed by the go stdlib, which is the lowest common denominator between various systems. Databases often use many system specific, advanced calls, to gain more control.
- tetete 4y agoWhy would you want to do this? SQLLITE is so stable. Why would you switch to some port that will be dead in a year?
- LVB 4y agoThe project converts the SQLite C code and has been kept very up to date, so it is far more than a one-off port. The reason is to avoid dealing with CGO in your project, which is necessary with other drivers such as mattn/go-sqlite3.
- masklinn 4y agoImplementation decisions of Go have various advantages in terms of control and memory use but they make linking to binary artefacts problematic. Not complicated, but it impacts the toll chain’s flexibility drastically. The FFI is also extremely slow. That is why the go ecosystem tends to reimplement everything in go.
- jjgreen 4y agoC has no telemetry
- lanstin 4y agoEach call out to a C library from Go locks an OS thread rather than participating in the go routine scheduling. If stuff happens that makes those threads slow or have some issue, they just sit here. I had a server with C Kafka libraries and each time the network glitched the CPU would spike up wards and not decline. With pure go the scheduler is able to just swap out and ignore code that is stuck some where (unless it were in a busy loop). With this SQLite implementation I can stick it in my server (as a background, near real time mirror of memory state to disk) without worrying that the main loop will get clogged or my real time path will be hurt. I use glabarez’ wrapper which makes it have an API like other Go databases: https://github.com/glebarez/sqlite/ https://github.com/glebarez/sqlite/ and hey have been very responsive to issues I raised. I have been running load tests against it for a few weeks, and it is quite solid.
- cube2222 4y ago
- ehhthing 4y agoBeware: This library seems to have bugs that cause it to break on some writes. Recently held an event that heavily used this library and it broke on us half way through, had to wipe the database and switch it back to the cgo version (the data inside was mostly ephemeral, which was quite a relief).
- adastra22 4y agoYeah I would be very suspect about this library. Writing a transpiler that handles all aspects of the C spec correctly is hard enough, to say nothing of replicating stuff that is technically undefined behavior but known to work a certain way. C is not Go and Go is not C--simply rewriting syntax will not result in exactly the same behavior.
- terinjokes 4y agoI've noticed it has different transaction behavior than normal Sqlite, which can cause things to break more easily. I patch Go software that uses it to use the C library as a result, and I stopped having corruption problems.
- dpiddy 4y agoHey, sorry to hear that. I'm a contributor on the project and if you're able to open an issue (https://gitlab.com/cznic/sqlite/-/issues/new https://gitlab.com/cznic/sqlite/-/issues/new) with any info you have it would be very appreciated.
- KingOfCoders 4y agoHow does this compare to https://github.com/glebarez/go-sqlite https://github.com/glebarez/go-sqlite ? Using that for https://www.octobench.com/ https://www.octobench.com/ and I'm very happy.
- mariusor 4y agoIt looks like your library is using some of the plumbing from the parent. I can't tell what benefit it adds on top of it because the parent is also compatible with the Go standard library sql/driver API.
- KingOfCoders 4y agoThanks!
- jamietanna 4y agoI've been using this for one of my personal projects (https://dmd.tanna.dev https://dmd.tanna.dev) to simplify the cross-compilation process and so far it's been very good. When I noticed the SHA3 extension was missing (https://gitlab.com/cznic/sqlite/-/issues/139 https://gitlab.com/cznic/sqlite/-/issues/139) it was noted it could be very easily patched client-side which is handy, as well as adding functionality into the core library to handle it.
- shp0ngle 4y agoNote that this is not “go port”. It’s C transpiled to go with their custom C compiler, and it’s transpiled differently for each architecture (as it’s using unsafe go heavily) I look at it like a neat exercise but not really something to use in production. (I actually use it in tests though.)
- benhoyt 4y agoIt is definitely neat, but I think it's more than an exercise. I believe he runs it through the extensive SQLite test suite (I've seen that before, though I can't see where that's stated now).
- Icathian 4y agoMy understanding was that the sqlite test suite is closed source, and kept under pretty tight grip as it represents their competitive advantage to anyone trying to fork sqlite. Pretty sure the lead dev said something along those lines when he did an episode of Corecursive?
- dpiddy 4y agoThe bits in https://gitlab.com/cznic/sqlite/-/blob/master/tcl_test.go https://gitlab.com/cznic/sqlite/-/blob/master/tcl_test.go run the full suite of SQLite TCL tests as part of a Go test. That's in turn run as part a CI setup with results at https://modern-c.appspot.com/-/builder/?importpath=modernc.org%2Fsqlite https://modern-c.appspot.com/-/builder/?importpath=modernc.o....
- arp242 4y agoThose are not all the tests for SQLite: https://sqlite.org/testing.html#harnesses https://sqlite.org/testing.html#harnesses Only the TCL and SQL Logic Test are publicly available; it's one of the ways that the SQLite developers get paid.
- dpiddy 4y agoYou are correct, I should have clarified that it's just the publicly-available tests. Still, those are pretty expansive.
- hknmtt 4y agoi used it for a bit but was constantly getting transaction lock errors. i even added manual mutex for all transactions since it natively does not have exclusive transactions but even that did not worked. there is not a single good sqlite library for go out there. period. waste of time fiddling with it.
- hsnewman 4y agoWow, I sure wish I had this a couple of years ago!
- randomwebdev 4y agoIt is really a cool project. We are using it as a default driver in PocketBase and although it is not a "drop-in" replacement of the CGO alternative `mattn/go-sqlite3` (different dsn format, some differences in the error messages, etc.), with a small abstraction it works fine for most cases and greatly simplify cross compilation. Performance wise I haven't done intensive benchmark tests yet, but from my local experiments last year it performed ~1.5-2x slower than the CGO version for some queries (it is especially noticeable with LIKE expressions on large string data), but as mentioned previously, for most use cases it is already good enough. Here are the most recent benchmarks I could find (I think from one of the maintainers of the lib) comparing the CGO version and the pure Go port - https://docs.google.com/spreadsheets/d/1YOP1D_ZhuR-ednQhTH6SSrCkcR5bOHzCDoQBZMXvyAw/edit https://docs.google.com/spreadsheets/d/1YOP1D_ZhuR-ednQhTH6S...