5 ms·
A pure Go embedded SQL database
- kokizzu2 5y agobenchmark? does it support automatic replication like in rqlite/litestream?
- eatonphil 5y agoThe Go package docs and this design file have a bit more info but it doesn't seem like the author has really written about this project. https://pkg.go.dev/modernc.org/ql?utm_source=godoc https://pkg.go.dev/modernc.org/ql?utm_source=godoc https://gitlab.com/cznic/ql/-/blob/master/design/doc.go https://gitlab.com/cznic/ql/-/blob/master/design/doc.go
- ufmace 5y agoThis project seems rather in need of a summary of what it is and what it does. How does it compare to for ex Sqlite?
- eatonphil 5y agoWell for one, if it's in Go, it wouldn't need Cgo.
- anderspitman 5y agoIt's pretty easy to statically link SQLite[0]. Are there other disadvantages to CGO? [0]: https://www.arp242.net/static-go.html https://www.arp242.net/static-go.html
- christophilus 5y agoYes. It doesn’t play nice with goroutines and Go’s concurrency model, from what I understand.
- eloff 5y agoYes, function calls across the boundary are slow. It's fine if the functions do lots of work, it's very expensive for function calls that should be cheap. For sqlite it might not matter too much, since any query does quite a bit of work.
- throwaway894345 5y agoAgreed. I wouldn't be surprised if a pure-Go alternative would be slower. Someone remarked upthread that the same author wrote a C-to-Go transpiler specifically to make a pure-Go sqlite and it was (somewhat unsurprisingly) considerably slower than the CGo version. It's possible (and to an extent, likely) that the transpolar was generating suboptimal Go code, but I would wager that a hand-written version would also be slower.
- azinman2 5y agoA query may do a lot of work but iterating over many results could be very slow.
- eloff 5y agoYes, the API is probably suboptimal for Go. It's better to make a wrapper that does the query and fetches many results in one call, into a Go buffer.
- vbitz 5y agoWithout additional configuration cgo breaks cross-compilation.
- anderspitman 5y agoOh right how could I forget that.
- bbss 5y agoThere are a few go libs now that offer SQLite without cgo. https://github.com/crawshaw/sqlite https://github.com/crawshaw/sqlite https://github.com/zombiezen/go-sqlite https://github.com/zombiezen/go-sqlite
- benhoyt 5y agocrawshaw/sqlite actually uses CGo -- it's a wrapper around the C version of SQLite. For example, see https://github.com/crawshaw/sqlite/blob/23d646f8ac00d9dd239003c121ca4fad16e2445f/sqlite.go#L63 https://github.com/crawshaw/sqlite/blob/23d646f8ac00d9dd2390... zombiezen/go-sqlite uses cznic's pure Go converted version of SQLite, so avoids CGo. It's explicitly stated to be "a fork of crawshaw.io/sqlite that uses modernc.org/sqlite, a CGo-free SQLite package. It aims to be a mostly drop-in replacement for crawshaw.io/sqlite."
- bbss 5y agoAh yes my bad
- qaq 5y agoThere is pure go port of SQLite
- benhoyt 5y agoThe same author ("cznic") has also built a C-to-Go compiler/transpiler and used it to convert SQLite to pure Go source, so that it doesn't require CGo: https://gitlab.com/cznic/sqlite https://gitlab.com/cznic/sqlite ... the converted Go code is pretty much unreadable and it's somewhat slower than SQLite via CGo. But a pretty neat idea! I use it in one of my projects and it works well.
- eatonphil 5y agoI saw that recently. I would be interested in using it but it wasn't clear what level of compatibility it has or how many tests it's passing/not passing.
- benhoyt 5y agoYeah, fair concern. I saw a while back (looks like Aug 2020) that he got all 928,000 SQLite tests passing -- search for "0 errors out of" on this page: https://pkg.go.dev/modernc.org/sqlite#section-readme https://pkg.go.dev/modernc.org/sqlite#section-readme
- eatonphil 5y agoWow! I'd almost trust it more if only 99% of tests were passing rather than 100% of tests passing. :)
- mongol 5y ago
- throwaway894345 5y agoNot sure why this is downvoted. SQLite is great and all, but not depending on FFI is really nice for a lot of things (e.g., cross-compiling is a lot simpler). I'll trade quite a bit to avoid even the very highest quality C dependencies.
- arriu 5y agoWhat are the use cases for this? Is this a good fit for Memory based testing for db stores? Also, I can’t seem to find documentation for how this compares to SQLite
- synergy20 5y agouse go-sqlite3 to work with sqlite3 is one choice. https://github.com/etcd-io/bbolt https://github.com/etcd-io/bbolt is another pure go option. cznic seems like an alternative to bbolt. nice to have some options.
- svnpenn 5y agoBolt is great, here is similar option: https://github.com/genjidb/genji https://github.com/genjidb/genji
- synergy20 5y agoAPI not stable yet and it's a kv database, similar to bbolt/bolt. as far as sqlite3 is concerned, pure-go option we indeed only have this cznic available, which is nice.
- svnpenn 5y ago> it's a kv database, similar to bbolt/bolt Its pretty clear you dont know what youre talking about. It literally says its SQL in the title, you know that big bar at the top of your screen: > Document-oriented, embedded SQL database
- barsonme 5y agogo-sqlite3 is not pure Go. It is a Go wrapper around the SQLite amalgamation.
- ginko 5y agoSo why would you want pure Go? AFAIK Go is significantly slower than C. There's of course safety issues with C, but sqlite is one of the most rigorously tested sw packages out there.
- synergy20 5y agoso you can ship one binary contains everything. for go-sqlite3 you still need native sqlite3 preinstalled on the OS, which is OK for developers but not for many ordinary users.
- jjirsa 5y agoThe cardinal rule of databases: http://www.dbms2.com/2013/03/18/dbms-development-marklogic-hadoop/ http://www.dbms2.com/2013/03/18/dbms-development-marklogic-h... Ignoring that leads to long postmortems about how your embedded db failed in a way you didn’t expect: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-... SQLite is good Rocksdb is good You don’t need pure go.
- steve_taylor 5y agoHow many tens of millions of dollars were ploughed into SQLite?
- harryvederci 5y agoIf you take into account the fact that Richard Hipp is not a billionaire whilst his software is used in almost all devices on (and not-on) the planet, his stuff is used by countless of megacorporations, etc, etc, then by my calculations: not nearly enough. His paycheck alone should be tens of millions of dollars. He should be making so much money that if he'd drop a 100 dollar bill it wouldn't be worth his time to pick it up.
- deleted 5y ago[deleted]
- donio 5y agoDevelopment on this project has started in 2013 and scc shows "Estimated Cost to Develop (organic)" as $1.2M so for whatever that's worth it could be said to check those "5-7 years" and the "millions of dollars" boxes.
- barsonme 5y agoKeep in mind that this project isn’t creating a new SQL database from scratch. It’s literally ingesting SQLite’s C source code and spitting out Go. cznic’s done an absolutely incredible amount of work here. It’s not ready for prod yet, but it works and passes all of SQLite’s tests.
- tzury 5y agoIf (and I am not saying it is, but if) the aim is to have some consider replacing SQLite with this, then I would say - are you ready to walk the walk? For one, are you ready to write tests the way SQLite does? see https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- idoubtit 5y agoThis modernc.org/ql is an embedded database accessed through QL, which is a sub-SQL language "less complex and less powerful than SQL" (according to their homepage). I fail to see when this would be useful compared to the de facto standard SQLite. BTW, modernc.org/sqlite (by the same author) is a port of SQLite to native Go. I've used it with great success for a few personal projects. Before that, I used the go-sqlite package, which needs CGO, and makes it hard to build static executables.
- stuchl4n3k 5y agoFIY. the author, CZNIC, is a CZ domain registry (https://www.nic.cz/ https://www.nic.cz/), the same guys who develop e.g. Turris routers.
- 0xjnml 5y agoFTR: I'm not affiliated with nic.cz, even though I worked there 10+ years ago.