5 ms·
Show HN: voidDB – A transactional key-value DB written in Go for 64-bit Linux
- joel-ling 2y agoSeemingly faster and more compact than lmdb-go, bbolt, Badger, and goleveldb at <2K LOC. goos: linux goarch: arm64 pkg: github.com/voidDB/voidDB/test BenchmarkVoidPut-2 131072 14933 ns/op BenchmarkVoidGet-2 131072 1060 ns/op BenchmarkVoidGetNext-2 131072 245.8 ns/op BenchmarkLMDBPut-2 131072 22414 ns/op BenchmarkLMDBGet-2 131072 1826 ns/op BenchmarkLMDBGetNext-2 131072 602.2 ns/op BenchmarkBoltPut-2 131072 66984 ns/op BenchmarkBoltGet-2 131072 2552 ns/op BenchmarkBoltGetNext-2 131072 254.6 ns/op BenchmarkLevelPut-2 131072 44182 ns/op BenchmarkLevelGet-2 131072 30949 ns/op BenchmarkLevelGetNext-2 131072 3441 ns/op BenchmarkBadgerPut-2 131072 15182 ns/op BenchmarkBadgerGet-2 131072 33114 ns/op BenchmarkBadgerGetNext-2 131072 12895 ns/op BenchmarkNothing-2 131072 0.3239 ns/op Check out the docs at https://pkg.go.dev/github.com/voidDB/voidDB https://pkg.go.dev/github.com/voidDB/voidDB
- KAdot 2y agoThe benchmark doesn't accurately represent the real-world database performance because the dataset is too small (roughly half a gigabyte based on [1]?), meaning it fits into the page cache bypassing disk I/O. [1]: https://github.com/voidDB/voidDB/blob/master/test/bench_test.go https://github.com/voidDB/voidDB/blob/master/test/bench_test...
- chronode 2y agoThis is a cool effort but their claims are wildly misleading as their benchmarks aren't fair across the board: For example: - BenchmarkVoidPut runs a single O(n) sized transaction - BenchmarkLMDBPut runs a single O(n) sized transaction - BenchmarkBadgetPut runs O(n) O(n) sized transactions (!!!) I made a local change so that all Put benchmarks ran O(n) O(1) sized transactions and the results were quite different: Void was the slowest, followed by LMDB, Bold, LevelDB, then Badger. I'd also wager the LMDB author would also (lovingly!) tell us we're holding it wrong
- arandomusername 2y agoLMDB is the only one here in C, so the interop is probably what makes LMDB so slow in here
- apatheticonion 2y agoCurious, what overhead is there calling C code from other languages?
- atombender 2y agoGo doesn't use the C calling convention, but has its own growable stack system and goroutine scheduler that maps to goroutines to threads. So a goroutine can't just call a C function directly. In order to interface with C code safely, Go's runtime has to jump to the system stack and do some additional setup, make the call, and then switch back. (Adding to that, if the call takes too long, this prevents other goroutines on the same OS thread from running, so the scheduler must jump in and move those goroutines to a different thread.) All of this is expensive, though we are talking about nanoseconds, not milliseconds. Performance is mostly a problem when doing lots of very quick calls (e.g. you're writing a game engine interacting with something like OpenGL) or lots of slow calls (causing scheduler trashing).
- apatheticonion 2y agoThanks! Is this also the case even Rust consumes a C library?
- atombender 2y agoNo, my understanding is that Rust uses normal stacks, and it uses a classic threading model, so aside from async, calling C doesn't need to any runtime stuff.
- steveklabnik 2y ago
- fasteo 2y ago// voidDB is a cherished toy, a journey into the Unknown, a heroic struggle, // and a work of love. It is the “Twee!” of a bird; a tree falling in the // forest; yet another programmer pouring their drop into the proverbial [bit] // bucket. Above all, it is a shrine unto simple, readable, and functional // code; an assertion that the dichotomy between such aesthetics and practical // performance is mere illusion. [1] https://github.com/voidDB/voidDB/blob/ac104867749e2c698b3d613ffba509fb3088a5eb/overview.go#L148C1-L153C33 https://github.com/voidDB/voidDB/blob/ac104867749e2c698b3d61...
- srameshc 2y agoI admire the author who took up the challenge of writing a KV DB and cherishing self's efforts. I hope someday I could do something similar and feel accomplished.
- joel-ling 2y agoFive years ago, I was new to KV DBs and Golang. You can!
- kunley 2y agoLove the poetry in the comment prose! Endeavour so much needed in the industry. It can only come out of the joy of coding, how else could it be born...
- jakewins 2y agoIt always made me happy to hack nearby the Neo4j page cache "Muninn", because it meant I'd pass by Chris Vest's poem and giant ascii Raven: https://github.com/neo4j/neo4j/blob/release/5.26.0/community/io/src/main/java/org/neo4j/io/pagecache/impl/muninn/MuninnPageCache.java#L74 https://github.com/neo4j/neo4j/blob/release/5.26.0/community...
- joel-ling 2y agoBeautiful reminder of the often forgotten aspect of technical work. Huginn: thought, perception, comprehension, ... Muninn: care, urge, wonder, curiosity, interest, ... https://en.wikipedia.org/wiki/Huginn_and_Muninn#Etymology https://en.wikipedia.org/wiki/Huginn_and_Muninn#Etymology
- sushidev 2y agoFaster than rocksdb?
- dangoodmanUT 2y agoprobably not if it's LMDB-based (at least for writes)
- sirjaz 2y agoSince there are no strict OS dependency I can see, has anyone tried to compile this and run it on Windows?
- 38 2y agoseems pretty bloated https://github.com/voidDB/voidDB/blob/master/go.sum https://github.com/voidDB/voidDB/blob/master/go.sum
- seanw444 2y agoA lot of it is indirect dependencies, and most of the direct dependencies in the go.mod are just for benchmarks, as far as I can tell.
- joel-ling 2y agoThat is correct. Remove directories "test/" and "cursor/test/", then run "go mod tidy"; you should see only one dependency in go.mod: require golang.org/x/sys v0.29.0 Could have used "syscall" in the standard library instead of "golang.org/x/sys/unix" if not for these: https://pkg.go.dev/golang.org/x/sys/unix#F_OFD_GETLK https://pkg.go.dev/golang.org/x/sys/unix#F_OFD_GETLK ... and this: https://go.googlesource.com/proposal/+/refs/heads/master/design/freeze-syscall.md https://go.googlesource.com/proposal/+/refs/heads/master/des...
- maxmcd 2y agoOnly "x/sys/unix" outside of tests/benchmarks ~/go/src/github.com/voidDB/voidDB git:(master) $ go run github.com/tailscale/depaware@latest github.com/voidDB/voidDB dependencies: (generated by github.com/tailscale/depaware) github.com/voidDB/voidDB/common from github.com/voidDB/voidDB+ github.com/voidDB/voidDB/cursor from github.com/voidDB/voidDB github.com/voidDB/voidDB/free from github.com/voidDB/voidDB github.com/voidDB/voidDB/node from github.com/voidDB/voidDB+ github.com/voidDB/voidDB/reader from github.com/voidDB/voidDB golang.org/x/sys/unix from github.com/voidDB/voidDB+ bytes from github.com/voidDB/voidDB+ cmp from internal/fmtsort+ encoding/binary from github.com/voidDB/voidDB/common+ errors from bytes+ D fmt from golang.org/x/sys/unix hash from github.com/voidDB/voidDB+ hash/fnv from github.com/voidDB/voidDB io from bytes+ io/fs from internal/filepathlite+ iter from reflect+ math from encoding/binary+ math/bits from golang.org/x/sys/unix+ os from fmt+ path from io/fs reflect from encoding/binary+ slices from encoding/binary+ LD sort from golang.org/x/sys/unix strconv from fmt+ LD strings from golang.org/x/sys/unix sync from encoding/binary+ sync/atomic from internal/bisect+ syscall from github.com/voidDB/voidDB/cursor+ time from github.com/voidDB/voidDB+ unicode from bytes+ W unicode/utf16 from internal/poll+ unicode/utf8 from bytes+
- binary132 2y agoI can’t help but think this would be more valuable as a C (or even Rust or Zig!) library with a Go API. Then it could be used elsewhere too.
- apatheticonion 2y agoMy thoughts as well!
- natas 2y agoit's screaming for a REPL...
- alexpadula 2y agoVery cool! keep it up :)