10 ms·
A MySQL compatible database engine written in pure Go
- karmakaze 2y agoCompatible has many aspects. I'd be interested in the replication protocols.
- jddj 2y agoI always found the idea behind dolt to be very enticing. Not enticing enough to build a business around, due to it being that bit too different and the persistence layer being that bit too important. But the sort of thing that I'd love it if the mainstream DBs would adopt. I didn't realise the engine was written in Go, and honestly the first place my mind wonders is to performance.
- zachmu 2y agoWe haven't benchmarked the in-memory database implementation bundled in go-mysql-server in a while, but I would be surprised if it's any slower than MySQL, considering that Dolt runs on the same engine and is ~2x slower than MySQL including disk-access. https://docs.dolthub.com/sql-reference/benchmarks/latency https://docs.dolthub.com/sql-reference/benchmarks/latency
- jchanimal 2y agoIf you like the idea of the Dolt prolly trees[1], I'm building a database[2] that uses them for indexing, (eventually) allowing for shared index updates across actors. Our core uses open-source JavaScript[3], but there are a few other implementations including RhizomeDB in Rust[4]. I'm excited about the research in this area. [1] https://docs.dolthub.com/architecture/storage-engine/prolly-tree https://docs.dolthub.com/architecture/storage-engine/prolly-... [2] https://fireproof.storage https://fireproof.storage [3] https://github.com/mikeal/prolly-trees https://github.com/mikeal/prolly-trees [4] https://jzhao.xyz/thoughts/Prolly-Trees https://jzhao.xyz/thoughts/Prolly-Trees
- timsehn 2y agoVersion Control is not the type of thing "mainstream DBs would adopt". We needed to build a custom storage engine to make querying and diffing work at scale: https://docs.dolthub.com/architecture/storage-engine https://docs.dolthub.com/architecture/storage-engine It based on the work of Noms including the data structure they invented, Prolly Trees. https://docs.dolthub.com/architecture/storage-engine/prolly-tree https://docs.dolthub.com/architecture/storage-engine/prolly-...
- jbverschoor 2y agoThis seems to be a wire-protocol proxy for mysql -> SQL. The default proxied database is dolt. I'm guessing this is extracted from dolt itself as that claims to be wire-compatible with mysql. Which all makes total sense.
- zachmu 2y agoNot a proxy in the traditional sense, no. go-mysql-server is a set of libraries that implement a SQL query engine and server in the abstract. When provided with a compatible database implementation using the provided interfaces, it becomes a MySQL compatible database server. Dolt [1] is the most complete implementation, but the in-memory database implementation the package ships with is suitable for testing. We didn't extract go-mysql-server from Dolt. We found it sitting around as abandonware, adopted it, and used it to build Dolt's SQL engine on top of the existing storage engine and command line [2]. We decided to keep it a separate package, and implementation agnostic, in the hopes of getting contributions from other people building their own database implementations on top of it. [1] https://github.com/dolthub/dolt https://github.com/dolthub/dolt [2] https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-server/ https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
- jsteenb2 2y agoReally excellent work! For the curious, would you all be creating an in-memory database implementation that is postgres compatible for the doltgres project?
- zachmu 2y agoWe are moving in that direction but it's not a primary goal at this point. Once we have more basic functionality working correctly in doltgres we will examine splitting off a separate package for it. The in memory implementation has a bunch of MySQL specific stuff in it right now and we're still learning what pieces need to be generalized to share code.
- kamikaz1k 2y agoshouldn't these projects have a perf comparison table? there was a post a couple days ago about the an in-memory Postgres, but same problem on the perf. if someone is considering running it, they're probably considering it against the actual thing. and I would think the main decision criteria is: _how much faster tho?_
- zachmu 2y agoThis is a reasonable point, we'll run some benchmarks and publish them. We expect that it's faster than MySQL for small scale. Dolt is only 2x slower than MySQL and that includes disk access. https://docs.dolthub.com/sql-reference/benchmarks/latency https://docs.dolthub.com/sql-reference/benchmarks/latency
- kamikaz1k 2y agoThanks! Appreciate your response. In dynamic language land, we tend to use real DBs for test runs. So having a faster DB wouldn't hurt!
- sgammon 2y agoIsn't that........ Vitess?
- latortuga 2y ago[flagged]
- pizza234 2y agoThe compatibility (and functionality in general) is severely limited, not usable in production: > No transaction support. Statements like START TRANSACTION, ROLLBACK, and COMMIT are no-ops. > Non-performant index implementation. Indexed lookups and joins perform full table scans on the underlying tables. I actually wonder if they support triggers, stored procedures etc.
- zgk7iqea 2y agoOnly for the in-memory implementation. It is also specifically stated that you shouldn’t use the in-memory stub in production
- zachmu 2y agoYes, triggers and stored procedures are supported. Concurrency is the only real limitation in terms of functionality. The bundled in-memory database implementation is mostly for use in testing, for people who run against mysql in prod and want a fast compatible go library to test against. For a production-ready database that uses this engine, see Dolt: https://github.com/dolthub/dolt https://github.com/dolthub/dolt
- deleted 2y ago[deleted]
- didip 2y agotidb has been around for a while, it is distributed, written in Go and Rust, and MySQL compatible. https://github.com/pingcap/tidb https://github.com/pingcap/tidb Somewhat relatedly, StarRocks is also MySQL compatible, written in Java and C++, but it's tackling OLAP use-cases. https://github.com/StarRocks/starrocks https://github.com/StarRocks/starrocks But maybe this project is tackling a different angle. Vitess MySQL library is kind of hard to use. Maybe this can be used to build ORM-like abstraction layer?
- verdverm 2y agoDolt supports git like semantics, so you can commit, pull, merge, etc...
- deleted 2y ago[deleted]
- neximo64 2y agoIs there anything like this for postgres?
- perplexa 2y agocockroachdb might be close: https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach
- malkia 2y agoIs this for integration/smoke testing?
- timsehn 2y agoMost direct users of go-mysql-server use it to test Golang <> MySQL interactions without needing a running server. We here at DoltHub use it to provide SQL to Dolt.
- zachmu 2y agoHi, this is my project :) For us this package is most important as the query engine that powers Dolt: https://github.com/dolthub/dolt https://github.com/dolthub/dolt We aren't the original authors but have contributed the vast majority of its code at this point. Here's the origin story if you're interested: https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-server/ https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
- geenat 2y agoWhat's the replication story currently like?
- zachmu 2y agoThe vanilla package can replicate to or from MySQL via binlog replication. But since it's memory only, that's probably not what you want. You probably want to supply the library a backend with persistence, not the built-in memory-only one Dolt can do the same two directions of MySQL binlog replication, and also has its own native replication options: https://docs.dolthub.com/sql-reference/server/replication https://docs.dolthub.com/sql-reference/server/replication
- geenat 2y agoInteresting! > If you have an existing MySQL or MariaDB server, you can configure Dolt as a read-replica. As the Dolt read-replica consumes data changes from the primary server, it creates Dolt commits, giving you a read-replica with a versioned history of your data changes. This is really cool.
- maxloh 2y agoI know it is a matter of choice, but why was MySQL chosen instead of PostgreSQL? The latter seems to be more popular on Hacker News.
- tobinfekkes 2y agoTypically, things that are more popular on Hacker News are not most popular with the rest of the world.
- timsehn 2y ago> Why is Dolt MySQL flavored? TLDR; Because go-mysql-server existed. https://www.dolthub.com/blog/2022-03-28-have-postgres-want-dolt/#why-is-dolt-mysql-flavored-anyway https://www.dolthub.com/blog/2022-03-28-have-postgres-want-d... We have a Postgres version of Dolt in the works called Doltgres. https://github.com/dolthub/doltgresql https://github.com/dolthub/doltgresql We might have a go-postgres-server package factored out eventually.
- amelius 2y agoPerformance comparison against the original?
- geenat 2y agoWith Vitess likely consolidating its runtimes (vtgate, vtctl, vttablet, etc) into a single unified binary: https://github.com/vitessio/vitess/issues/7471#issuecomment-2035406575 https://github.com/vitessio/vitess/issues/7471#issuecomment-... ... it would be a wild future if Vitess replaced the underlying MySQL engine with this (assuming the performance is good enough for Vitess).
- zachmu 2y agoI don't think this is in the cards for vitess, their whole architecture is built around managing sharded mysql instances.
- cvalka 2y agoTiDB!
- fedxc 2y agoI always look at these implementations and go wow! But then I think, is there any real use for this?
- zbuttram 2y agoThe readme mentions at least one interesting use which presumably is the impetus for its creation: https://github.com/dolthub/dolt https://github.com/dolthub/dolt
- zachmu 2y agoIf your program integrates with mysql in production, you can use this for much faster local tests. It doesn't have to be a go program, although that makes it easier.
- kitd 2y agoIf you want to run arbitrary queries on structured data then SQL is a good language to do it in. This library gives you the opportunity to build such a SQL layer on top of your own custom structured data sources, whatever they may be.
- west0n 2y agoInteresting, another project implemented in Go that is compatible with MySQL server, alongside others like Vitess and TiDB.
- taf2 2y agoCould this be used as kind of connection proxy to allow for more clients to a single pool of database servers?
- osigurdson 2y agoI suspect Go is probably better, but as a long time C# developer I cringe at the idea of implementing a DB with GC language. It seems that you would be fighting the GC all the time and have to write lots a lot of non-obvious low allocation code, using unmanaged structures, unsafe, etc., a lot. All doable of course, but seems like it would be starting on the wrong foot. Maybe fine for a very small team, but onboarding new devs with the right skill set would be hard.
- winrid 2y agoWell, with a manually managed language you have to do those things pretty much all the the time, but with a GC you can pick which parts are manually managed. Also I suspect this project isn't for holding hundreds of GB of stuff in memory all the time, but I could be wrong.
- neonsunset 2y agoYou would be surprised by performance of modern .NET :) Writing no-alloc is oftentimes done by reducing complexity and not doing "stupid" tricks that work against JIT and CoreLib features. For databases specifically, .NET might actually be positioned very well with its low-level features (intrisics incl. SIMD, FFI, struct generics though not entirely low-level) and high-throughput GC. Interesting example of this applied in practice is Garnet[0]/FASTER[1]. Keep in mind that its codebase still has many instances of un-idiomatic C# and you can do way better by further simplification, but it already does the job well enough. [0] https://github.com/microsoft/garnet https://github.com/microsoft/garnet [1] https://github.com/microsoft/FASTER https://github.com/microsoft/FASTER
- osigurdson 2y agoUsing net6. I agree, performance is generally great / just as fast as its peers (i.e. Java and Go). However, if you need to think about memory a lot, GCed runtimes are an odd choice.
- neonsunset 2y agoWhy is that?
- 090rf 2y ago[dead]
- davgoldin 2y agoCongrats, looks like a lot of hard work! Could I swap storage engine with own key value storage e.g. rocksdb or similar?
- zachmu 2y agoYes, that's the idea. Writing a simple read only database back end is not too tough.
- TechTechTech 2y agoIt would be great if this evolves to support mysql to postgresql and mysql to sqlite. Then we can finally have multiple database engine support for WordPress and others.
- aargh_aargh 2y agoIt is always the edge cases that will kill you. In the case of WP on PostgreSQL, the reason you want WP in the first place is the plugins and those will be hit or miss on PostgreSQL. Just give up on the combination of those two.
- kreetx 2y agoIsn't there an adapter from mysql-to-postgres which would essentially mimic all the quirks in mysql onto an actual postgres?
- aargh_aargh 2y agoI believe this is what Janus (NEXTGRES) does. To clarify, the wire protocol is the easy part, the semantic differences how each database does things is a whole other can of worms. Such emulation will never be 100%, quirks and all.
- speleding 2y agoHow hard would it be to use this as an in-memory replacement for MySQL for testing, let's say, a Rails project? Given how important the DB layer is I would be careful to use something like this in production, but if it allows speeding up the test suite it could be really interesting.
- ceving 2y agoWhy not standard conforming SQL instead of MySQL?
- hwbunny 2y agoWhat's the purpose of this idea? Snapshotted mysql server? Who uses that and for what purpose?