zachmu

16 ms·

181.
by zachmu 3y ago
We implemented a query optimizer with a flexible intermediate representation in pure Go: https://github.com/dolthub/go-mysql-server Getting the IR correct so that it's both easy to use and flexible enough to be us
182.
by zachmu 3y ago
Our implementation of the Merkle DAG data structure is optimized for tabular data since Dolt is a SQL database, but a lot of it generalizes to other kinds of data. For example, we put a lot of work into making sure that chunk sizes had a pr
183.
by zachmu 3y ago
Does the app store structured text in a structured way? You probably need to for this to work the way you want. We are writing a version controlled database (Dolt) and we recently implemented automatic merging of the contents of JSON docume
184.
by zachmu 3y ago
Even if Starlink's approach isn't a proven advantage yet, a competitor's ability to deploy new tech at will, at cost, when you are at their mercy to do so on their schedule and at their price, is not a great signal
185.
by zachmu 3y ago
Using a version controlled database gives you "soft deletes" with none of the downsides he mentions (except GDPR -- a versioned data store is probably hard to make GDPR compliant). https://www.dolthub.com/blog&#x2F
186.
by zachmu 3y ago
Apparently he's going to write about Merkle CRDTs next, but if you can't wait: https://www.dolthub.com/blog/2022-06-27-prolly-chunker/
187.
by zachmu 3y ago
I forgot all about the hendo hoverboard scam. Thinking of the 11 rubes who paid $10k each for their own uncontrollable hoverboards that only work over conductive surfaces. https://www.kickstarter.com/projects/142464853&
188.
by zachmu 3y ago
Yes, it's definitely an uphill battle to launch a new DB for this reason. Our current customers are people who need version control bad enough they're willing to take a chance on a younger implementation. That said, we have 99.99%
189.
by zachmu 3y ago
Dolt is the only version controlled SQL database. 1.7x MySQL is still very fast. You're talking about .5ms for a point lookup instead of .3ms, it's not something users will notice at typical scales. And we're still iterating
190.
by zachmu 3y ago
Dolt started in 2018: https://doltdb.com Yes we have commit history from 2015 but that's from an earlier db project (noms) that we forked and built on top of
191.
Config file should be a database(dolthub.com)
2 points by zachmu 3y ago|0 comments
192.
by zachmu 3y ago
We wanted a solution that scales up to terabytes and has fast database access. Storing giant CSV files in git natively doesn't scale, and if you store them in S3 you lose the fast diff and all merge capability. GDPR does require rebasi
193.
by zachmu 3y ago
Very cool! Have you heard about dolt, which is also "Git for Data?" https://github.com/dolthub/dolt We also built dolthub, which is like github for dolt databases: https://www.dolthub.com/
194.
by zachmu 3y ago
Dolt does 3-way merge for both schema and data https://github.com/dolthub/dolt
195.
by zachmu 4y ago
I actually find go a joy to use, I dunno. I like statically typed languages, and go makes defining and using types so simple and straightforward that it feels like you get a lot of the benefits of dynamic typing without the headaches.
196.
by zachmu 4y ago
You're suffering from a failure of imagination, maybe because you've never been able to version data usefully before. There are already lots of interesting applications, and it's still quite new. https://www.dolthu
197.
by zachmu 4y ago
Sure, SQL knowledge is helpful but optional. There are ways to import CSV or other flat files, either on the command line or on dolthub. You just need to make your file's schema match the table's.
198.
by zachmu 4y ago
Yup, there are several solutions you can employ. Here's a more recent write-up on chunking that details some of them, but it mostly comes down to using a better hash function, and having a fail-safe max chunk size for truly pathologica
199.
by zachmu 4y ago
Yeah I'm kinda the same way. Ultimately we decided to copy git's terminology for this rather than making users learn new terms. (ours is the working copy, theirs is the thing you're merging in)
200.
by zachmu 4y ago
Git-like version control requires a Merkle DAG. Unless you know something I don't, there are no RDBMS products that incorporate a Merkle DAG for storage. Dolt is the first. Table data is stored in a cross between a Merkle DAG and a B T
201.
by zachmu 4y ago
If you're a data cleaning guy, you might be interested in our data bounties program. It's collaborative data importing / cleaning. The current one is building a catalog of museum collections. We launch a new one every month.
202.
by zachmu 4y ago
If we were less busy we would adopt versionpress and port it to dolt, such a cool product. Some day.
203.
by zachmu 4y ago
You're thinking too small. The transaction is generally not the unit you want to apply version control to, databases already have robust concurrency support at that level. What you want is to have branches on your data that have many t
204.
by zachmu 4y ago
MVCC is not version control, and time travel / historical querying is not version control. Dolt's unique functionality isn't time travel, although it has that. It's version control, i.e. branch and merge, push and pull,
205.
by zachmu 4y ago
Yup, TerminusDB has a very similar set of capabilities, just for a graph DB instead of SQL / relational. Very cool product if you're in the market for a graph DB.
206.
by zachmu 4y ago
Right, but it is with Dolt. That's the point. CALL DOLT_REVERT('bad-commit-hash') Works just like git revert: creates an inverse of the changes in the commit given and applies it as a patch.
207.
by zachmu 4y ago
I won't speak for other data versioning products, but Dolt is definitely for developers. Our customers are software engineers writing database applications that need version control features.
208.
by zachmu 4y ago
QLDB isn't version controlled, it's an immutable ledger database. If that's what you're looking for, definitely choose it, it's a great product. More discussion of the immutable DB product space: https://
209.
by zachmu 4y ago
https://www.dolthub.com/team
210.
by zachmu 4y ago
Definitely. We have plans to allow clone to be a shallow operation, so that the original data stays remote and only your changes are local. Will make queries slower obviously, but that's the right trade-off for some people.

More ›