6 ms·
Why you might want a domain-specific database like TigerBeetleDB
- Natsu 4y agoI spent way too long wondering why someone would need a special database to store info about the subfamily of Cicindelinae instead of just reading that this is some kind of special accounting database.
- throwawaymaths 4y agoScyllaDB would be much more interesting as a database of sea monsters. Not sure how to feel about couchdb or cockroachdb
- gardenfelder 4y agohttps://github.com/tigerbeetledb/tigerbeetle https://github.com/tigerbeetledb/tigerbeetle
- weitzj 4y agoAnd in zig lang which is quite interesting
- eatonphil 4y agoWow! I did not expect this to end up on HN. This thread was meant as an informal introduction to a few things I'm excited about regarding TigerBeetle, which was just spun out as a company last week and is where I now work. Some of the thread is a bit hand-wavy so feel free to subscribe to the mailing list [0] for less hand-wavy posts on the subject in the future. :) [0] https://mailchi.mp/8e9fa0f36056/subscribe-to-tigerbeetle https://mailchi.mp/8e9fa0f36056/subscribe-to-tigerbeetle
- anyfactor 4y agoGeez Phil! You do this every time. Every week one of your post makes it to HN and you get suprised by it. At this point this is just normal.
- LAC-Tech 4y agoWow a domain specific database in Zig. This is literally my side project right now (though a completely different industry). Good to see proof you can actually make a company out of it.
- _vvhw 4y agoThanks!
- kristoff_it 4y agoTigerBeetle is a beautiful example of why correctness is what we should strive for, instead of hyper-focusing on (and settling for) memory safety. The design document shows the types of faults they aim to cover: https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/DESIGN.md https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/... And here you can see the safety section in their style guide: https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/TIGER_STYLE.md#safety https://github.com/tigerbeetledb/tigerbeetle/blob/main/docs/...
- kristoff_it 4y ago...and there's also an open bug bounty so if you think that Zig is not a meaningful improvement over C in terms of safety, it's certainly going to be easy money for you :^) https://github.com/tigerbeetledb/viewstamped-replication-made-famous https://github.com/tigerbeetledb/viewstamped-replication-mad...
- ipaddr 4y agoLook at the list of things you can't submit. Excludes: reconfiguration protocol, snapshot, recovery protocol. Todo items, things they know about but can't/won't fix. Excludes any kind of security issue. Easy money it is not but this bounty program doesn't mean what you think it does.
- kristoff_it 4y agoSeems to me that memory safety bugs (eg UAF) can be reported as they would certainly impact correctness.
- BiteCode_dev 4y agoIf you work for one month and you are at the mercy of the other party to decide if they need to pay you, it's not any more easy money than the lottery.
- barbazoo 4y ago> TB comes with flags for things like rejecting transfers where debits would exceed credits on an account. What when the inevitable happens and this business constraint needs to be loosened. What should have been business logic is now database schema, what do you do then? Or did I misunderstand what it's implementing?
- packetlost 4y agoYou misunderstand. It's a database built purposely for ensuring transactions happen in an ordered manner an as correct of a way as possible. The constraints and basic rules of accounting are generally pretty slowly changing so it makes sense in a domain that has: * Very slowly changing validity requirements. Things like (optionally) preventing overspending are things that are pretty intrinsic requirements of the domain (depending on the context, obviously many banks allow overdrafts so they can charge you a fee for spotting you the difference). The validation space is relatively small such that you'll want to allow most transactions that aren't blatantly wrong * Emphasizes correctness and ordering over everything else * The transaction rate is relatively inelastic, that is there aren't huge swings on how many transactions are requests per second such that dynamic scaling is not a desirable property to have if it means being less reliable/correct. Conversely, it's a known slow operation that most people are fine with not being instant. Ex. I'd much prefer my rent payment take 10s to process and be more or less guaranteed to be correct than it take 1s and not. That's why it's domain-specific. You can optimize for the things you care about in your domain most while reducing operational complexity to the minimum viable to meet those demands. It's amazingly well-focused and solves what I assume are hard, limiting problems that the existing financial infrastructure is not equipped to support.
- infogulch 4y agoHow well would TB fare as a more general datastore for tasks that aren't specifically accounting? I guess the hard part may be mapping the source domain model neatly into a double-entry accounting representation. Some diverse examples off the top of my head (and my guess): chat messages (no); logging (no); application metrics (yes, but it doesn't fit very well because metrics can trade a lot of features for throughput?); tracking inventory (definitely); tracking deployment and status of cloud resources [like k8s' etcd] (probably?).
- _vvhw 4y agoYou could one day replace the state machine with your own and have, for example, Redis.
- jandrewrogers 4y agoI have always been an unabashed fan of the idea of domain-specific databases. Database implementations make a lot of compromises for the sake of generality and avoid useful features that overfit a single domain. The idea of a database engine perfectly optimized and feature-fit for a domain has obvious qualitative advantages, and it isn't controversial that it is eminently possible to build such a database. So why are they so rare? It is expensive in several dimensions to build a narrowly tailored database engine from scratch. Database engines are not modular, you can't assemble them from arbitrary parts while maintaining control of their basic characteristics. Building an optimized domain-specific database from scratch is not trivial because you will have to do a lot of the really hard parts yourself. In my head, there has always been a missing piece of software - a "database compiler", that can take the very complex and high-dimensionality abstract specification for a domain and codegen a purpose-built database engine. I also recognize that this is exceedingly non-trivial; I codegen storage engines to spec and that is already difficult enough. Doing that for a full database engine in a specific domain would have hundreds of input parameters, many of which would require expertise in the underlying codegen to know how to use them. It would be awesome if such a thing existed though.
- haney 4y agoIt’s kind of coming from the opposite direction, but it feels like Postgres extensions solve many of the needs you’re describing. They allow a developer to use the underlying database engine while extending it to include domain specific primitives.
- jandrewrogers 4y agoThat's the thing, I've written a lot of Postgres extensions over the decades. Those extensions are fundamentally limited by the underlying architecture of Postgres in myriad ways, though I've been very creative about pushing the limits. In the big picture the kinds of extensions that work well in Postgres are for a relatively narrow set of domains that roughly match what Postgres was designed for but for which it may lack some specific features. The limitations and capabilities of a database are largely architectural. It isn't something you can "extend", it is either there or it isn't. This is why being able to codegen a purpose-built architecture would be interesting; instead of trying to force a domain into an architecture that doesn't fit well, we could produce an architecture that is a perfect fit for the domain while inheriting many of the basic quality of life features you expect.
- cortesoft 4y agoI feel like there is only one aspect mentioned that is actually a domain specific thing (the double entry bit)... everything else is just a list of features that all sorts of general purpose databases have... am I missing something?
- zie 4y agoIt also holds balances of accounts.
- packetlost 4y agomost things don't really have a global intrinsic ordering outside of transaction serialization.
- _vvhw 4y agoYou're right.
- j-pb 4y agoAs someone who has used Zig in anger for the last year (to build an in-memory db coincidentally). I find the idea of using it in production, especially for financial data, absolutely unsettling. The language is awesome, and developing rapidly, but you don't have to wander far to hit a compiler bug, there is no advanced testing infrastructure a la quick-check and friends, and documentation and documentation tooling except for the core reference is non existent. I see a bright future ahead for the language, and I'm glad that I got a Zig version on release day, but if you want to write production code, you might wanna stick to Rust or C for now, or put a small team on just improving the ecosystem.
- _vvhw 4y agoWe think carefully about this. We do in fact have some extremely advanced testing infrastructure, even going as far as using a deterministic Linux hypervisor to do coverage guided fuzzing of our compiled binaries from the outside in. At the same time, we fuzz from the inside out, also Deterministic Simulation Testing, with a ton of assertions (literally a 1000+ and counting) as a force multiplier for fuzzing. Our experience in all this, is that while Zig is early in terms of timeline, the quality is nevertheless extremely high. Andrew and team know what they're doing. They've got some of the best people in the world in their respective fields. For example, Frank Denis of libsodium heading up Zig's crypto. Beyond this, we do the basic things like consciously restrict our use of language features to stable features only (which may be why our experience has not been the same as yours with respect to compiler bugs?), and invest in our own I/O stack around io_uring instead of depending on the std lib, which we know will churn. Two of our team are Zig core team members and we sponsor the Zig Software Foundation to invest in the ecosystem. Don't forget also that Zig's ecosystem is really C's ecosystem so there's an escape hatch there. Also, TB is not yet production ready. We'll ship our production release when we're confident that our TB binary is safe, or able to shut down if it detects any safety violation. At the end of the day, databases are a big investment. It's important to think about the next 20 to 30 years. C would absolutely have been the wrong choice for that future. As you say, the future is bright for Zig, and we're convinced that Zig is the right choice for a database that follows embedded coding standards with respect to memory. For example, TigerBeetle has to handle memory allocation failure, and only does static memory allocation so we never malloc or free after startup.
- endisneigh 4y agoIsn't this basically the premise of FoundationDB? I'd argue if you know your access patterns and they're not going to change you're always going to be better off using a key/value store and appropriately designing a schema around those access patterns.
- _vvhw 4y agoBasically, and we're huge fans of FoundationDB! As you zoom in, you will see differences. For example, TigerBeetle's data structures are all cache line aligned, and we use static allocation etc. The storage fault model is significantly different. We use the same testing techniques though. FDB are pioneers in the space.