6 ms·
Show HN: PumpkinDB, an event sourcing database engine
- yrashk 10y agoIt's a lower-level "database engine" that allows you to build different types of higher level databases based on a very simple foundation: 1) BTree-based K/V engine (which gives you an ability to iterate over lexicographically sorted keys) 2) Strong immutability guarantees (data can not be overwritten) 3) ACID transactions 4) Server-side executable imperative language that gives you a control over querying costs In a sense, it's as much of a database constructor as different MUMPS systems (GT.M, for example: https://en.wikipedia.org/wiki/GT.M https://en.wikipedia.org/wiki/GT.M) PumpkinDB also aims to provide a good set of standard primitives that help building more sophisticated databases, ranging from hashing to JSON support, and more to come.
- mindcrash 10y agoSo basically something similar to FoundationDB?
- yrashk 10y agoTo a certain degree, yes
- dozzie 10y agoFrom what I see in the published documentation, it's way too early to make it "show HN". If I didn't know what event sourcing is, I couldn't make heads or tails of your daemon, and even then, I don't understand how is user supposed to interact with it in a production-like deployment.
- yrashk 10y agoWhat to Submit Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread.
- Normal_gaussian 10y agoIts never too early to show HN. Especially for open source projects how do you suggest you onboard people to get this thing progressed? The voting mechanism helps us determine whether it is interesting anyway.
- lobster_johnson 10y agoIs the idea that apps talk to PumpkinDB in order to achieve this layering, or do you see it as a library? Not often that you see MUMPS referenced on HN, by the way. It's one of those oddly productive niche languages that are (as far as I know) alive and well, but rarely encountered except if you work in that niche, e.g. finance or healthcare.
- yrashk 10y agoSome of the layering in terms of basic building blocks and higher level languages will become a standard part of PumpkinDB. For the rest, yes, it's expected that end applications or frameworks will compile higher level constructs to PumpkinScript. I've been following MUMPS and using it for some ideas for some time and that's how some of their ideas became inspirations for PumpkinDB. As quirky M is, MUMPS was indeed oddly productive and I wanted to piggyback on that.
- solidsnack9000 10y agoI hope that PumpkinDB is an alliterative reference to MUMPS :)
- yrashk 10y ago:D
- ams6110 10y agoThanks. I read the entire "Documentation" page and still didn't feel confident that I understood what this really was. "Event sourcing" to me implies that it's generating events.
- ziikutv 10y agoWell, its a fancy word. Thats basically what I took away from it. Also, I agree, read the whole doc and took way longer to understand what it does. Clearly, they are smart, but bad copywriters :) Please write better docs.
- yrashk 10y agoDoing our best, one step at a time! It's clear to me that we haven't spent enough time on documentation yet. The whole project is only 3 weeks old and our initial focus was to get a working version out with some level of documentation (a goal that we somewhat attained, I believe)
- ziikutv 10y agoFair enough. I'm gonna be done this shitty program and have a piece of papers with some fancy seal on it in a couple of months. I'd be happy to contribute. Bookmarked until then.
- yrashk 10y agoAlso, if this topic is interesting to anybody who's good at writing, we are very open to contributions of all kinds!
- yrashk 10y agoWe are definitely looking to improve the documentation. This is 0.1, after all. This project was started as a backend for a lazy event sourcing approach (https://blog.eventsourcing.com/lazy-event-sourcing-ed7e59007e17 https://blog.eventsourcing.com/lazy-event-sourcing-ed7e59007... , https://m.youtube.com/watch?v=aqv8d1pjmU8 https://m.youtube.com/watch?v=aqv8d1pjmU8) and beyond. The idea behind it is that it provides primitives for building systems that are designed around immutable events, journals, indexing, etc. Hence the current positioning. We thought it would be useful to be targeting fairly narrowly early on. Either way, we will definitely need to expand on that in our materials
- mtdewcmu 10y agoIt seems to serve approximately the same purpose as Berkeley DB.
- yrashk 10y agoWell, that would have been true if we had no PumpkinScript. Then it would have been just a tiny wrapper around LMDB that we are using as a storage backend, indeed.
- vtuulos 10y agoI am a big fan of this approach: Low-level, high-performance, immutable-by-default database engine as a building block, with bindings to multiple languages for easy application development. TrailDB (http://traildb.io http://traildb.io), which has many elements of event sourcing in it, follows this philosophy and it has proven to be pretty successful for its intended use cases. I was delighted to notice that PumpkinDB has an imperative query language inspired by Forth. We recently open-sourced a similarly imperative query language inspired by AWK, http://github.com/traildb/reel http://github.com/traildb/reel :) I will definitely follow PumpkinDB with great interest!
- nik736 10y agoWhat's an actual use-case for this? I am reading the documentation but still don't see why I should use it and what the actual advantages compared to current solutions are.
- yrashk 10y agoIt was built as a kind of a database constructor for event sourced / journalled systems. It's design inspiration is largely stemming from MUMPS which provided a great ("oddly productive") combination of a database and a programming language. Being a constructor it's also a great tool for building applications with a better control over querying mechanics (since everything is actually described in PumpkinScript)
- tyingq 10y agoIt sounds sort of like "overlayfs" for data. Which might allow a non-destructive, no-copy way to... - Do what-if analysis. Change the price of oil at some point in history and see how your financials would have played out from that point forward. - Fork your database and have two live copies acting on different data or rules for a live comparison...without all the plumbing overhead. Perhaps having one set work with a fiscal year that is calendar year, and another with a different fiscal boundary.
- makmanalp 10y agoVery interesting! I think one thing that this would benefit from is a lot of usage examples, especially around pumpkinscript. I was reading recently about MUMPS and Caché and it's interesting to see a modern implementation of similar ideas. One question - what is the storage layout like? Do you have plans to support efficient range queries at all?
- yrashk 10y agoWe definitely need better documentation! That's for sure. We only did a basic one just to get the basics out. As for the layout -- everything is built around btree k/v, and the original idea behind PumpkinDB is to give primitives that are useful in building databases, indices in particular. The expectation is that, over time, we will grow our library to have more sophisticated primitives, including ready-made indices of different kind. Does this help?
- playing_colours 10y agoWritten in Rust :) Inspiring, I am learning Rust now to try implementing HDFS-like storage.
- cestith 10y agoWhere would I use this in place of, say, Kafka and Samza?
- JoelSanchez 10y agoWhat an interesting commit message format.
- simonw 10y agoI like how every commit message is formatted as a problem and a solution: https://github.com/PumpkinDB/PumpkinDB/commits/master https://github.com/PumpkinDB/PumpkinDB/commits/master
- DTrejo 10y agoThis is going to make for some awesome release notes and new feature release marketing. Very well done @yrashk.
- jdiez17 10y agoIndeed, it's a neat "hack" to force yourself to write better commit messages. I think this style of commit messages originated from the zeromq community: https://github.com/zeromq/libzmq/commits/master https://github.com/zeromq/libzmq/commits/master https://github.com/zeromq/zproto/commits/master https://github.com/zeromq/zproto/commits/master
- yrashk 10y agoYes, I picked this style up form Pieter Hintjens
- digitalzombie 10y agoHoly cow it's in Rust. I'm doing a thesis in Classification Trees, doing R and hoping to do the backend of the R package in Rust (it looking to be C++). I'll look through the source code of this to see it's tree implementation. Probably used the rust standard library's implementation of BTree?
- elcritch 10y agoDocumentation says they use LMDB for the backend. Looking over the documentation, it looks like you could readily use pumpkin be directly to implement the database/datacaching scheme and interface with it from R. Unless your thesis is on implementation of B-trees, definitely try bootstrapping on something like this first. BTW, lmdb provides memory mapping which can be very fast for computations.
- fiatjaf 10y agoI don't get this whole "never overwrite data" thing, including Datomic, for example. Isn't the disk space needed for these schemes enormous?
- christophilus 10y agoI have the same question. Presumably, you could store fairly efficiently doing a git-like diff-based storage scheme or something. But I would be interested in hearing analysis of this.
- Salgat 10y agoExactly, events only represent changes in state. Furthermore, you aren't storing everything in the events; the business logic on services can add and build up models to much more than what the events hold (for example, an event could hold an id that the service uses to populate a bunch of properties into the actual model snapshot, something as simple as an order number would map to all kinds of order details).
- solidsnack9000 10y agoI have reservations, too: it's important to be able to remove data even though disk is cheap. * Removing very old data is a reasonable hedge for user privacy. * Sometimes confidential data makes its way into the data set and needs to be removed. * Old event data is often not useful but can impact performance or cost just the same. For example, one needs to allocate an EBS volume on AWS volume with a certain level of performance; but the cost of that is `IOPs * GBs`, not `IOPs * useful GBs`. * Replicating and backing up the dataset takes longer and longer as the application grows.
- yrashk 10y agoI agree, this is an importabt aspect. Our plan in PumpkinDB is to add key value association retirement, subject to defined retirement policies.
- stonewhite 10y agoLooks very interesting. I'd love to use it once it supports Akka persistence. Is this on the roadmap?