13 ms·
Immudb 1.0 – open-source, immutable database with SQL and verified timetravel
- artemonster 5y agoCan someone ELI5 what is an "immutable database"? If you can add to the table, that means mutation, right? I am missing something...
- qsort 5y agoIt's immutable in the same sense a purely functional data structure is immutable. You represent mutation by making a new version of the data structure. Of course you don't literally do that on the database because it would be inefficient, but there are several algorithmical tricks that can expose an interface that works as if.
- artemonster 5y agothat makes sense on a language level, when you hold a reference to some data and you can assume nothing can be changed about it. how does that hold on DB level?
- qsort 5y agoIn the same way. A database is basically just a giant data structure, a table is not unlike a B-Tree (in some engines it literally is a B-tree). Data warehouses already do something like this informally, as they are structured in a star schema around a single "append-only" fact table.
- deleted 5y ago[deleted]
- ianpurton 5y agoYou would be able to query and INSERT but not DELETE and UPDATE. This is useful for example in banking applications that keep an audit trail for example. A sysadmin would not be able to update or delete items in the audit table and so can't cover up a crime. If the database is tampered with at the file level, they have a way to detect that. (Probably some kind of merkle tree.)
- artemonster 5y agoallright, makes perfect sense. thank you!
- goto11 5y agoIt basically means "append only". You can add new data to the database, but you can't change or delete existing data.
- dspillett 5y ago> immudb is the first database to provide tamper-evident data management, immutable history and client-cryptographic proof. Every change is preserved and can't be changed without clients noticing. Sounds like they are recording all changes (like SQL2011's system versioned tables, as implemented more-or-less by several common DB engines) but with some sort of hash-chain ledger so that history can be verified and therefore any tampering detected. > If you can add to the table, that means mutation, right? It isn't keeping the current view of the data immutable, but is keeping an immutable history of the data. It is immutable in the sense that nothing written to it is ever lost, and you can use the "time-travel" query functions (like SELECT stuff FROM atable FOR SYSTEM_TIME AS OF '2021-03-05') to retrieve it even if it looks to have been completely mangled or deleted if you use a non-time-travelling query.
- f38zf5vdt 5y agoSQL system versioned tables but with git hash tree versioning for every mutable command.
- endymi0n 5y agoThere, I did it for you in PostgreSQL: ALTER TABLE table_name SET (autovacuum_enabled = false); Snark aside, it‘s still not 100% clear what‘s the upside of using a completely different database, just for that use case.
- _bohm 5y agoHuh? Dead tuples are not queryable in Postgres.
- anentropic 5y ago> immudb is the first database which allows you to do queries across time. I don't think it is e.g. Datomic already had this for a long time, no?
- branko_d 5y agoOracle has had flashback queries for a long time. Though this does not do what immudb claims: > immudb is the first database to provide tamper-evident data management, immutable history and client-cryptographic proof. And: > Clients do not need to trust the server and every new client adds trust to the deployment
- waheoo 5y agoYes. https://youtube.com/watch?v=Cym4TZwTCNU https://youtube.com/watch?v=Cym4TZwTCNU
- endymi0n 5y agoDatomic does, so does Oracle, Snowflake and BigQuery.
- CharlesW 5y agoTeradata Vantage, too.
- dspillett 5y agoSeveral databases (MS SQL Server, MariaDB, Postgres with appropriate extension) support system versioned temporal tables (added in the SQL2011 standard, though I don't know if any DB entirely follows the standard) which I'm pretty sure counts as "queries across time". Maybe they are claiming to be the first with it built-in as a core part of the engine that it is specifically optimised for, but even that might not be true.
- refset 5y ago> even that might not be true It's not. For example, see SAP HANA's "Timeline Index" https://websci.informatik.uni-freiburg.de/publications/sigmod2013-timeline https://websci.informatik.uni-freiburg.de/publications/sigmo...
- tutfbhuf 5y agoI would like to have such a database based on git. Where every change is a git commit. This should then work with things like github where you can connect to your database via github api. The db git repositories could be either private or even public. You can then deploy a serverless webpage to gh-pages and use a serverless gh-gitdb as storage. serverless := you don't have to operate the infrastructure yourself
- agbell 5y agoIt seems like this is somewhat in that direction. It looks like it is using merkle trees to store the history.
- quasiperson 5y agoYou should check out https://www.dolthub.com/ https://www.dolthub.com/ then. They are working on something very similar.
- lifty 5y agoCheck out https://replicache.dev https://replicache.dev and https://github.com/attic-labs/noms https://github.com/attic-labs/noms
- gregwebs 5y ago> Data stored in immudb is cryptographically coherent and verifiable, just like blockchains, but without all the complexity. Unlike blockchains, immudb can handle millions of transactions per second, and can be used both as a lightweight service or embedded in your application as a library. > Companies use immudb to protect credit card transactions and to secure processes by storing digital certificates and checksums. This explanation is available on their github repo [1]. It has been a common refrain on Hacker News that you don't need a blockchain and instead can just use a database, but this product may actually fill the gap where tamper resistance is desired. [1] https://github.com/codenotary/immudb https://github.com/codenotary/immudb
- simtel20 5y agoYeah, this interests me because I'm thinking about how to use grafeas - it's role is critical for reliable software development going forward - but storing it's data in a backend like this would add one more layer of trust and verifiability to a software supply chain. There are some interesting possibilities with making e.g. public software repos' metadata clonable verifiable and queryable via local immutable copies.
- decodebytes 5y agoMaybe take a look at rekor, part of the sigstore project, it's built specifically for software supply chain transparency (disclaimer I am one of the community): https://github.com/sigstore/rekor https://github.com/sigstore/rekor
- codetrotter 5y ago> this product may actually fill the gap where tamper resistance is desired I think in the future, all enterprise storage solutions will be append-only by default. To protect against cryptolocker malware. But also with isolated functionality for actually deleting data, for example because of GDPR requests or because of malware that tries to fill all writable storage with garbage. So that data can still be deleted, but not from any of the regular servers that are reading and appending data to the system. Instead from separate servers that are isolated and for data storage management only.
- supergirl 5y agonot exactly immutable is it? their docs say you can do UPSERT for example. the key is that once you update something, the clients can check using crypto that something was changed. you can't do this in regular databases.
- dmacvicar 5y agoImmutable in the sense that the old value is preserved, even if you update it, and you can't change the history (tamper-evident).
- endisneigh 5y agoHow is this any different than taking every mutation, signing it using whatever signing mechanism you'd like and adding a column, in addition to the ones you'd like with the hash. Then, if anything changes you know it's been mutated because the computed signature has changed.
- ianpurton 5y agoYour solution wouldn't handle the case of row deletion. It's a little harder than you might think to make a database with tamper resistance.
- endisneigh 5y agoOh I'm sure - but without delving into philosophy, how would you know that something was deleted and tampered with vs. Immudb (for example) being compromised and turns out it's possible to delete something without you knowing vs. it never existed to begin with? In my mind the only way to guarantee is to maintain a copy yourself and check against the "original", but if you're going to do that, then what I described is sufficient, no? I only mention this because the project mentions that the history is protected by clients, which I imagine is similar to what I'm describing, e.g. copying and checking against the original.
- ianpurton 5y ago> In my mind the only way to guarantee is to maintain a copy yourself and check against the "original", but if you're going to do that, then what I described is sufficient, no? The attacker in that case could update your copy. But you have somewhat started to fix the issue. To cover the case where a bad admin has access to the DB and any copies, you need to send a hash every so often to an outside source. In this case they use clients (I'm not sure exactly how they do this). In fact you need a list of hashes one for every 100 rows for example. Re-generated the hashes and checking against an external source should detect a tamper. In the case of Bitcoin (which is extremely tamper resistant) every node operator is a validator. The hashes are stored in a merkle tree.
- 5y ago
- robto 5y agoReminds me a lot of Fluree[0], an immutable, cryptographically verifiable, temporal database, but with RDF as a query language, which I think is very nice. SQL is nice because it's familiar but it's honestly not that hard to improve on. [0]https://flur.ee/ https://flur.ee/
- nerdponx 5y agoSo is this something I would want to use for a basic CRUD application, and reap the benefits of time travel and immutability? Or are there downsides that would relegate it to specific use cases? A what would those use cases be?
- brokencube 5y agoIt wouldn't be suitable for any application where you care about GDPR (i.e. you store personal information and have users in the EU) The "right to be forgotten" is not compatible with immutable data. You can't simply need to mark data as deleted, you need to 'purge' it from your system (and possible backups, depending on how long you keep historic backups) - that isn't possible in a system with immutable data.
- blablabla123 5y agoI mean there are solutions for this. About CQRS/Event sourcing I've read that it's possible to solve it by encrypting the data with different keys and then rotating/throwing away the keys every now and then. Seems a bit hacky but probably there are more elegant approaches.
- hutrdvnj 5y agoWhat happens if you have to delete some data e.g. due to law?
- jacquesm 5y agoYou have several options here: - store the data encrypted using a secondary protocol, lose the key - rewrite the whole db If either of these is not feasible then you should have thought longer about what tech is suitable for which application. Operating your company in a legal manner is a pretty strong factor when making such choices.
- remram 5y agoIs losing the key sufficient to comply with the law? "We didn't actually delete anything but I promise I don't remember how to decrypt it" would be acceptable for the court to not e.g. seize your drives?
- speed_spread 5y agoIt's the same as "we actually deleted the data and I promise we didn't keep any backup copies", except it's probably even easier to enforce, since you already to have to secure the key instead of the whole database.
- imhoguy 5y agoIANAL With GDPR right to forget you need to get rid of any identifable subject information. If you can't tell a subject from data then you comply. Encrypted data without a key is just a noise. You are allowed to keep aggregations and hashes of data. These shouldn't allow to identify a subject. E.g. you can keep list of banned emails as MD5s to verify on sign up etc.
- remram 5y agoIn this situation though, any client who still knows the key can access the data, since there is no way to remove data from the database server, or make it unavailable at the server level. Assuming the clients and server are operated by different entities (otherwise the immutability and verifiability are not that interesting), if someone comes to the server operator with a court order and ask that data be removed, it seems like there is nothing they can do.
- foobarbazetc 5y agoDefinitely not the first database to allow time travel, TM or not.
- cyberge99 5y agoI love what you’ve done. I think you may have an issue with the TimeTravel trademark however. Snowflake uses it in your exact market segment (not to mention where else it may be used in a similar context). Good stuff though, I’ll be checking it out.
- 0xbadcafebee 5y ago> This new functionality allows travel back in time through the data change history, and even compares these values in the same query! So we can actually treat our databases like immutable infrastructure and actually roll back changes now without the hulking cludge that is snapshots/restores and database migrations? That's game-changing.
- JulianMorrison 5y agoIf this is deployed in a situation where record volumes are large, example: recording credit card transactions, there is going to have to be a process to "retire" old records (and perhaps, move them to external archives). The alternative is endlessly growing storage, and the resulting performance degradation. At a first glance, I don't see anything like that in there.
- arpinum 5y agoThe QLDB performance comparison looks quite dodgy, but I can't find their QLDB benchmark code to see what they are doing wrong.
- parentheses 5y agoSeems like a database that stores content hashes. Very cool but what makes it better than simply adding a table to my database (or a DB specifically for this) and running `insert into content_hashes...`? The above approach also allows me to choose any database because I can model this data however I want.
- jeroiraz 5y agoimmudb can hold the actual data. An equivalent approach using an existent database without this features will involve creating a cryptographic data structure which captures not only individual content but the entire history of changes. Also having the functionality to construct and verify the cryptographic proofs to validate read data
- LukeEF 5y agoThere seems to be a growth in the number of time traveling immutable-first databases available. We have OpenCrux, Datomic, TerminusDB, Noms, Dolt, and now Immudb. Three using datalog for query and two forcing SQL (not sure about Noms). What sort of use cases are most common? GitHub repository says: > Companies use immudb to protect credit card transactions and to secure processes by storing digital certificates and checksums. But I am not sure how people are building that into their architecture to be honest.
- clusterhacks 5y agoI have used "immutable" schema designs when there were strong requirements for full audit needs over time. It works very well even in a normal RDBMs system. It also allows some very neat reporting e.g. compare the same report at different points in time. The basic idea was that every operation (create, update, delete) are actually normal SQL inserts and all reads are against views defined such that the most recent tuples are returned unless they are flagged as "deleted." I have typically used these types of designs in mostly simple applications with tables where the row counts are in the low millions of tuples. Dealing with this design in the billions of tuples (probably sharded somehow) might have motivated us out of normal RDBMs and into one of the specialized immutable DBs mentioned.
- ak39 5y agoThanks. Can you comment on how this differs from a “mutable” RDBMS model but one with automatic history based on triggers for example?
- akra 5y agoI would imagine there are a few differences. Events are the primary entity, and the current state is simply a projection of that not the other way around. Those events may come from other systems and are often defined in business terms, not SQL terms. For example an event may also constitute a business update which can update one or many tables. Think of a transaction event updating a balance for two accounts. TL;DR My thinking it allows you to capture more the intent of that event. Although I'm not sure you need an immutable database to do this from scratch - I've seen this in schema designs in the past?
- boshomi 5y agoGDPR requires to erease user date if users withdraw their consent or their data are no longer required for purpose which you originally collected or processed it for. Therefore, you must carefully check that no personal data is stored in immutable databases.
- vchain-dz 5y agostay tuned - this is on the immudb roadmap not too far in the future
- vchain-dz 5y agothe importance is to maintain full history and verification of your actions as well, so you have proof of the value deletion.
- pihentagy 5y agoSo should you also delete userdata from existing backups? :)
- deknos 5y agothis is hugely interesting, i have to look into this, but... for dev/test environments, can i have a "unverified" version, where clients reget/reset the state?
- 1cvmask 5y agoAny major customers using this and if so how?
- alrs 5y ago> For any question contact us on Discord. Hard no.
- dmacvicar 5y agoAlternatively, the team is hosting a virtual release-party on May 31st, 2021 at 6pm (18:00) CET. https://www.codenotary.com/blog/immudb-release-1-0-release-party/ https://www.codenotary.com/blog/immudb-release-1-0-release-p...
- dmacvicar 5y agoThe team will host a release party on Monday, May 31st at 6pm CET (18:00) - 10:00 AM PDT. If you have questions about immudb, you are welcome to join us! https://www.codenotary.com/blog/immudb-release-1-0-release-party https://www.codenotary.com/blog/immudb-release-1-0-release-p...