12 ms·

immudb – world’s fastest immutable database, built on a zero trust model

226 points by dragonsh 5y ago

tarr11 5y ago
Previous HN thread about immutable databases: https://news.ycombinator.com/item?id=23290769 https://news.ycombinator.com/item?id=23290769
1cvmask 5y ago
Words like immutable make me allergic
- abc_lisper 5y ago
  See a doctor then . It isn’t expected. Could be a lack of CS education, in which case, read some books. If that doesn’t fix it, see a psychiatrist - something could be wrong with your brain.
ShamelessC 5y ago
> Data stored in immudb is cryptographically coherent and verifiable. Unlike blockchains, immudb can handle millions of transactions per second, and can be used both as a lightweight service or embedded in your application as a library. immudb runs everywhere, on an IoT device, your notebook, a server, on-premise or in the cloud. Seems pretty useful actually. Can anyone with a relevant background comment on when this would be a bad idea to use?
- KarlKemp 5y ago
  The data that is at risk of being changed with malicious intent is certainly not insignificant, but still just a fraction of all data. Changing to this adds a new and complicated system, replacing whatever you're currently using, which will have seen far better testing and is known by the people working with it.
- staticassertion 5y ago
  If you can trust your writers there's likely no need for this. A modern approach tends to have databases owned by a single service, which exposes the model via RPCs. So you generally don't have more than one writer, which means you're pretty much de-facto "zero trust" if that single writer follows a few rules (ie: mutual auth, logging, etc). But in some cases you don't have that same constraint. For example, databases that store logs (Elastic, Splunk, etc) might have many readers and writers, including humans. In that case enforced immutability might be a nice property to have. Attackers who get access to your Splunk/ES cluster certainly will have fun with it.
- imglorp 5y ago
  There are a few properties to be aware of. Although it might be a KV store, you're probably going to want sensible queries using other than the primary key. Eg time series or secondary keys. So in addition to the KV store, there is probably a need for an external index and query mechanism. Another issue is obtaining consistent hashing, where multiple documents might have the same content but vary by order or by date format. Finally, do you have to go to the beginning and hash everything to get a proof of one transaction, or is there some shortcut aggregation possible? We evaluated AWS QLDB for these things in our application as a financial ledger and were impressed at their progress with a novel data store. They invented some of the tech in house for this product instead of grabbing an off the shelf open product. Lockin would be a downside here. Immudb looks promising because it's not locked to a cloud host. https://aws.amazon.com/qldb/faqs/ https://aws.amazon.com/qldb/faqs/
  - mistrial9 5y ago
    > Eg time series or secondary keys not an "all or nothing" question.. for example, a fast-enough "return the most recent in a time series" is not exactly time-series, but solves many use cases
  - jaboutboul 5y ago
    note that it does KV and SQL
- rattlesnakedave 5y ago
  Seems like it would still be vulnerable to rollback attacks. Signed rows would probably get you farther with less novel tech involved if you want immutability.
furstenheim 5y ago
GDPR compliance will be tricky. How does one delete data?
- okr 5y ago
  You clone the database and remove/update the corresponding lines. GDPR does not mean, you have to fix it right away, imho.
  - gnabgib 5y ago
    Article 17 does include the term "without undue delay"[0], but such vague language seems ripe for some court precedent. A clone and remove/update per GDPR request seems like undue delay, certainly one that could be avoided by alternative architecture choices (keep the personally identifiable information (PII) in a mutable store) [0]: https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/
    - deleted 5y ago
      [deleted]
    - sigzero 5y ago
      No, it's not undue delay. That's just how it currently works and that is a fine argument.
  - peoplefromibiza 5y ago
    But you have to do in a pretty short timeframe > Under Article 12.3 of the GDPR, you have 30 days to provide information on the action your organization will decide to take on a legitimate erasure request. This timeframe can be extended up to 60 days depending on the complexity of the request. even if they ask for more time, first communication has to come within 30 days
    - topspin 5y ago
      So, no deadline for the actual deletion? Your 'first communication' could say "your data will be deleted at some point between now and ten years from now" and be compliant?
artemonster 5y ago
Can someone ELI5 how immutability applies to databases and which advantages it brings. Thank you!
- throwaway984393 5y ago
  Immutability is probably the most powerful concept that applies to how modern technology can be used. Versioned, immutable, and cryptographically-signed artifacts do a bunch of things for you. From an operational standpoint, it allows you to roll out a change in exactly the way you tested, confident that it will work the way it's intended. It also allows you to roll back or forward to any change with the same confidence. It also means you can restore a database immediately to the last known good state. Changes essentially cannot fail; no monkey-patching a schema or dataset, no "migrations" that have to be meticulously prepared and tested to make sure they won't accidentally break in production. From a security and auditing standpoint, it ensures that a change is exactly what it's supposed to be. No random changes by who-knows-who at who-knows-when. You see a reliable history of all changes. From a development standpoint, it allows you to see the full history of changes and verify the source or integrity of data, which is important in some fields like research.
  - bob1029 5y ago
    There is also a performance advantage if you can build everything under these constraints. A pointer to something held in an immutable log will never become invalid or otherwise point to garbage data in the future. At worst, whatever is pointed to has since been updated or compensated for in some future transaction which is held further towards the end of the log. Being able to make these assumptions allows for all kinds of clever tricks. The inability to mutate data pointed to in prior areas of the log does come with tradeoffs regarding other performance optimizations that expressly rely on mutability, but in my experience constraining the application to work with an immutable log (i.e. dealing with stale reads & compensating transactions) usually results in substantial performance uplift compared to solutions relying on mutability. One recent idea that furthers this difference is NAND storage, where there may be a substantial cost to be paid if one wants to rewrite prior blocks of data (depending on the type of controller/algorithm used by the device).
throwaway984393 5y ago
Don't forget to star this repo if you like immudb! I didn't realize GitHub had "Like and subscribe" culture now. : /
willcipriano 5y ago
> You can add new versions of existing records, but never change or delete records. This lets you store critical data without fear of it being tampered. > immudb can be used as a key-value store or relational data structure and supports both transactions and blobs, so there are no limits to the use cases. This is game changing. Use it for say a secondary data store for high value audit logs. I'll consider using it in the future.
- dillondoyle 5y ago
  I don't totally understand the value of the second, but isn't the first already exist in things like BigQuery?
  - zimpenfish 5y ago
    > isn't the first already exist in things like BigQuery? You can truncate a BQ table and reload it if you want to change things. Had to do this at a previous gig (twice a day!) because the data warehouse people would only take data from BQ but the main data was in Firebase (yes, it was an insane place.)
    - dillondoyle 5y ago
      oh that sounds horrible !! i would be terrified to mess that up
  - mathnmusic 5y ago
    Or a traditional database with read-only credentials and a function that adds "ORDER BY version DESC LIMIT 1".
    - deleted 5y ago
      [deleted]
    - lojack 5y ago
      but is a traditional database cryptographically secure? if a super user with write permissions (or, say, direct access to the physical data store) modifies records are users able to validate the integrity of the data?
ledgerdev 5y ago
Does this have, or are there any plans for a change-feed? Has anyone used this as an event sourcing db?
gigatexal 5y ago
So is this a useful alternative to blockchains or just hype?
chalcolithic 5y ago
>millions of transactions per second I wonder if I wanted to survey a landscape of all databases that claim such numbers how could I possibly find them?
timdaub 5y ago
I went on their website and tried to understand how immutability is enforced but I couldn't find anything. I'm sceptical, but particularly because they make a deliberate comparison to blockchain that I doubt they'll be able to deliver. The PoW immutability of e.g. BTC and ETH is strong as it yields the following guarantees for stored data: - Immutability of the BTC blockchain is protected through all cummulative work that has happened on a specific branch of the chain. Even if someone replayed BTC, it'd take millenias to recompute the work on an average machine - The immutability isn't enforced on a file level, as I suspect it is with immudb. Immutability is enforced through the network that has additionally shown to have conservative political views too. You can go, sync a BTC node and change the underlying level db. Still that won't change the network state. Immutability on a single system is physically impossible if e.g you consider deleting the file as mutation. - immudb says "it's immutable like a blockchain but less complicated", but Bitcoin isn't more complicated than some sophisticated enterprise db solution. - I think immudb should be maximally upfront what they mean by immutability: It seems they want to communicate that they're doing event sourcing - that's different from immutability Finally there's a rather esotheric argument. If you run an immutable database as an organizatiom where one individual node cannot alter the network state but you have (in)direct control over all nodes: Isn't it always mutable as you could e.g. choose to swap out consensus? So from a philosophical perspective, then immutability can truly only occur if mutability is out of an individual's control. Why do I have the authority to say this? Because I too have once worked for a database with blockchain characteristics called https://www.bigchaindb.com https://www.bigchaindb.com Edit: The best solution that also has a theoretically unlimited throughput is this toy project: https://github.com/hoytech/quadrable https://github.com/hoytech/quadrable Conceptually, it computes a merkle tree over all data and regularly commits to Ethereum. Through this commitment the data may still change locally: But then at least would be provably tampered. So I guess for databases, the artibute we can really implement is "tamper-proof".
- jandrese 5y ago
  The big question is if someone gets on your DB server and wants to change a record how does the software prevent them from altering a record and then recomputing the remainder of the chain?
YogurtFiend 5y ago
I'm not sure that this is a _useful_ tool. Let's talk about the threat model or the attacks that this defends against. If a Client is malicious, they might try to manipulate the data in the database in an untoward way. In a "normal" database, this might cause data loss, if the database isn't being continuously backed up. But immudb does continuous backups (effectively, since it's immutable) so, if a malicious client has been detected, it's possible to restore an older version of the database. The real problem is how would you know that a client has tampered with your database? Well, because this database is "tamper-proof," duh! But the issue lies in the definition of tamper-proof. From my reading of the source code and documentation, the "proof that no tampering has occurred" is a proof that the current state of the database can be reached by applying some database operations to a previous state. As a result, a malicious client could simply ask the database to "delete everything and insert this new data," to make the database look like whatever it wanted. This is a valid way to transition the state of the database from its old state to the new state, and so shouldn't be rejected by the tamper detection mechanism. "Ah," but you say, "it would look super sus [as the kids say] to just delete the entire database. We'd know that something was up!" The problem with this solution is how are you going to automate "looking super sus?" You could enact a policy to flag any update that updates more than N records at a time, but that's not really a solution. The "right" solution is to trace the provenance of database updates. Rather than allowing arbitrary database updates, you want to allow your database to be changed only by updates that are sensible for your application. The _actual_ statement you want to prove is that "the current state of the database is a known past state of the database updated by operations that my application ought to have issued." Of course what are "operations that my application ought to have issued?" Well, it depends how deep you want to go with your threat model. A simple thing you could do is have a list of all the queries that your application issues, and check to make sure all operations come from that list. This still allows other attacks through, and you could go even more in depth if you wanted to. Importantly, immudb doesn't appear to contend with any of this. They claim that their database is "tamper-proof," when in reality you'd need a complicated external auditing system to make it meaningfully tamper-proof for your application. (Again, a threat model ought to include a precise definition of "tamper-proof," which would help clear up these issues.) It's also worth comparing this to https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency, which is an append-only database. Compared to immudb, the _exposed data model_ for certificate transparency logs is an append-only set, which means that it doesn't have any of these same problems. The problem with immudb is that the data model it exposes is more complicated, but it's built-in verification tools haven't been upgraded to match. (Also, for context, I've tried to obtain a copy of their white paper, but after an hour the email with the link to it never arrived.)
cabalamat 5y ago
Would it be possible to have something like this that works by writing to a PROM? That would make it immutable at the hardware level.
pharmakom 5y ago
Is it possible to delete data for compliance reasons? Not as a frequent operation, but say on a monthly batch?
- jeroiraz 5y ago
  logical deletion is in place, physical deletion is already on the roadmap
  - kristjansson 5y ago
    That’s a lot of words to say no
  - xvector 5y ago
    How can physical deletion work with the Merkle Tree mechanics? Do you just store the hash but not the underlying KV pair?
    - jeroiraz 5y ago
      the hash tree, transactions headers, payloads and index are all independently stored. So it will be possible to physically delete payloads and still be able to build all the proofs (but not being able to provide the actual values). Truncating transaction headers should be possible, but each physical deletion will be associated to a real lost of data, so proof generation may be affected on deleted data.
newtonapple 5y ago
Has anyone tried immudb in production? What are some of immudb's performance characteristics? It'd be nice to know how it performs under various conditions: query per sec, database / table sizes, SQL join performance etc. Also, what are the system requirements for immudb? What kind of machine would I need to run a medium to large website (say, 1TB of data, 5-25K qps, e.g. Wikipedia)? It mentioned in the documentation that it can use S3 as its storage? Are there performance implications if you do this?
spookylukey 5y ago
I find it very strange to have claims about the database being tamper-evident etc without a clear description of the threat/trust model, and how/for whom it works. For example, what data does the client need to store to be sure no tampering has occurred?
- YogurtFiend 5y ago
  I agree 100% with the need for the threat model. I think the current system is, at-best, a system with a tamper-evident audit log. Which, could still be interesting, but the authors' use of ill-defined terms makes it easy for people to think their software does something that it's not. What's supposed to happen is that the server will give the client a path in a https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree to the current state, the prove that the key-value pair is included in the Merkle tree. There are, however, some subtle issues which can arise if you're not careful. In particular, what happens if I set the key k to value v1, and then set the key k to value v2? If I subsequently ask for the value of k, I ought to see v2, and a proof that k is v1 shouldn't check out. However, in order for this to work, it's not sufficient for the server to prove that (k, v) is in the audit log, since that would allow for the server to maliciously roll-back the state. What you really want to prove is that v is not just _a_ value that k was set to, but _the most recent value_. It's unclear to me whether the code actually does this--there's no architecture guide which describes the cryptographic algorithms at play (or what the threat model is), and the code appears to be mostly devoid of comments. There is a reference to separate inclusion and consistency proofs, which might be this distinction. But it's really hard to tell from the digging that I've done.
  - jeroiraz 5y ago
    This is a very interesting point. Rollbacks are protected by consistency proofs and with inclusion proofs it's possible to detect such situation but it may require scanning over the transactions. immudb does not only provide access by key or key prefixes but it's also possible to fetch a particular tx by its unique identifier or scan over them. For sure it's not the optimal solution, some ideas to cover this scenario were discussed but not yet fully defined.
infogulch 5y ago
I'm interested in how they intend to implement data pruning from the roadmap.
vyrotek 5y ago
Is this comparable or different than these "cryptographically verifiable" ledger DB services? https://docs.microsoft.com/en-us/azure/azure-sql/database/ledger-overview https://docs.microsoft.com/en-us/azure/azure-sql/database/le... https://aws.amazon.com/qldb/ https://aws.amazon.com/qldb/
renewiltord 5y ago
If I drop the trust requirement, what's the absolutely fastest blazing fast thing I can use that is network readable/writable and fault-tolerant?
throwaway81523 5y ago
This sort of reminds me of happstack, though the design and implementation are much different.
yencabulator 5y ago
Note: no `ALTER TABLE` in their SQL. Sounds like maintaining this over longer term with evolving data would be quite painful.
- jeroiraz 5y ago
  sure, schema changes are already in the roadmap
yencabulator 5y ago
They talk about performance a lot, but their benchmarks seem to always explicitly batch the inserts. I'm seeing <700 inserts per second in a simple loop. Quietly reporting batched inserts is a subtle lie.
- jeroiraz 5y ago
  each transaction may have several entries, in order to gain more performance, it's better to include several entries per transaction. But all the entries in the same transaction are atomically stored. submitting several transactions at the same time is also possible and convenient, processing is concurrently done until the last step which requires a strict serialization. benchmarking is a huge topic and we'd love to have contributions on this. It may refer to using embedded immudb, as a stand-alone server, kv, sql, etc....

226 points by dragonsh 5y ago

Previous HN thread about immutable databases: https://news.ycombinator.com/item?id=23290769 https://news.ycombinator.com/item?id=23290769

1cvmask 5y ago

Words like immutable make me allergic

abc_lisper 5y ago
See a doctor then . It isn’t expected. Could be a lack of CS education, in which case, read some books. If that doesn’t fix it, see a psychiatrist - something could be wrong with your brain.

ShamelessC 5y ago

> Data stored in immudb is cryptographically coherent and verifiable. Unlike blockchains, immudb can handle millions of transactions per second, and can be used both as a lightweight service or embedded in your application as a library. immudb runs everywhere, on an IoT device, your notebook, a server, on-premise or in the cloud. Seems pretty useful actually. Can anyone with a relevant background comment on when this would be a bad idea to use?

KarlKemp 5y ago
The data that is at risk of being changed with malicious intent is certainly not insignificant, but still just a fraction of all data. Changing to this adds a new and complicated system, replacing whatever you're currently using, which will have seen far better testing and is known by the people working with it.
staticassertion 5y ago
If you can trust your writers there's likely no need for this. A modern approach tends to have databases owned by a single service, which exposes the model via RPCs. So you generally don't have more than one writer, which means you're pretty much de-facto "zero trust" if that single writer follows a few rules (ie: mutual auth, logging, etc). But in some cases you don't have that same constraint. For example, databases that store logs (Elastic, Splunk, etc) might have many readers and writers, including humans. In that case enforced immutability might be a nice property to have. Attackers who get access to your Splunk/ES cluster certainly will have fun with it.
imglorp 5y ago
There are a few properties to be aware of. Although it might be a KV store, you're probably going to want sensible queries using other than the primary key. Eg time series or secondary keys. So in addition to the KV store, there is probably a need for an external index and query mechanism. Another issue is obtaining consistent hashing, where multiple documents might have the same content but vary by order or by date format. Finally, do you have to go to the beginning and hash everything to get a proof of one transaction, or is there some shortcut aggregation possible? We evaluated AWS QLDB for these things in our application as a financial ledger and were impressed at their progress with a novel data store. They invented some of the tech in house for this product instead of grabbing an off the shelf open product. Lockin would be a downside here. Immudb looks promising because it's not locked to a cloud host. https://aws.amazon.com/qldb/faqs/ https://aws.amazon.com/qldb/faqs/
- mistrial9 5y ago
  > Eg time series or secondary keys not an "all or nothing" question.. for example, a fast-enough "return the most recent in a time series" is not exactly time-series, but solves many use cases
- jaboutboul 5y ago
  note that it does KV and SQL
rattlesnakedave 5y ago
Seems like it would still be vulnerable to rollback attacks. Signed rows would probably get you farther with less novel tech involved if you want immutability.

furstenheim 5y ago

GDPR compliance will be tricky. How does one delete data?

okr 5y ago
You clone the database and remove/update the corresponding lines. GDPR does not mean, you have to fix it right away, imho.
- gnabgib 5y ago
  Article 17 does include the term "without undue delay"[0], but such vague language seems ripe for some court precedent. A clone and remove/update per GDPR request seems like undue delay, certainly one that could be avoided by alternative architecture choices (keep the personally identifiable information (PII) in a mutable store) [0]: https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/
  - deleted 5y ago
    [deleted]
  - sigzero 5y ago
    No, it's not undue delay. That's just how it currently works and that is a fine argument.
- peoplefromibiza 5y ago
  But you have to do in a pretty short timeframe > Under Article 12.3 of the GDPR, you have 30 days to provide information on the action your organization will decide to take on a legitimate erasure request. This timeframe can be extended up to 60 days depending on the complexity of the request. even if they ask for more time, first communication has to come within 30 days
  - topspin 5y ago
    So, no deadline for the actual deletion? Your 'first communication' could say "your data will be deleted at some point between now and ten years from now" and be compliant?

artemonster 5y ago

Can someone ELI5 how immutability applies to databases and which advantages it brings. Thank you!

throwaway984393 5y ago
Immutability is probably the most powerful concept that applies to how modern technology can be used. Versioned, immutable, and cryptographically-signed artifacts do a bunch of things for you. From an operational standpoint, it allows you to roll out a change in exactly the way you tested, confident that it will work the way it's intended. It also allows you to roll back or forward to any change with the same confidence. It also means you can restore a database immediately to the last known good state. Changes essentially cannot fail; no monkey-patching a schema or dataset, no "migrations" that have to be meticulously prepared and tested to make sure they won't accidentally break in production. From a security and auditing standpoint, it ensures that a change is exactly what it's supposed to be. No random changes by who-knows-who at who-knows-when. You see a reliable history of all changes. From a development standpoint, it allows you to see the full history of changes and verify the source or integrity of data, which is important in some fields like research.
- bob1029 5y ago
  There is also a performance advantage if you can build everything under these constraints. A pointer to something held in an immutable log will never become invalid or otherwise point to garbage data in the future. At worst, whatever is pointed to has since been updated or compensated for in some future transaction which is held further towards the end of the log. Being able to make these assumptions allows for all kinds of clever tricks. The inability to mutate data pointed to in prior areas of the log does come with tradeoffs regarding other performance optimizations that expressly rely on mutability, but in my experience constraining the application to work with an immutable log (i.e. dealing with stale reads & compensating transactions) usually results in substantial performance uplift compared to solutions relying on mutability. One recent idea that furthers this difference is NAND storage, where there may be a substantial cost to be paid if one wants to rewrite prior blocks of data (depending on the type of controller/algorithm used by the device).

throwaway984393 5y ago

Don't forget to star this repo if you like immudb! I didn't realize GitHub had "Like and subscribe" culture now. : /

willcipriano 5y ago

> You can add new versions of existing records, but never change or delete records. This lets you store critical data without fear of it being tampered. > immudb can be used as a key-value store or relational data structure and supports both transactions and blobs, so there are no limits to the use cases. This is game changing. Use it for say a secondary data store for high value audit logs. I'll consider using it in the future.

dillondoyle 5y ago
I don't totally understand the value of the second, but isn't the first already exist in things like BigQuery?
- zimpenfish 5y ago
  > isn't the first already exist in things like BigQuery? You can truncate a BQ table and reload it if you want to change things. Had to do this at a previous gig (twice a day!) because the data warehouse people would only take data from BQ but the main data was in Firebase (yes, it was an insane place.)
  - dillondoyle 5y ago
    oh that sounds horrible !! i would be terrified to mess that up
- mathnmusic 5y ago
  Or a traditional database with read-only credentials and a function that adds "ORDER BY version DESC LIMIT 1".
  - deleted 5y ago
    [deleted]
  - lojack 5y ago
    but is a traditional database cryptographically secure? if a super user with write permissions (or, say, direct access to the physical data store) modifies records are users able to validate the integrity of the data?

ledgerdev 5y ago

Does this have, or are there any plans for a change-feed? Has anyone used this as an event sourcing db?

gigatexal 5y ago

So is this a useful alternative to blockchains or just hype?

chalcolithic 5y ago

>millions of transactions per second I wonder if I wanted to survey a landscape of all databases that claim such numbers how could I possibly find them?

timdaub 5y ago

I went on their website and tried to understand how immutability is enforced but I couldn't find anything. I'm sceptical, but particularly because they make a deliberate comparison to blockchain that I doubt they'll be able to deliver. The PoW immutability of e.g. BTC and ETH is strong as it yields the following guarantees for stored data: - Immutability of the BTC blockchain is protected through all cummulative work that has happened on a specific branch of the chain. Even if someone replayed BTC, it'd take millenias to recompute the work on an average machine - The immutability isn't enforced on a file level, as I suspect it is with immudb. Immutability is enforced through the network that has additionally shown to have conservative political views too. You can go, sync a BTC node and change the underlying level db. Still that won't change the network state. Immutability on a single system is physically impossible if e.g you consider deleting the file as mutation. - immudb says "it's immutable like a blockchain but less complicated", but Bitcoin isn't more complicated than some sophisticated enterprise db solution. - I think immudb should be maximally upfront what they mean by immutability: It seems they want to communicate that they're doing event sourcing - that's different from immutability Finally there's a rather esotheric argument. If you run an immutable database as an organizatiom where one individual node cannot alter the network state but you have (in)direct control over all nodes: Isn't it always mutable as you could e.g. choose to swap out consensus? So from a philosophical perspective, then immutability can truly only occur if mutability is out of an individual's control. Why do I have the authority to say this? Because I too have once worked for a database with blockchain characteristics called https://www.bigchaindb.com https://www.bigchaindb.com Edit: The best solution that also has a theoretically unlimited throughput is this toy project: https://github.com/hoytech/quadrable https://github.com/hoytech/quadrable Conceptually, it computes a merkle tree over all data and regularly commits to Ethereum. Through this commitment the data may still change locally: But then at least would be provably tampered. So I guess for databases, the artibute we can really implement is "tamper-proof".

jandrese 5y ago
The big question is if someone gets on your DB server and wants to change a record how does the software prevent them from altering a record and then recomputing the remainder of the chain?

YogurtFiend 5y ago

I'm not sure that this is a _useful_ tool. Let's talk about the threat model or the attacks that this defends against. If a Client is malicious, they might try to manipulate the data in the database in an untoward way. In a "normal" database, this might cause data loss, if the database isn't being continuously backed up. But immudb does continuous backups (effectively, since it's immutable) so, if a malicious client has been detected, it's possible to restore an older version of the database. The real problem is how would you know that a client has tampered with your database? Well, because this database is "tamper-proof," duh! But the issue lies in the definition of tamper-proof. From my reading of the source code and documentation, the "proof that no tampering has occurred" is a proof that the current state of the database can be reached by applying some database operations to a previous state. As a result, a malicious client could simply ask the database to "delete everything and insert this new data," to make the database look like whatever it wanted. This is a valid way to transition the state of the database from its old state to the new state, and so shouldn't be rejected by the tamper detection mechanism. "Ah," but you say, "it would look super sus [as the kids say] to just delete the entire database. We'd know that something was up!" The problem with this solution is how are you going to automate "looking super sus?" You could enact a policy to flag any update that updates more than N records at a time, but that's not really a solution. The "right" solution is to trace the provenance of database updates. Rather than allowing arbitrary database updates, you want to allow your database to be changed only by updates that are sensible for your application. The _actual_ statement you want to prove is that "the current state of the database is a known past state of the database updated by operations that my application ought to have issued." Of course what are "operations that my application ought to have issued?" Well, it depends how deep you want to go with your threat model. A simple thing you could do is have a list of all the queries that your application issues, and check to make sure all operations come from that list. This still allows other attacks through, and you could go even more in depth if you wanted to. Importantly, immudb doesn't appear to contend with any of this. They claim that their database is "tamper-proof," when in reality you'd need a complicated external auditing system to make it meaningfully tamper-proof for your application. (Again, a threat model ought to include a precise definition of "tamper-proof," which would help clear up these issues.) It's also worth comparing this to https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency, which is an append-only database. Compared to immudb, the _exposed data model_ for certificate transparency logs is an append-only set, which means that it doesn't have any of these same problems. The problem with immudb is that the data model it exposes is more complicated, but it's built-in verification tools haven't been upgraded to match. (Also, for context, I've tried to obtain a copy of their white paper, but after an hour the email with the link to it never arrived.)

cabalamat 5y ago

Would it be possible to have something like this that works by writing to a PROM? That would make it immutable at the hardware level.

pharmakom 5y ago

Is it possible to delete data for compliance reasons? Not as a frequent operation, but say on a monthly batch?

jeroiraz 5y ago
logical deletion is in place, physical deletion is already on the roadmap
- kristjansson 5y ago
  That’s a lot of words to say no
- xvector 5y ago
  How can physical deletion work with the Merkle Tree mechanics? Do you just store the hash but not the underlying KV pair?
  - jeroiraz 5y ago
    the hash tree, transactions headers, payloads and index are all independently stored. So it will be possible to physically delete payloads and still be able to build all the proofs (but not being able to provide the actual values). Truncating transaction headers should be possible, but each physical deletion will be associated to a real lost of data, so proof generation may be affected on deleted data.

newtonapple 5y ago

Has anyone tried immudb in production? What are some of immudb's performance characteristics? It'd be nice to know how it performs under various conditions: query per sec, database / table sizes, SQL join performance etc. Also, what are the system requirements for immudb? What kind of machine would I need to run a medium to large website (say, 1TB of data, 5-25K qps, e.g. Wikipedia)? It mentioned in the documentation that it can use S3 as its storage? Are there performance implications if you do this?

spookylukey 5y ago

I find it very strange to have claims about the database being tamper-evident etc without a clear description of the threat/trust model, and how/for whom it works. For example, what data does the client need to store to be sure no tampering has occurred?

YogurtFiend 5y ago
I agree 100% with the need for the threat model. I think the current system is, at-best, a system with a tamper-evident audit log. Which, could still be interesting, but the authors' use of ill-defined terms makes it easy for people to think their software does something that it's not. What's supposed to happen is that the server will give the client a path in a https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree to the current state, the prove that the key-value pair is included in the Merkle tree. There are, however, some subtle issues which can arise if you're not careful. In particular, what happens if I set the key k to value v1, and then set the key k to value v2? If I subsequently ask for the value of k, I ought to see v2, and a proof that k is v1 shouldn't check out. However, in order for this to work, it's not sufficient for the server to prove that (k, v) is in the audit log, since that would allow for the server to maliciously roll-back the state. What you really want to prove is that v is not just _a_ value that k was set to, but _the most recent value_. It's unclear to me whether the code actually does this--there's no architecture guide which describes the cryptographic algorithms at play (or what the threat model is), and the code appears to be mostly devoid of comments. There is a reference to separate inclusion and consistency proofs, which might be this distinction. But it's really hard to tell from the digging that I've done.
- jeroiraz 5y ago
  This is a very interesting point. Rollbacks are protected by consistency proofs and with inclusion proofs it's possible to detect such situation but it may require scanning over the transactions. immudb does not only provide access by key or key prefixes but it's also possible to fetch a particular tx by its unique identifier or scan over them. For sure it's not the optimal solution, some ideas to cover this scenario were discussed but not yet fully defined.

infogulch 5y ago

I'm interested in how they intend to implement data pruning from the roadmap.

vyrotek 5y ago

Is this comparable or different than these "cryptographically verifiable" ledger DB services? https://docs.microsoft.com/en-us/azure/azure-sql/database/ledger-overview https://docs.microsoft.com/en-us/azure/azure-sql/database/le... https://aws.amazon.com/qldb/ https://aws.amazon.com/qldb/

renewiltord 5y ago

If I drop the trust requirement, what's the absolutely fastest blazing fast thing I can use that is network readable/writable and fault-tolerant?

throwaway81523 5y ago

This sort of reminds me of happstack, though the design and implementation are much different.

yencabulator 5y ago

Note: no `ALTER TABLE` in their SQL. Sounds like maintaining this over longer term with evolving data would be quite painful.

jeroiraz 5y ago
sure, schema changes are already in the roadmap

yencabulator 5y ago

They talk about performance a lot, but their benchmarks seem to always explicitly batch the inserts. I'm seeing <700 inserts per second in a simple loop. Quietly reporting batched inserts is a subtle lie.

jeroiraz 5y ago
each transaction may have several entries, in order to gain more performance, it's better to include several entries per transaction. But all the entries in the same transaction are atomically stored. submitting several transactions at the same time is also possible and convenient, processing is concurrently done until the last step which requires a strict serialization. benchmarking is a huge topic and we'd love to have contributions on this. It may refer to using embedded immudb, as a stand-alone server, kv, sql, etc....