5 ms·

Immutability is an interesting property, but reality is messy. Depending on the use case you envision for the database, data that needs to be scrubbed will acc

by mey 6y ago

Immutability is an interesting property, but reality is messy. Depending on the use case you envision for the database, data that needs to be scrubbed will accidentally be added to the system. This could be an important private key, or something legally dubious. In an append only/sync system there isn't really a good way to scrub that data. (Replay the entire history, filter bad data, then delete all remotes and resync from clean copy is not good). Again depends entirely on the use case of the database you envision, but if you wish to make it general purpose it is something to consider.

Spooky23 6y ago
It’s interesting because it reflects the reality of modern computing — for many use cases storage capacity is not a meaningful constraint. I ran pretty important systems as a DBA years ago that we’re significant but would never, ever require a meaningful constraint on growth. (Performance, different story) Sometimes looking at a problem like a database with a new set of constraints leads to interesting and novel things.
- dllthomas 6y ago
  Sure... but needing to purge sensitive data also reflects the reality of modern computing. unsure whether to laugh or cry
- foo_barrio 6y ago
  I've been working on personal and professional projects where growth isn't a truly restricting constraint but is still an actual constraint. What do you mean by "meaningful"? Once I get into the 10's of terabytes there's no storage system where copying a TiB of data is "instant" or ignorable. Professionaly I work on a financial reporting system where there are around 1.5 million accounts with trading activity. Things like moving a month of data around and checksumming still take on the order of 1 hour. If the size of that monthly dataset doubled, we'd be OKAY but it it grew 10x we'd have to start restructuring things both on the software side and the hardware side. For personal work I am working with searching video frame substrings regardless of codec, resolution etc within videos in a ~50 TiB video set. Having to re-encode hundrends of GiB of video at a time is some thing that takes a huge amount of time and nothing something I can just "throw into an S3 bucket". The growth of said working set is not truly constrained but is actually constrained. I can only afford so many 8-10 TiB HDDs at a time! They also fail and have to be mirrored/backedup with rsycn at the very least which adds to the cost for me.
  - Spooky23 6y ago
    I mean that it’s not a technical constraint or limit. Not that its trivial to do! If there is an ROI in keeping stuff forever, you can. Think about what you’re saying about your personal project. What you’re talking about wasn’t financially achievable for TV companies just 20 years ago. Hell, anything older than a few days would be stored offline in some tape silo, or in a shelf. (And 10-12 people to attend to it!) Now you as an individual could afford to do all sorts of stuff with that video data, albeit for some significant $.
dybber 6y ago
Yeah, immutability doesn’t seem desirable in a world with GDPR and the right to be forgotten, which requires you to support deletion.
- coddle-hark 6y ago
  I really don’t get why GDPR gets dragged up every time someone mentions immutable data stores; was everyone really building append-only systems before 2018?
  - breakfastduck 6y ago
    It's because GDPR made things a law, not just a best practice. It does need to be considered, and it's worth making sure people remember it.
    - fanf2 6y ago
      It has been a legal requirement in the UK that databases must be able to delete personal data since the data protection act 1984 - https://www.legislation.gov.uk/ukpga/1984/35/enacted https://www.legislation.gov.uk/ukpga/1984/35/enacted - in particular see the data protection principles in schedule 1: the 6th principle requires that data is retained no longer than necessary. There are provisions elsewhere in the act that can require a data holder to delete data. Most of GDPR is not really new. There was a significant amount of teaching about data protection when I was at university in 1994-1997 and it wasn’t even new then.
- nlitened 6y ago
  Database immutability does not mean it's impossible to delete content. It means you separate two use cases: retracting information (I remove an item from my shopping cart because I want another item) and purging data (I delete my account and all my shopping history because I don't want the shop to store my data anymore). You can still purge data easily in Datomic, for example.
sixdimensional 6y ago
Temporal databases - where SQL:2011 left off. I think the concepts of bitemporality and tritemporality are what the post author is looking for. Something the original post reminds me of a little is Crux [1]. [1] https://opencrux.com/main/index.html https://opencrux.com/main/index.html
- coxmichael 6y ago
  This article[1] from 2015 has stuck with me for years, talking about the same approach. The interesting thing about transactional/bitemporal databases is really the effect on how we consume them with our applications. The schema can be separated from the transaction data, schema migrations become easy, tables become cached materialized views (with the cache updated concurrently at write time), reduced need for application-level cache, no cache cold start, and pub/sub becomes the norm. [1] https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou... (Nice to see it's still relevant – the article is linked at the very top of the crux about page: https://opencrux.com/about/what-is-crux.html https://opencrux.com/about/what-is-crux.html)
  - sixdimensional 6y ago
    I’m the same regarding that article. I always thought of that article as a kind of “deconstruction” or “decoupling” and “rearrangement” of traditional DBMS architecture. The end result you speak of is a more flexible data platform, which can handle more use cases, including applying decoupled parts to different things. A good example, in my mind, is Apache Spark - which is like taking a compute and SQL query engine, making it distributed and decoupling it from the underlying storage. Now, suddenly SQL or code can be used on anything you can fit into a data frame, even streams - wow! Pulling out the transaction log - and wow, suddenly we have durable, potentially immutable queues we can use for even long term storage. I was so excited by the line of thinking in that article. :)
black_knight 6y ago
The problem of deleting stuff from a history of changes is challenging. But I think Darcs, the distributed version control system written in Haskell, does this rather neatly. Darcs is different from git in that a "commit" (called a "patch" in Darcs lingo) does not depend upon the entire history of what has come before. Rather it has specified dependencies, which Darcs can infer itself – or you can specify if you want more semantic content in the patch structure. A repository is a collection of patches, which are merged using something called patch algebra. So, if you want to delete a patch, you just issue the command "darcs obliterate" and it will remove the patch, and anything which depends on it. Then working directory is reconstructed by merging together any loose ends using patch algebra. With careful administration of dependencies, this approach makes deleting specific things very slick.
throwaway_pdp09 6y ago
> Git repositories can only grow in size, and that is not only an acceptable condition, but also one of the reasons to use it. That is so backwards I can't take the guy seriously.
roenxi 6y ago
> Immutability is an interesting property, but reality is messy. Words have fuzzy boundaries - immutable obviously doesn't mean actually immutable. It means immutable like how git is immutable - you're not meant to change history. It'll fight like a tiger to stop you easily changing history. It'll give you two different ways to get what you want while preserving history. But if you want to change history you can. SQL is said to be declarative but if you get hit dealing with the wrong sort of workload in SQL then the implementation details can start to matter extremely quickly. In practice, the author probably only means 'much more immutable' rather than 'illegal to use with European data' level immutability.
- juped 6y ago
  Git's primary advantage over other VCS is how it makes it trivially easy to change history, though.
  - andriosr 6y ago
    True. And what makes it easy is it's immutability. When you change history in Git, you rewrite it (rebase). But re-writing is only possible if you know every bit of the journey up to now. When you have actual updates, history is lost.
  - afiori 6y ago
    the way git changes history is a stretch on the word change, rebasing creates a parallel history and then promotes it to be the true one. the reflog still remembers the previous history.
zupa-hu 6y ago
Let me disagree. I'm working on a transactional OS [1] with an immutable filesystem, it's primarily a website builder. Every request creates a new snapshot of your website. It's append-only for the duration of the transaction. That's the only way to provide robust rollback and also provides version snapshots for free - but it doesn't mean snapshot's can't later be removed. My solution was to provide an undo-chain of a fixed given length. Versions older than that simply "fall off the chain" and will be picked up by GC eventually. You can also make snapshots long-lived snapshots if you want. (Of course removing a snapshot is an append-only operation :P but GC will go after non-referenced snapshots eventually.) [1] https://boomla.com/ https://boomla.com/

by mey 6y ago

Spooky23 6y ago

It’s interesting because it reflects the reality of modern computing — for many use cases storage capacity is not a meaningful constraint. I ran pretty important systems as a DBA years ago that we’re significant but would never, ever require a meaningful constraint on growth. (Performance, different story) Sometimes looking at a problem like a database with a new set of constraints leads to interesting and novel things.

dllthomas 6y ago
Sure... but needing to purge sensitive data also reflects the reality of modern computing. unsure whether to laugh or cry
foo_barrio 6y ago
I've been working on personal and professional projects where growth isn't a truly restricting constraint but is still an actual constraint. What do you mean by "meaningful"? Once I get into the 10's of terabytes there's no storage system where copying a TiB of data is "instant" or ignorable. Professionaly I work on a financial reporting system where there are around 1.5 million accounts with trading activity. Things like moving a month of data around and checksumming still take on the order of 1 hour. If the size of that monthly dataset doubled, we'd be OKAY but it it grew 10x we'd have to start restructuring things both on the software side and the hardware side. For personal work I am working with searching video frame substrings regardless of codec, resolution etc within videos in a ~50 TiB video set. Having to re-encode hundrends of GiB of video at a time is some thing that takes a huge amount of time and nothing something I can just "throw into an S3 bucket". The growth of said working set is not truly constrained but is actually constrained. I can only afford so many 8-10 TiB HDDs at a time! They also fail and have to be mirrored/backedup with rsycn at the very least which adds to the cost for me.
- Spooky23 6y ago
  I mean that it’s not a technical constraint or limit. Not that its trivial to do! If there is an ROI in keeping stuff forever, you can. Think about what you’re saying about your personal project. What you’re talking about wasn’t financially achievable for TV companies just 20 years ago. Hell, anything older than a few days would be stored offline in some tape silo, or in a shelf. (And 10-12 people to attend to it!) Now you as an individual could afford to do all sorts of stuff with that video data, albeit for some significant $.

dybber 6y ago

Yeah, immutability doesn’t seem desirable in a world with GDPR and the right to be forgotten, which requires you to support deletion.

coddle-hark 6y ago
I really don’t get why GDPR gets dragged up every time someone mentions immutable data stores; was everyone really building append-only systems before 2018?
- breakfastduck 6y ago
  It's because GDPR made things a law, not just a best practice. It does need to be considered, and it's worth making sure people remember it.
  - fanf2 6y ago
    It has been a legal requirement in the UK that databases must be able to delete personal data since the data protection act 1984 - https://www.legislation.gov.uk/ukpga/1984/35/enacted https://www.legislation.gov.uk/ukpga/1984/35/enacted - in particular see the data protection principles in schedule 1: the 6th principle requires that data is retained no longer than necessary. There are provisions elsewhere in the act that can require a data holder to delete data. Most of GDPR is not really new. There was a significant amount of teaching about data protection when I was at university in 1994-1997 and it wasn’t even new then.
nlitened 6y ago
Database immutability does not mean it's impossible to delete content. It means you separate two use cases: retracting information (I remove an item from my shopping cart because I want another item) and purging data (I delete my account and all my shopping history because I don't want the shop to store my data anymore). You can still purge data easily in Datomic, for example.

sixdimensional 6y ago

Temporal databases - where SQL:2011 left off. I think the concepts of bitemporality and tritemporality are what the post author is looking for. Something the original post reminds me of a little is Crux [1]. [1] https://opencrux.com/main/index.html https://opencrux.com/main/index.html

coxmichael 6y ago
This article[1] from 2015 has stuck with me for years, talking about the same approach. The interesting thing about transactional/bitemporal databases is really the effect on how we consume them with our applications. The schema can be separated from the transaction data, schema migrations become easy, tables become cached materialized views (with the cache updated concurrently at write time), reduced need for application-level cache, no cache cold start, and pub/sub becomes the norm. [1] https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou... (Nice to see it's still relevant – the article is linked at the very top of the crux about page: https://opencrux.com/about/what-is-crux.html https://opencrux.com/about/what-is-crux.html)
- sixdimensional 6y ago
  I’m the same regarding that article. I always thought of that article as a kind of “deconstruction” or “decoupling” and “rearrangement” of traditional DBMS architecture. The end result you speak of is a more flexible data platform, which can handle more use cases, including applying decoupled parts to different things. A good example, in my mind, is Apache Spark - which is like taking a compute and SQL query engine, making it distributed and decoupling it from the underlying storage. Now, suddenly SQL or code can be used on anything you can fit into a data frame, even streams - wow! Pulling out the transaction log - and wow, suddenly we have durable, potentially immutable queues we can use for even long term storage. I was so excited by the line of thinking in that article. :)

black_knight 6y ago

The problem of deleting stuff from a history of changes is challenging. But I think Darcs, the distributed version control system written in Haskell, does this rather neatly. Darcs is different from git in that a "commit" (called a "patch" in Darcs lingo) does not depend upon the entire history of what has come before. Rather it has specified dependencies, which Darcs can infer itself – or you can specify if you want more semantic content in the patch structure. A repository is a collection of patches, which are merged using something called patch algebra. So, if you want to delete a patch, you just issue the command "darcs obliterate" and it will remove the patch, and anything which depends on it. Then working directory is reconstructed by merging together any loose ends using patch algebra. With careful administration of dependencies, this approach makes deleting specific things very slick.

throwaway_pdp09 6y ago

> Git repositories can only grow in size, and that is not only an acceptable condition, but also one of the reasons to use it. That is so backwards I can't take the guy seriously.

roenxi 6y ago

> Immutability is an interesting property, but reality is messy. Words have fuzzy boundaries - immutable obviously doesn't mean actually immutable. It means immutable like how git is immutable - you're not meant to change history. It'll fight like a tiger to stop you easily changing history. It'll give you two different ways to get what you want while preserving history. But if you want to change history you can. SQL is said to be declarative but if you get hit dealing with the wrong sort of workload in SQL then the implementation details can start to matter extremely quickly. In practice, the author probably only means 'much more immutable' rather than 'illegal to use with European data' level immutability.

juped 6y ago
Git's primary advantage over other VCS is how it makes it trivially easy to change history, though.
- andriosr 6y ago
  True. And what makes it easy is it's immutability. When you change history in Git, you rewrite it (rebase). But re-writing is only possible if you know every bit of the journey up to now. When you have actual updates, history is lost.
- afiori 6y ago
  the way git changes history is a stretch on the word change, rebasing creates a parallel history and then promotes it to be the true one. the reflog still remembers the previous history.

zupa-hu 6y ago

Let me disagree. I'm working on a transactional OS [1] with an immutable filesystem, it's primarily a website builder. Every request creates a new snapshot of your website. It's append-only for the duration of the transaction. That's the only way to provide robust rollback and also provides version snapshots for free - but it doesn't mean snapshot's can't later be removed. My solution was to provide an undo-chain of a fixed given length. Versions older than that simply "fall off the chain" and will be picked up by GC eventually. You can also make snapshots long-lived snapshots if you want. (Of course removing a snapshot is an append-only operation :P but GC will go after non-referenced snapshots eventually.) [1] https://boomla.com/ https://boomla.com/