4 ms·

Hi, I think that you are completely right. Schema evolution and easy partition/sharding are broken on current open-source RDBMS systems. Relational data has s

• gfodor 16y ago
  The solution is pretty clear, simply make it easy to do what the author suggests in their article, in a safe and repeatable way. You define your new schema, and you define the tranformation rules to run on the old data. In turn, the database applies these lazily. There'd be some formality necessary to prove certain operations could be applied lazily to provide a consistent view of the data. People are doing this by hand now; it should be part of the system. One can imagine a mathematical model that involves deferred relational operators. The constraints that would be imposed would probably be that certain operations would not be permissable until others complete. For example, it might be necessary, when running one of these lazy migrations, to "flush" the remaining untransformed rows to the end before you are able to apply another one. The upside is your migration scripts run instantly, becuase they have no side effects until the next transaction. The CAP theorem I think often gets rolled up into these "NoSQL" discussions erroneously. One can still model data relationally but materialize it in a way that reduces ACID compliance to survive in the face of network partitions. Arguably this is a red herring though. In practice, the "network partition" problem to me seems to be only really worth worrying about if you are Google or Amazon and are running cross-data center data storage. For the majority of the world this is not something that needs to be worried about. (See: VoltDB's choice on the matter.) Edit: By the way, I'm not saying any of this is easy. It's hard, really hard. (I couldn't do it.) But the point here is that it's easy to get caught up and think these 'new' document databases are solving these problems. They're not. They're putting you back down to the assembly code level and forcing you to do these things yourself. The whole point of the RDBMS is to provide an abstraction layer. They're removing this abstraction layer and heralding it as a step forward. It is, in a bizarre sense similar to how it's a "step forward" for you to be able to access the registers on your CPU, but it's surely a step backwards in many ways as well.
  • jchrisa 16y ago
    The compelling thing about CouchDB is the ability to take a copy of the database offline, make changes, and then sync back up with other copies of the database later (or never). If this isn't a network partition I don't know what is. Of course thinking like this is totally unlike what most relational users are accustomed to, but this use case is only becoming more prominent as applications spread to diverse and occasionally connected devices. I have a hard time imagining practical offline APIs without the full MVCC document model. Once you see this use case's value it's hard not to want to write all your apps so that they can be replicated offline for disconnected use.
    • mjw 16y ago
      One relevant point here is that it's relatively easy to merge two branches of changes for a simple (key/value) data model, but harder to merge changes to relational data where there are additional constraints (like foreign keys) which must be preserved. That said, just because it's easy to merge key/value data, doesn't mean that the results of the merge are necessarily going to be correct. The constraints still exist, they're just not explicit in the data model, meaning that when your network partition gets joined up again and it magically merges all those changes at the key-value level, you may be ending up with data that's broken in subtle, difficult-to-track-down ways because other implicit constraints weren't respected by the merge. So, merging changes in a way that respects constraints which apply to that data, is a hard problem in general. Key/value stores, with their less-structured data model, sweep that problem under the carpet somewhat in order to make things easier in distributed settings. Relational databases suffer from trying to address the problem more fully.
      • gfodor 16y ago
        This is a really important point. There seems to be this joy people are having from shedding their schema since they can just get stuff done and not have it get in the way. This should be just as much scary as liberating. You're basically flying on manual now, and there's nothing protecting your data integrity anymore.