4 ms·

RethinkDB is filled with developers who have conflicted thoughts on this. The difference between structured and unstructured database systems has always seemed

RethinkDB is filled with developers who have conflicted thoughts on this. The difference between structured and unstructured database systems has always seemed very analogous static vs. dynamic type systems to me. Most of us at RethinkDB like our type systems static, Haskell being one of the favored languages, and we've had many conversation about why our database doesn't reflect these same principles. I'll admit that to some degree RethinkDB is responding to what we see as the direction the market is moving. People seem to really like developing in dynamic languages using structureless databases. I think that a lot of times this gives people more ways to shoot themselves in the foot (and I sense you agree). But even if this type of development was purely a band of fools shooting off each other's feet we still might have to respond to it. That being said I don't think it is quite so foolish an endeavor. The thing about structured databases that sucks, and a place where the analogy to dynamic languages breaks down, is that your data integrity becomes entrenched. Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure. This can be a very cumbersome process made more cumbersome by the fact that a lot of structured databases make optimizations based on things like knowing the size of each row (very hard to change on the fly). So while RethinkDB doesn't dislike data integrity we do really dislike data stasis and that's what we wanted to avoid. We do have plans for the future of having a way to enforce arbitrary constraints over you database, we just aren't willing to do them in such a way that they're hard to change. Unfortunately doing them in that way is time consuming and we haven't gotten to it yet. jdoliner @ rethinkdb

astral303 14y ago
I have to agree with the last sentiment. In my experience, one never ends up having all of one's update/migration (to a new version) changes constrained only to schema alterations. This is because the static database structure can only capture so much metadata. One inevitably ends up facing putting migration logic into the application itself, or logic that says "if old data, treat differently." So now one ends up with a mixed data migration situation: both database-level and application-level. Thus, it's a quite valid, pragmatic tradeoff to make: handle all data migration/schema changes in the application, let the store be completely dynamic and have it all in one level.
saurik 14y ago
> I'll admit that to some degree RethinkDB is responding to what we see as the direction the market is moving. Is this really true? I mean, it is clear the majority of stuff you read about is from people who don't like types, but honestly as time has marched on development has been "democratized" to a large extent... yet, if I think about "people with both a lot of experience and a lot money who are willing to both pay for things to use and people to maintain those things" (what I would call "the market" for a database solution) I see lots and lots of rigid types. I mostly see "schemaless" from highly-popular (to the masses) database system where I don't really see much money being made or even moved (on database software). It could be, however, that I'm just totally misunderstanding how well some of these companies are doing. However, when I see a company evaluating things to plunk money down on, they seem to be 1) clustering support for PostgreSQL, 2) hosted management of MySQL, and 3) system administrators and licenses for Oracle. In contrast, people providing NoSQL "schemaless" systems seem mosty to be living in a world where the developers expect everything to be free; they sell support contracts, but I never hear of anyone actually buying them... do a lot of companies do this? It also always sounds like a magical accomplishment when you hear one being used a large company with a lot of money (such as the developers at UbiSoft giving a talk on how they are using CouchBase at re:Invent)... I haven't heard RDBMS systems have to justify "look, this company used us at scale and we worked fine" in years. Note: to be clear, I do hear of companies with "NoSQL" systems who have a lot of money, but they are all using Casaandra, HBase, or SOLR. SOLR let's you make define dynamic columns, but everything is still rigidly typed. HBase and Cassandra both have schemas with fixed columns; sure, Cassandra allowed you to largely ignore the schema, but if you've seen CQL3 they are actually nigh-into turning into an RDBMS: it is hilarious, and I think awesome. Even then, it isn't clear to me how much these companies pay money for their databases. > Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure. This is not actually true, as the structure can be metadata stored describing the data. When I add or delete fields in PostgreSQL, the operation is instantaneous, as is renaming existing fields. AFAIK, only MySQL sucks at this. The only time I need to do a table rewrite is if I change the datatype of an existing field, and I am pretty certain that you can also handle the majority of those cases without a rewrite. Even with the rewrite, you shouldn't have to block incoming queries. It isn't like not having a schema fixes this, as there is a schema after all... my code certainly has an assumption about how the data is stoted. It just means I either a) can't change the schema at all, b) can change the schema, but have to do so by doing the table rewrite manually, or c) can manually implement the metadata behavior (but then why not just add this to the database server). Given how the patterns one uses to solve these problems don't really rely on the data being stored, it simply makes sense to standardize them in the database server.

astral303 14y ago

I have to agree with the last sentiment. In my experience, one never ends up having all of one's update/migration (to a new version) changes constrained only to schema alterations. This is because the static database structure can only capture so much metadata. One inevitably ends up facing putting migration logic into the application itself, or logic that says "if old data, treat differently." So now one ends up with a mixed data migration situation: both database-level and application-level. Thus, it's a quite valid, pragmatic tradeoff to make: handle all data migration/schema changes in the application, let the store be completely dynamic and have it all in one level.

saurik 14y ago

> I'll admit that to some degree RethinkDB is responding to what we see as the direction the market is moving. Is this really true? I mean, it is clear the majority of stuff you read about is from people who don't like types, but honestly as time has marched on development has been "democratized" to a large extent... yet, if I think about "people with both a lot of experience and a lot money who are willing to both pay for things to use and people to maintain those things" (what I would call "the market" for a database solution) I see lots and lots of rigid types. I mostly see "schemaless" from highly-popular (to the masses) database system where I don't really see much money being made or even moved (on database software). It could be, however, that I'm just totally misunderstanding how well some of these companies are doing. However, when I see a company evaluating things to plunk money down on, they seem to be 1) clustering support for PostgreSQL, 2) hosted management of MySQL, and 3) system administrators and licenses for Oracle. In contrast, people providing NoSQL "schemaless" systems seem mosty to be living in a world where the developers expect everything to be free; they sell support contracts, but I never hear of anyone actually buying them... do a lot of companies do this? It also always sounds like a magical accomplishment when you hear one being used a large company with a lot of money (such as the developers at UbiSoft giving a talk on how they are using CouchBase at re:Invent)... I haven't heard RDBMS systems have to justify "look, this company used us at scale and we worked fine" in years. Note: to be clear, I do hear of companies with "NoSQL" systems who have a lot of money, but they are all using Casaandra, HBase, or SOLR. SOLR let's you make define dynamic columns, but everything is still rigidly typed. HBase and Cassandra both have schemas with fixed columns; sure, Cassandra allowed you to largely ignore the schema, but if you've seen CQL3 they are actually nigh-into turning into an RDBMS: it is hilarious, and I think awesome. Even then, it isn't clear to me how much these companies pay money for their databases. > Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure. This is not actually true, as the structure can be metadata stored describing the data. When I add or delete fields in PostgreSQL, the operation is instantaneous, as is renaming existing fields. AFAIK, only MySQL sucks at this. The only time I need to do a table rewrite is if I change the datatype of an existing field, and I am pretty certain that you can also handle the majority of those cases without a rewrite. Even with the rewrite, you shouldn't have to block incoming queries. It isn't like not having a schema fixes this, as there is a schema after all... my code certainly has an assumption about how the data is stoted. It just means I either a) can't change the schema at all, b) can change the schema, but have to do so by doing the table rewrite manually, or c) can manually implement the metadata behavior (but then why not just add this to the database server). Given how the patterns one uses to solve these problems don't really rely on the data being stored, it simply makes sense to standardize them in the database server.