8 ms·

Schema-less is usually a lie

47 points by mrkurt 13y ago

taeric 13y ago
Isn't this akin to the idea that many dynamic programs actually have fairly static types?
- tome 13y ago
  That sounds like a good analogy to me.
- 6ren 13y ago
  Stretching it: natural language was grammatical before grammar was invented. Natural laws existed before they were discovered. But it can be easier to change grammar/law/types/schema when not explicit.
  - taeric 13y ago
    I thought that was the point, though. Just because the schema is not explicitly set in the database, does not mean there is not a "de facto" schema. Sure, some changes will be easier this way, but just changing things is almost assuredly going to break something.
  - joe_the_user 13y ago
    "...it can be easier to change grammar/law/types/schema when not explicit." Now that's quite interesting statement. I'd say it is true for humans but false for computers and it's paradoxical to think about why this is. If computers were an intelligent as humans, you wouldn't have to worry about giving your program any structure because the computer could change that structure later. But sadly, spaghetti code isn't what it's cracked up to be. Similarly, an application where don't bother thinking about your schema beforehand isn't going to be application which you can change easily later.
    - taeric 13y ago
      It is not just spaghetti code, though. Right? It is code that does not constantly check that input fits an expected pattern. Which is most code. I'm reminded of this talk by Sussman.[1] Essentially, in software we are (understandably) adamant about not modifying the input of our application for "pure" functions. Nature goes right the opposite, often attempting to make the input work at all costs. So, whereas a schema based application that models a user with an address is unable to cope with being given two addresses. A schema-less approach where it just stores blobs simply stores what it was given. If code that reads this is unable to make sense of multiple addresses, it will raise an error. Not necessarily unlike a user being told to send a package to an address, but given a list of addresses. [1] http://www.infoq.com/presentations/We-Really-Dont-Know-How-To-Compute http://www.infoq.com/presentations/We-Really-Dont-Know-How-T...
Jare 13y ago
> Indexes on collections in MongoDB are very much “schema” I think that's all you need to read in this post. It's... rather thin IMHO.
jbellis 13y ago
Cassandra is not schema-less. I covered this in my recent article on Cassandra Myths: http://www.infoq.com/articles/cassandra-mythology http://www.infoq.com/articles/cassandra-mythology
leeoniya 13y ago
from https://news.ycombinator.com/item?id=3838131 https://news.ycombinator.com/item?id=3838131 '...but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"'
- marcosdumay 13y ago
  And missing most of the centralized documentation people use everywhere. But if you only have one application, you don't need centralized documentation.
  - tracker1 13y ago
    You can create an API for data access that validates a given document schema... you don't have to operate in the open.
just2n 13y ago
Isn't a schemaless database just a filesystem?
- camus 13y ago
  it'w worse , it is basically a bin where you throw paper. You can still search for a specific one ,but as you throw more paper in the trashcan , it gets more complicated to find the paper you want. but hey ,it is easier and faster than to organize them on a rack they say ...
  - cmccabe 13y ago
    So it's a "circular file" system.
  - tracker1 13y ago
    It really depends on the non-sql solution... Most of which do have indexed keys for retrieving data at the very least (key-value storage), and use Map-Reduce in order to perform basic queries, or you setup your own indexed records. MongoDB specifically allows for additional indexes to be used as part of general queries, or the aggregation methods available. MongoDB even has some nice built in features for querying against geolocation data. They do have some limitations, but in general it works quite well. Your analogy comparing non-sql to a trashcan is a bad comparison... a better comparison would be to a set of file cabinets.
- tracker1 13y ago
  Some honestly aren't much more... they all have at least a "key" or id field you can lookup a record by, and most have a map/reduce implementation. Some require you to do your own indexed records and others (like MongoDB and RethinkDB) have the ability to add additional indexes (similar to SQL). I find that MongoDB tends to perform at least as well as an SQL based solution in general use. Some use cases are better suited to SQL (anything with multiple records in a single transaction as a hard requirement for example). It really depends on your needs.
programminggeek 13y ago
100% agree with this post and it's important to understand where/when schema in the code makes more sense than schema in the db. If the DB is determining the schema and your code just mirrors it, you will inevitably do things like triggers, stored procedures, etc. that essentially put application code inside the DB. This makes testing and maintenance of such things all but impossible, even if it does make the DB queries fast. At the same time, your application still needs to mirror the schema of the DB or things will break. "Schemaless" (query schemas) isn't better per se, but having the mindset of putting the schema in the code means you can write tests to ensure the schema over time. Depending on your perspective, one might be better than the other, but as they say "knowing is half the battle"
- joe_the_user 13y ago
  If the DB is determining the schema and your code just mirrors it, you will inevitably do things like triggers, stored procedures, etc. that essentially put application code inside the DB. This makes testing and maintenance of such things all but impossible, even if it does make the DB queries fast. I don't believe this is necessarily true. I think it depends on the structure of the enterprise. The traditional relational database seems to center on what could be called the "traditional large enterprise". This large enterprise tends to have many projects sharing the same static data and here a single store for that data makes sense and that store can and should have a standard interface, which can be embodied as stored procedures if necessary and would have to be well-defined enough that individual applications can deal with it. That's the "traditional large enterprise". Today, however, we have large companies where single application has become the essence of the company (Google, Facebook, etc)and the one-datastore, multiple-applications model no longer makes sense and it does make sense to move more data manipulation the application level. Still, when making pronouncements about what works best, I think context is important.
  - dragonwriter 13y ago
    > Today, however, we have large companies where single application has become the essence of the company (Google, Facebook, etc)and the one-datastore, multiple-applications model no longer makes sense Which "one application" is the essence of Google?
kailuowang 13y ago
To me, schema-less simply means that you don't need to maintain the data structure in a separate schema definition. It doesn't mean that your data doesn't have a structure - the structure is in your data, query and indexes, it's just easier to change without that extra "schema" definition.
- rhizome 13y ago
  Here, step aside while I demonstrate my any-endian arbitrary wordlength CPU.
- wvenable 13y ago
  It's not easier to change. When you have a schema-less database you don't change the schema, you add a new one for new data. Any data with an old schema continues to have it. So at any time you have up to n schemas simultaneously in your database if you changed it n times.
Encave 13y ago
Here are some slides from a good talk by Martin Fowler about schemalessness and how there is always an 'implicit schema' http://martinfowler.com/articles/schemaless/ http://martinfowler.com/articles/schemaless/
gregwebs 13y ago
I disagree with the idea that MongoDB has struck the right balance. I have seen production issues from manual editing of data which resulted in a floating point instead of an Integer. I really wish MongoDB had support for a strong schema in the DB besides indexes (in addition to existing support for schema-less). I only actually want schema-less at most 20% of the time, but I am stuck with it for everything outside of indexes.
- spamizbad 13y ago
  Completely agree. I'd love it if you could provide collections a baseline set of document validation rules- rules that only apply to fields you explicitly mention. I guess what I want is a schema-ish database, where I can say "these 5 fields are required and must conform to these rules. Anything else is fair game."
  - jasonmccay 13y ago
    I really like the term "schema-ish" and I think that most people probably want something like this. They will inevitably still have a defined structure for their data, but they need that extra bit of flexibility to keep the overall data structure simplistic and sensible.
  - tokenizerrr 13y ago
    I recall reading about how MongoDB saves all the key names with each record, since there is no schema. This results in quite a lot of disk space going to waste. It seems like this could be a solution to that problem as well, I like it.
    - jasonmccay 13y ago
      The guys at Tokutek are solving that problem with MongoDB. Pretty interesting stuff ... you should check it out.
  - vyrotek 13y ago
    I've been looking some something similar to what you've described. I recently discovered PostgreSQL has a 'hstore' feature which seems to support this. You seem to be able to make a normal table and then specify one column to act like a key/value dictionary. These values can even be included in queries. http://www.postgresql.org/docs/9.0/static/hstore.html http://www.postgresql.org/docs/9.0/static/hstore.html
thelarry 13y ago
i feel like a lot of people seem to jump on to "schemaless" dbs because they feel the can just dump whatever they want into the database. they also run into a problem later on when they realize that they need to care about data integrity and types.
- tracker1 13y ago
  Or you have a data structure that would require a lot of tables for what equates to a single record of interest. In my last job, we were presenting search, and display of classified listings for cars. The normalized data required, iirc, 28 join operations to get most of the data for a single record (for display), about 12 iirc for the search support (not including geo/location based searches), and a second lookup for related data. This could be replaced by a single query in a non-sql database. There is a real cost to these kinds of structures in SQL... There are a LOT of use cases where a single record structured as a complete object is much better than having to break up said structure into dozens of fields.
gsabo 13y ago
I believe Kyoto Cabinet is an example of a truly Schema-less DB: http://fallabs.com/kyotocabinet/ http://fallabs.com/kyotocabinet/

47 points by mrkurt 13y ago

taeric 13y ago

Isn't this akin to the idea that many dynamic programs actually have fairly static types?

tome 13y ago
That sounds like a good analogy to me.
6ren 13y ago
Stretching it: natural language was grammatical before grammar was invented. Natural laws existed before they were discovered. But it can be easier to change grammar/law/types/schema when not explicit.
- taeric 13y ago
  I thought that was the point, though. Just because the schema is not explicitly set in the database, does not mean there is not a "de facto" schema. Sure, some changes will be easier this way, but just changing things is almost assuredly going to break something.
- joe_the_user 13y ago
  "...it can be easier to change grammar/law/types/schema when not explicit." Now that's quite interesting statement. I'd say it is true for humans but false for computers and it's paradoxical to think about why this is. If computers were an intelligent as humans, you wouldn't have to worry about giving your program any structure because the computer could change that structure later. But sadly, spaghetti code isn't what it's cracked up to be. Similarly, an application where don't bother thinking about your schema beforehand isn't going to be application which you can change easily later.
  - taeric 13y ago
    It is not just spaghetti code, though. Right? It is code that does not constantly check that input fits an expected pattern. Which is most code. I'm reminded of this talk by Sussman.[1] Essentially, in software we are (understandably) adamant about not modifying the input of our application for "pure" functions. Nature goes right the opposite, often attempting to make the input work at all costs. So, whereas a schema based application that models a user with an address is unable to cope with being given two addresses. A schema-less approach where it just stores blobs simply stores what it was given. If code that reads this is unable to make sense of multiple addresses, it will raise an error. Not necessarily unlike a user being told to send a package to an address, but given a list of addresses. [1] http://www.infoq.com/presentations/We-Really-Dont-Know-How-To-Compute http://www.infoq.com/presentations/We-Really-Dont-Know-How-T...

Jare 13y ago

> Indexes on collections in MongoDB are very much “schema” I think that's all you need to read in this post. It's... rather thin IMHO.

jbellis 13y ago

Cassandra is not schema-less. I covered this in my recent article on Cassandra Myths: http://www.infoq.com/articles/cassandra-mythology http://www.infoq.com/articles/cassandra-mythology

leeoniya 13y ago

from https://news.ycombinator.com/item?id=3838131 https://news.ycombinator.com/item?id=3838131 '...but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"'

marcosdumay 13y ago
And missing most of the centralized documentation people use everywhere. But if you only have one application, you don't need centralized documentation.
- tracker1 13y ago
  You can create an API for data access that validates a given document schema... you don't have to operate in the open.

just2n 13y ago

Isn't a schemaless database just a filesystem?

camus 13y ago
it'w worse , it is basically a bin where you throw paper. You can still search for a specific one ,but as you throw more paper in the trashcan , it gets more complicated to find the paper you want. but hey ,it is easier and faster than to organize them on a rack they say ...
- cmccabe 13y ago
  So it's a "circular file" system.
- tracker1 13y ago
  It really depends on the non-sql solution... Most of which do have indexed keys for retrieving data at the very least (key-value storage), and use Map-Reduce in order to perform basic queries, or you setup your own indexed records. MongoDB specifically allows for additional indexes to be used as part of general queries, or the aggregation methods available. MongoDB even has some nice built in features for querying against geolocation data. They do have some limitations, but in general it works quite well. Your analogy comparing non-sql to a trashcan is a bad comparison... a better comparison would be to a set of file cabinets.
tracker1 13y ago
Some honestly aren't much more... they all have at least a "key" or id field you can lookup a record by, and most have a map/reduce implementation. Some require you to do your own indexed records and others (like MongoDB and RethinkDB) have the ability to add additional indexes (similar to SQL). I find that MongoDB tends to perform at least as well as an SQL based solution in general use. Some use cases are better suited to SQL (anything with multiple records in a single transaction as a hard requirement for example). It really depends on your needs.

programminggeek 13y ago

100% agree with this post and it's important to understand where/when schema in the code makes more sense than schema in the db. If the DB is determining the schema and your code just mirrors it, you will inevitably do things like triggers, stored procedures, etc. that essentially put application code inside the DB. This makes testing and maintenance of such things all but impossible, even if it does make the DB queries fast. At the same time, your application still needs to mirror the schema of the DB or things will break. "Schemaless" (query schemas) isn't better per se, but having the mindset of putting the schema in the code means you can write tests to ensure the schema over time. Depending on your perspective, one might be better than the other, but as they say "knowing is half the battle"

joe_the_user 13y ago
If the DB is determining the schema and your code just mirrors it, you will inevitably do things like triggers, stored procedures, etc. that essentially put application code inside the DB. This makes testing and maintenance of such things all but impossible, even if it does make the DB queries fast. I don't believe this is necessarily true. I think it depends on the structure of the enterprise. The traditional relational database seems to center on what could be called the "traditional large enterprise". This large enterprise tends to have many projects sharing the same static data and here a single store for that data makes sense and that store can and should have a standard interface, which can be embodied as stored procedures if necessary and would have to be well-defined enough that individual applications can deal with it. That's the "traditional large enterprise". Today, however, we have large companies where single application has become the essence of the company (Google, Facebook, etc)and the one-datastore, multiple-applications model no longer makes sense and it does make sense to move more data manipulation the application level. Still, when making pronouncements about what works best, I think context is important.
- dragonwriter 13y ago
  > Today, however, we have large companies where single application has become the essence of the company (Google, Facebook, etc)and the one-datastore, multiple-applications model no longer makes sense Which "one application" is the essence of Google?

kailuowang 13y ago

To me, schema-less simply means that you don't need to maintain the data structure in a separate schema definition. It doesn't mean that your data doesn't have a structure - the structure is in your data, query and indexes, it's just easier to change without that extra "schema" definition.

rhizome 13y ago
Here, step aside while I demonstrate my any-endian arbitrary wordlength CPU.
wvenable 13y ago
It's not easier to change. When you have a schema-less database you don't change the schema, you add a new one for new data. Any data with an old schema continues to have it. So at any time you have up to n schemas simultaneously in your database if you changed it n times.

Encave 13y ago

Here are some slides from a good talk by Martin Fowler about schemalessness and how there is always an 'implicit schema' http://martinfowler.com/articles/schemaless/ http://martinfowler.com/articles/schemaless/

gregwebs 13y ago

I disagree with the idea that MongoDB has struck the right balance. I have seen production issues from manual editing of data which resulted in a floating point instead of an Integer. I really wish MongoDB had support for a strong schema in the DB besides indexes (in addition to existing support for schema-less). I only actually want schema-less at most 20% of the time, but I am stuck with it for everything outside of indexes.

spamizbad 13y ago
Completely agree. I'd love it if you could provide collections a baseline set of document validation rules- rules that only apply to fields you explicitly mention. I guess what I want is a schema-ish database, where I can say "these 5 fields are required and must conform to these rules. Anything else is fair game."
- jasonmccay 13y ago
  I really like the term "schema-ish" and I think that most people probably want something like this. They will inevitably still have a defined structure for their data, but they need that extra bit of flexibility to keep the overall data structure simplistic and sensible.
- tokenizerrr 13y ago
  I recall reading about how MongoDB saves all the key names with each record, since there is no schema. This results in quite a lot of disk space going to waste. It seems like this could be a solution to that problem as well, I like it.
  - jasonmccay 13y ago
    The guys at Tokutek are solving that problem with MongoDB. Pretty interesting stuff ... you should check it out.
- vyrotek 13y ago
  I've been looking some something similar to what you've described. I recently discovered PostgreSQL has a 'hstore' feature which seems to support this. You seem to be able to make a normal table and then specify one column to act like a key/value dictionary. These values can even be included in queries. http://www.postgresql.org/docs/9.0/static/hstore.html http://www.postgresql.org/docs/9.0/static/hstore.html

thelarry 13y ago

i feel like a lot of people seem to jump on to "schemaless" dbs because they feel the can just dump whatever they want into the database. they also run into a problem later on when they realize that they need to care about data integrity and types.

tracker1 13y ago
Or you have a data structure that would require a lot of tables for what equates to a single record of interest. In my last job, we were presenting search, and display of classified listings for cars. The normalized data required, iirc, 28 join operations to get most of the data for a single record (for display), about 12 iirc for the search support (not including geo/location based searches), and a second lookup for related data. This could be replaced by a single query in a non-sql database. There is a real cost to these kinds of structures in SQL... There are a LOT of use cases where a single record structured as a complete object is much better than having to break up said structure into dozens of fields.

gsabo 13y ago

I believe Kyoto Cabinet is an example of a truly Schema-less DB: http://fallabs.com/kyotocabinet/ http://fallabs.com/kyotocabinet/