6 ms·

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it. Using K/V stores will only help you wri

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it. Using K/V stores will only help you write data. But the whole impedance nightmare is right there, waiting for you to try and make some sense of the data. Especially if you want to do relations. And you are dead wrong about relational DB's. It is either due to habit or because of being a better fit that users demand representations of data that are best served from a relational source. So you better plan your data models wisely, because you WILL pump this data into a relational source, sooner or later. It would be wise do keep a schema around all the time. In the end it is merely a question of data normalization and the use case at hand.

latch 15y ago
> The Object/Relational impedance mismatch stays right there where you leave it. Somewhat umbrella...but I do think that document-based storage does make a difference. It certainly doesn't erase it and, as you say, it varies based on the use-case at hand, but I think most people would consider the development experience to have less friction (and that's certainly been the overwhelming anecdotal evidence I've heard (and can give)).
- prodigal_erik 15y ago
  I have to wonder whether that's because many devs started using schemaless databases for the first time over the last couple of years, and haven't yet really experienced the nightmare of data which was scribbled on by various forgotten buggy versions of the apps and never rigorously migrated (because they're self-selected to regard that as unimportant). I once worked on a Lotus Notes-based system with documents eventually reaching such nonsensical states that the dev team couldn't even say what app behavior would be appropriate, much less what the latest version of all our code would happen to do.
  - JanezStupar 15y ago
    Exactly, I have done most of my work on legacy (think 10-15 years worth of data, millions of documents that were spawned in countless application versions, without any schema tracking whatsoever - everything is implicit in the document itself) Lotus Domino applications. Since there is no explicit database schema in these types of databases, what you didn't do at write, you have to do at read. And usually you want to use the latest view or representation of data, what do you do with data that wasn't there ten versions ago. What do you do about data fields of wrong format? What do you do with data that is "orphaned" and cannot be referenced to other data. Yet it still is data and still is important. Don't get me wrong, I Love NoSQL and I like to use it. I just have enough experience with it to know that it is definitely not a silver bullet. By the way: If someone is looking to hire a guy who is not afraid of tackling this kind of issue, contact me. I have plenty of experience with coercion of non relational data into a form suitable for analysis.
DenisM 15y ago
>In the end it is merely a question of data normalization Can you elaborate? I understand any JSON can be shredded into a third normal form (minus the ordering problem, but let's leave that aside for now), is this what you refer to?
- JanezStupar 15y ago
  Yes this is exactly what I mean. Lets start from[1]. The kosher way of designing document oriented database is to simply embed everything within a document. This is great for operations such as full text search, retrieving whole objects, etc. We call this denormalized form. On the other hand, sometimes we want to analyze our data, e.g. we want to extract subsets of data and view them in isolation. With completely denormalized form this is expensive, since we either need to manually touch each and every document and extract data or we need to maintain indices that help us out. Both are extremely resource intensive. The third option is that we chop our objects into smaller objects and then link them together. But this will mean that retrieving a whole document will take longer (multiple database requests) it also adds an overhead of eliminating duplicates (two objects may appear identical, but really are not), etc... Denormalization gives you horizontal scalability, but takes away ad-hoc querying. It also wastes storage (document size is minor issue, Indices however will kill you [2]). Normalized data will take away horizontal scalability, give you ad-hoc querying, and save storage space. In the end for any kind of nontrivial system, you will eventually reach a point, where you will need to maintain two storages - a normalized and non-normalized form. The only difference is what your primary problem is and this sets whether you start out from Normalized or Denormalized storage, this will be your primary storage and source. The other kind will be an offline slave that will offer secondary functionality. E.g: If you start out from relational data and you want to build a FTS, you WILL have to denormalize data. On the other hand, if you start from Object/Document store and want to offer ad-hoc analytics, you WILL have to normalize your data. Its good to keep it in the back of your head. [1]: http://www.mongodb.org/display/DOCS/Schema+Design http://www.mongodb.org/display/DOCS/Schema+Design [2]: An application I worked on had 2GB (100 million documents) worth of data, however completely indexed database would take 25GB of storage and Index rebuild would take ~8 hours.
ilaksh 15y ago
I don't think that it is accurate to describe MongoDB as a key/value store. My designs for use of MongoDB have always included my own versioned schema representation to be used, for starters, in generating data entry screens. Which NoSQL databases did you use?
- JanezStupar 15y ago
  I have extensive experience with Lotus Domino and CouchDB. I know enough about MongoDB, that I know its not all that much different as far as design and usage patterns go. This link [1] tells me that basically mongo's data structure is indeed a Key/Value. The schema you are talking about is not the schema I mean. What I was talking about that for any nontrivial Key/Value based database system it would be prudent to keep a recipe of how to normalize the data to 3rd Normal Form. Keeping this 3rd Normal Form schema around would greatly ease many troubles that arise from using NoSQL databases. So what were you thinking about when you say "schema" - is it "relational" (normalized) schema or is it just the recipe that tells you what particular fields are for? [1]: http://www.mongodb.org/display/DOCS/Schema+Design http://www.mongodb.org/display/DOCS/Schema+Design
  - ilaksh 15y ago
    "Document-based" is more accurate. One thing that's nice about MongoDB is that you get a little bit more than Map/Reduce as far as querying. The idea is that when a type is embedded or linked (using the terms from the Schema Design link you gave) there is a record of that. And the data types for the fields also need to be recorded. It would be nice if attributes for which there exists a type (collection) were correctly specified as attributes of that type rather some other more general type (otherwise you may need to correctly associate those fields later for analysis). I plan to record my schema something like this (although probably will want to include some other information like field descriptions): [HTML] version: 1 default: text [Hash] version: 1 default: text [Post] version: 1 (id_: ObjectID) postid: text title: text author: {linked} User authorname: {embedded} User.screenname comments: {embedded} Comments [User] version: 1 (id_: ObjectID) screenname: text shash: hash [Comments] (id_: ObjectID) version: 1 text: text created: date name: text [Comments] (id_: ObjectID) version: 2 text: html created: date name: text approved: boolean Also, each record (here I am referring to the actual data collections, not the schema collection) will contain the id and version number of the type (referring to the schema collection). This is for generating screens. As fields are added, changed, or removed, new type versions are recorded. I am still hopeful that I can avoid having to dump the data into regular relational tables for most ad hoc reporting purposes by generating stuff like this http://www.mongodb.org/display/DOCS/Querying http://www.mongodb.org/display/DOCS/Querying and http://www.mongodb.org/display/DOCS/Aggregation+Framework http://www.mongodb.org/display/DOCS/Aggregation+Framework from a GUI tool to support filters, arrangement of fields, and grouping/aggregation.

latch 15y ago

> The Object/Relational impedance mismatch stays right there where you leave it. Somewhat umbrella...but I do think that document-based storage does make a difference. It certainly doesn't erase it and, as you say, it varies based on the use-case at hand, but I think most people would consider the development experience to have less friction (and that's certainly been the overwhelming anecdotal evidence I've heard (and can give)).

prodigal_erik 15y ago
I have to wonder whether that's because many devs started using schemaless databases for the first time over the last couple of years, and haven't yet really experienced the nightmare of data which was scribbled on by various forgotten buggy versions of the apps and never rigorously migrated (because they're self-selected to regard that as unimportant). I once worked on a Lotus Notes-based system with documents eventually reaching such nonsensical states that the dev team couldn't even say what app behavior would be appropriate, much less what the latest version of all our code would happen to do.
- JanezStupar 15y ago
  Exactly, I have done most of my work on legacy (think 10-15 years worth of data, millions of documents that were spawned in countless application versions, without any schema tracking whatsoever - everything is implicit in the document itself) Lotus Domino applications. Since there is no explicit database schema in these types of databases, what you didn't do at write, you have to do at read. And usually you want to use the latest view or representation of data, what do you do with data that wasn't there ten versions ago. What do you do about data fields of wrong format? What do you do with data that is "orphaned" and cannot be referenced to other data. Yet it still is data and still is important. Don't get me wrong, I Love NoSQL and I like to use it. I just have enough experience with it to know that it is definitely not a silver bullet. By the way: If someone is looking to hire a guy who is not afraid of tackling this kind of issue, contact me. I have plenty of experience with coercion of non relational data into a form suitable for analysis.

DenisM 15y ago

>In the end it is merely a question of data normalization Can you elaborate? I understand any JSON can be shredded into a third normal form (minus the ordering problem, but let's leave that aside for now), is this what you refer to?

JanezStupar 15y ago
Yes this is exactly what I mean. Lets start from[1]. The kosher way of designing document oriented database is to simply embed everything within a document. This is great for operations such as full text search, retrieving whole objects, etc. We call this denormalized form. On the other hand, sometimes we want to analyze our data, e.g. we want to extract subsets of data and view them in isolation. With completely denormalized form this is expensive, since we either need to manually touch each and every document and extract data or we need to maintain indices that help us out. Both are extremely resource intensive. The third option is that we chop our objects into smaller objects and then link them together. But this will mean that retrieving a whole document will take longer (multiple database requests) it also adds an overhead of eliminating duplicates (two objects may appear identical, but really are not), etc... Denormalization gives you horizontal scalability, but takes away ad-hoc querying. It also wastes storage (document size is minor issue, Indices however will kill you [2]). Normalized data will take away horizontal scalability, give you ad-hoc querying, and save storage space. In the end for any kind of nontrivial system, you will eventually reach a point, where you will need to maintain two storages - a normalized and non-normalized form. The only difference is what your primary problem is and this sets whether you start out from Normalized or Denormalized storage, this will be your primary storage and source. The other kind will be an offline slave that will offer secondary functionality. E.g: If you start out from relational data and you want to build a FTS, you WILL have to denormalize data. On the other hand, if you start from Object/Document store and want to offer ad-hoc analytics, you WILL have to normalize your data. Its good to keep it in the back of your head. [1]: http://www.mongodb.org/display/DOCS/Schema+Design http://www.mongodb.org/display/DOCS/Schema+Design [2]: An application I worked on had 2GB (100 million documents) worth of data, however completely indexed database would take 25GB of storage and Index rebuild would take ~8 hours.

ilaksh 15y ago

I don't think that it is accurate to describe MongoDB as a key/value store. My designs for use of MongoDB have always included my own versioned schema representation to be used, for starters, in generating data entry screens. Which NoSQL databases did you use?

JanezStupar 15y ago
I have extensive experience with Lotus Domino and CouchDB. I know enough about MongoDB, that I know its not all that much different as far as design and usage patterns go. This link [1] tells me that basically mongo's data structure is indeed a Key/Value. The schema you are talking about is not the schema I mean. What I was talking about that for any nontrivial Key/Value based database system it would be prudent to keep a recipe of how to normalize the data to 3rd Normal Form. Keeping this 3rd Normal Form schema around would greatly ease many troubles that arise from using NoSQL databases. So what were you thinking about when you say "schema" - is it "relational" (normalized) schema or is it just the recipe that tells you what particular fields are for? [1]: http://www.mongodb.org/display/DOCS/Schema+Design http://www.mongodb.org/display/DOCS/Schema+Design
- ilaksh 15y ago
  "Document-based" is more accurate. One thing that's nice about MongoDB is that you get a little bit more than Map/Reduce as far as querying. The idea is that when a type is embedded or linked (using the terms from the Schema Design link you gave) there is a record of that. And the data types for the fields also need to be recorded. It would be nice if attributes for which there exists a type (collection) were correctly specified as attributes of that type rather some other more general type (otherwise you may need to correctly associate those fields later for analysis). I plan to record my schema something like this (although probably will want to include some other information like field descriptions): [HTML] version: 1 default: text [Hash] version: 1 default: text [Post] version: 1 (id_: ObjectID) postid: text title: text author: {linked} User authorname: {embedded} User.screenname comments: {embedded} Comments [User] version: 1 (id_: ObjectID) screenname: text shash: hash [Comments] (id_: ObjectID) version: 1 text: text created: date name: text [Comments] (id_: ObjectID) version: 2 text: html created: date name: text approved: boolean Also, each record (here I am referring to the actual data collections, not the schema collection) will contain the id and version number of the type (referring to the schema collection). This is for generating screens. As fields are added, changed, or removed, new type versions are recorded. I am still hopeful that I can avoid having to dump the data into regular relational tables for most ad hoc reporting purposes by generating stuff like this http://www.mongodb.org/display/DOCS/Querying http://www.mongodb.org/display/DOCS/Querying and http://www.mongodb.org/display/DOCS/Aggregation+Framework http://www.mongodb.org/display/DOCS/Aggregation+Framework from a GUI tool to support filters, arrangement of fields, and grouping/aggregation.