5 ms·

So do you believe there is a use case for a document-oriented database? I feel like your comment writes off a huge swath of useful technology. I said this belo

by blaines 14y ago

So do you believe there is a use case for a document-oriented database? I feel like your comment writes off a huge swath of useful technology. I said this below, but I'll say it again. Data is malleable, and writing apps to fit around any datastore seems wrong. I write applications to fulfill their use case. When the needs of the application change, so could it's database (or other dependencies). http://gigaom.com/cloud/mongodb-or-mysql-why-not-both/ http://gigaom.com/cloud/mongodb-or-mysql-why-not-both/

gregjor 14y ago
I don't know if there's a use case. I have never had one come up in my own projects. I do sometimes need to manage a large number of documents (unstructured data) but I have found the Unix file system sufficient for those, maybe augmented by metadata in the database. I've worked on big applications with big databases where DB operations and aggregations started to get too slow but so far I or someone smarter on the team has always been able to figure out a solution without throwing the RDBMS out. One big advantage of working with RDBMSs is the roads are paved and well-mapped. By contrast you still need a machete and good luck in the NoSQL world, and even then you can get sick or lost in the jungle. I haven't personally worked on a web app with the kinds of database scaling issues I've dealt with in logistics applications involving tens of millions of movements every day. RDBMSs were invented to allow data malleability. Go back to the old textbooks and look at what preceded the relational model. Remember that data is more valuable and permanent than application code. When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights.
- blaines 14y ago
  Respectfully, I believe your comment is not very open minded. I believe the obvious answer is "Of course there's a use case!", just like there's obviously a use case for RDBMS, NoSQL (Mongo, Cassandra, BigTable, Redis, etc) serves a purpose. I'll give you a couple examples. Google stores a huge amount of data in BigTable [1] which is fundamental for the operation of Gmail at scale. Another, Facebook initially used Cassandra [2] for it's Inbox search, then migrated to HBase when their needs changed for their new messaging feature. Do we know if the the underlying data is stored in RDBMS? I'm guessing not at Google, but yes for Facebook/Inbox search. But my point is I'm not ruling it out, I'm open to different options. [1] http://en.wikipedia.org/wiki/BigTable http://en.wikipedia.org/wiki/BigTable [2] http://en.wikipedia.org/wiki/Apache_Cassandra#History http://en.wikipedia.org/wiki/Apache_Cassandra#History
  - gregjor 14y ago
    I wrote "I don't know if there's a use case. I have never had one come up in my own projects." I don't work for Facebook or Google, perhaps they have a use case. I guess I could assume there's a use case because the tools exist and people use them, but in my experience I have not had a need for Mongo or any NoSQL database. I'm not dismissing the tools out of hand -- I haven't used them except to play around with tutorials. I am pretty sure a lot of people start out with Mongo or whatever without having a use case defined and without having good reasons to eliminate the tried-and-true RDBMS solutions. "Use something cool and new" is not a use case. "I've read on slashdot that SQL databases don't scale" is not a use case.
    - taligent 14y ago
      > I haven't used them except to play around with tutorials. And yet you feel the need to lecture others on their choices ? Wow you are something.
  - gregjor 14y ago
    I'll add that email and Facebook postings and Instagram photos don't require the same kind of data consistency and durability that medical records, bank transactions, or rail car movements require. If Google loses some old emails or Facebook loses the last thousand "likes" on a Jersey Shore fan page that's not much of a catastrophe, and they don't guarantee their data anyway. For those use cases Mongo or Cassandra et al. may be the right fit. I work on a app that stores documents (PDFs) in Amazon S3 containers, and sometimes those documents get lost or inaccessible for some reason. We have backups for those. The only consistency requirement there is that the PDF opens and the name matches the name in the RDBMS. That's a much lower bar for consistency and durability than our credit card transaction data, for example.
    - taligent 14y ago
      This is complete and utter nonsense. Facebook and Instagram itself I imagine would care VERY much if data went missing as it would mean not just (a) user data going missing but (b) potentially huge inconsistencies in the social graph. Unless you have some evidence that those companies do not care about data durability ?
- einhverfr 14y ago
  I suspect that a lot of NoSQL folks come to NoSQL from MySQL. As for this part: >"When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights." The obvious answer from the MySQL/NoSQL folks is "What? That's crazy! Multiple apps hitting the same db? Are you nuts???" See my blog post at http://ledgersmbdev.blogspot.com/2012/09/or-modelling-interlude-postgresql-vs.html http://ledgersmbdev.blogspot.com/2012/09/or-modelling-interl... for more on this mentality.
  - gregjor 14y ago
    I wasn't able to figure out how to post a comment/reply on Blogger, sorry. Thanks for pointing me to your article from the HN flame war about MongoDB. If only there were more useful articles like this and less flaming. You wrote "...in MySQL, the application is king and the db a humble servant." That is exactly my experience, and it leads not only to terrible databases but to piles of awful code written around terrible databases. In 1974 Kernighan and Plauger wrote "Choose a data representation that makes your program simple" (Elements of Programming Style). Other esteemed programmers have said the same thing. A well-designed and normalized database schema with validations and access rules implemented in the database will reduce the amount of application code needed. I've written about keeping data management code in the database where it belongs (http://typicalprogrammer.com/?p=101 http://typicalprogrammer.com/?p=101), but you make a much clearer argument with real examples. A few years ago the Ruby on Rails people championed the idea of "application databases" and writing HTTP+XML APIs, which probably seemed like a great idea given that few, if any, Rails applications ever had databases that other applications needed to access. Now the NoSQL folks have taken this even further, eliminating structure from the data and all ACID compliance, separating the data from integrity enforcement, all in the name of scalability problems their app might have if they get as big as Google. I am stuck with MySQL most of the time because I work with broken web applications, and so far all but a couple have used MySQL. As a long-time Oracle and SQL Server programmer I almost cry trying to figure out some of the MySQL databases I come actoss. I am working on one now that uses GUUIDs as primary keys in every table, generated in PHP code, for reasons I can only attribute to ignorance.
einhverfr 14y ago
Yes, as a pre-processing or post-processing layer for an RDBMS ;-). It might also be a useful backplane for some kinds of applications, but most of those are going to have some sort of fixed schema, so you run into a bunch of issues with this. Another area is for high-write throw-away data. If you are facebook, do you really care if every message gets through or every feed element gets propagated? Also perhaps content management (which is a write seldom, read often field) might be good, particularly where data can be reconstructed from other sources. Maybe doing the writes first in an RDBMS and pushing out to MongoDB might be a good thing there? You could then do things like referrer tracking and log analysis back in your RDBMS. Although hybrid databases are way cooler. One idea we are throwing around regarding LedgerSMB development right now is using JSON types in PostgreSQL 9.2 to store extended attributes, so you could have a customer account, and decide you want to store some extra data about this that we don't support right now, and have it stored in JSON.

by blaines 14y ago

gregjor 14y ago

I don't know if there's a use case. I have never had one come up in my own projects. I do sometimes need to manage a large number of documents (unstructured data) but I have found the Unix file system sufficient for those, maybe augmented by metadata in the database. I've worked on big applications with big databases where DB operations and aggregations started to get too slow but so far I or someone smarter on the team has always been able to figure out a solution without throwing the RDBMS out. One big advantage of working with RDBMSs is the roads are paved and well-mapped. By contrast you still need a machete and good luck in the NoSQL world, and even then you can get sick or lost in the jungle. I haven't personally worked on a web app with the kinds of database scaling issues I've dealt with in logistics applications involving tens of millions of movements every day. RDBMSs were invented to allow data malleability. Go back to the old textbooks and look at what preceded the relational model. Remember that data is more valuable and permanent than application code. When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights.

blaines 14y ago
Respectfully, I believe your comment is not very open minded. I believe the obvious answer is "Of course there's a use case!", just like there's obviously a use case for RDBMS, NoSQL (Mongo, Cassandra, BigTable, Redis, etc) serves a purpose. I'll give you a couple examples. Google stores a huge amount of data in BigTable [1] which is fundamental for the operation of Gmail at scale. Another, Facebook initially used Cassandra [2] for it's Inbox search, then migrated to HBase when their needs changed for their new messaging feature. Do we know if the the underlying data is stored in RDBMS? I'm guessing not at Google, but yes for Facebook/Inbox search. But my point is I'm not ruling it out, I'm open to different options. [1] http://en.wikipedia.org/wiki/BigTable http://en.wikipedia.org/wiki/BigTable [2] http://en.wikipedia.org/wiki/Apache_Cassandra#History http://en.wikipedia.org/wiki/Apache_Cassandra#History
- gregjor 14y ago
  I wrote "I don't know if there's a use case. I have never had one come up in my own projects." I don't work for Facebook or Google, perhaps they have a use case. I guess I could assume there's a use case because the tools exist and people use them, but in my experience I have not had a need for Mongo or any NoSQL database. I'm not dismissing the tools out of hand -- I haven't used them except to play around with tutorials. I am pretty sure a lot of people start out with Mongo or whatever without having a use case defined and without having good reasons to eliminate the tried-and-true RDBMS solutions. "Use something cool and new" is not a use case. "I've read on slashdot that SQL databases don't scale" is not a use case.
  - taligent 14y ago
    > I haven't used them except to play around with tutorials. And yet you feel the need to lecture others on their choices ? Wow you are something.
- gregjor 14y ago
  I'll add that email and Facebook postings and Instagram photos don't require the same kind of data consistency and durability that medical records, bank transactions, or rail car movements require. If Google loses some old emails or Facebook loses the last thousand "likes" on a Jersey Shore fan page that's not much of a catastrophe, and they don't guarantee their data anyway. For those use cases Mongo or Cassandra et al. may be the right fit. I work on a app that stores documents (PDFs) in Amazon S3 containers, and sometimes those documents get lost or inaccessible for some reason. We have backups for those. The only consistency requirement there is that the PDF opens and the name matches the name in the RDBMS. That's a much lower bar for consistency and durability than our credit card transaction data, for example.
  - taligent 14y ago
    This is complete and utter nonsense. Facebook and Instagram itself I imagine would care VERY much if data went missing as it would mean not just (a) user data going missing but (b) potentially huge inconsistencies in the social graph. Unless you have some evidence that those companies do not care about data durability ?
einhverfr 14y ago
I suspect that a lot of NoSQL folks come to NoSQL from MySQL. As for this part: >"When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights." The obvious answer from the MySQL/NoSQL folks is "What? That's crazy! Multiple apps hitting the same db? Are you nuts???" See my blog post at http://ledgersmbdev.blogspot.com/2012/09/or-modelling-interlude-postgresql-vs.html http://ledgersmbdev.blogspot.com/2012/09/or-modelling-interl... for more on this mentality.
- gregjor 14y ago
  I wasn't able to figure out how to post a comment/reply on Blogger, sorry. Thanks for pointing me to your article from the HN flame war about MongoDB. If only there were more useful articles like this and less flaming. You wrote "...in MySQL, the application is king and the db a humble servant." That is exactly my experience, and it leads not only to terrible databases but to piles of awful code written around terrible databases. In 1974 Kernighan and Plauger wrote "Choose a data representation that makes your program simple" (Elements of Programming Style). Other esteemed programmers have said the same thing. A well-designed and normalized database schema with validations and access rules implemented in the database will reduce the amount of application code needed. I've written about keeping data management code in the database where it belongs (http://typicalprogrammer.com/?p=101 http://typicalprogrammer.com/?p=101), but you make a much clearer argument with real examples. A few years ago the Ruby on Rails people championed the idea of "application databases" and writing HTTP+XML APIs, which probably seemed like a great idea given that few, if any, Rails applications ever had databases that other applications needed to access. Now the NoSQL folks have taken this even further, eliminating structure from the data and all ACID compliance, separating the data from integrity enforcement, all in the name of scalability problems their app might have if they get as big as Google. I am stuck with MySQL most of the time because I work with broken web applications, and so far all but a couple have used MySQL. As a long-time Oracle and SQL Server programmer I almost cry trying to figure out some of the MySQL databases I come actoss. I am working on one now that uses GUUIDs as primary keys in every table, generated in PHP code, for reasons I can only attribute to ignorance.

einhverfr 14y ago

Yes, as a pre-processing or post-processing layer for an RDBMS ;-). It might also be a useful backplane for some kinds of applications, but most of those are going to have some sort of fixed schema, so you run into a bunch of issues with this. Another area is for high-write throw-away data. If you are facebook, do you really care if every message gets through or every feed element gets propagated? Also perhaps content management (which is a write seldom, read often field) might be good, particularly where data can be reconstructed from other sources. Maybe doing the writes first in an RDBMS and pushing out to MongoDB might be a good thing there? You could then do things like referrer tracking and log analysis back in your RDBMS. Although hybrid databases are way cooler. One idea we are throwing around regarding LedgerSMB development right now is using JSON types in PostgreSQL 9.2 to store extended attributes, so you could have a customer account, and decide you want to store some extra data about this that we don't support right now, and have it stored in JSON.