6 ms·

Codd's relational vision – has NoSQL come full circle?

107 points by lukaseder 13y ago

yepguy 13y ago
Author thinks he is being contrarian by defending SQL... huh?
- deleted 13y ago
  [deleted]
  - deleted 13y ago
    [deleted]
- andrewflnr 13y ago
  It's a NoSQL conference, so yes.
  - yepguy 13y ago
    Ah, I missed that bit.
- lafar6502 13y ago
  This happens when someone is trying to actually use NoSQL instead of just praising it ;)
- softwaredoug 13y ago
  hey thanks for your comment (I'm the author) I must be biased by my peer group :). Maybe I'm less concerned about SQL being a default choice than Mongo or something being the default choice? I say this as somebody that helps people migrate to NoSQL databases as a job.
sowhatquestion 13y ago
Excellent article. As promised, the author brings a historical perspective that's typically lacking in this debate (at least for someone like me who has casually followed it on HN). Not to stray too far off topic, but this article raised a question that I've had since starting software development in earnest... Why do we have to choose between heavily "normalized" relational databases that structure all data & thus allow more arbitrary queries, OR un-structured databases that are fast and flexible but often slow to query in complex ways? Why can't there be a hybrid "smart" database that dynamically generates indexes (to use the term loosely) based on how it's queried, in order to speed up similar queries in the future? With some kind of additional weight given to the most frequent queries. Granted, it would need some time to "warm up" (not unlike a tracing JIT compiler?), and the implementation might be fairly complex, but other than that, I can't think of any downsides...?
- lukaseder 13y ago
  You might enjoy PostgreSQL, then. It supports a variety of "weakly" structured data types, such as JSON, hashmaps (key/value types, also called hstore in PostgreSQL). Many databases also support XML data types. And of course, you can always store unstructured data in CLOBs and BLOBs
  - sowhatquestion 13y ago
    Actually, you read my mind! I'm currently working on a Rails project that uses PostgreSQL with hstore for semi-structured data such as user information, configuration, etc. Haven't gotten too far into it yet, but it looks promising. :) What I was trying to imagine in my post, though, was some hypothetical DB where all this would be abstracted away--i.e., it would expose a NoSQL-like interface for all data, while working behind the scenes to provide SQL-like speed for frequent queries. Hence the analogy that it would be the "tracing JIT" of databases.
    - mtdewcmu 13y ago
      I've been thinking along similar lines, I think. One problem I see with data in RDBMSs is that data gets transformed in the process of importing it to match the schema, and this transformation loses information, i.e. it's hard to recover the original meaning and context, because it's been contorted to fit the schema. Schemas tend to need to evolve, and so you'd like not to be stuck with the assumptions of schema 1.0 forever. Data isn't that big these days in relation to the speed of computers, so you could keep the data in its original form and leave open the ability to "reimport" to fit the current schema as needed. If you're building a website, for instance, then your web app is going to be doing a limited number of queries over and over. If you start from the queries, you can infer things about how the data should be organized, e.g. which indexes are needed. In fact, if you took a query and pre-generated all the possible results of that query and saved them, that's basically an index, the one that fits that query. The inputs to this hypothetical database would be the original unprocessed data, the set of queries you need, and mappings to transform the unprocessed data into queryable form, i.e. the equivalent of a schema. The inputs apart from the data proper would also be considered as data.
alok-g 13y ago
See also: SQL and NoSQL are two sides of the same coin, mathematically https://news.ycombinator.com/item?id=3313234 https://news.ycombinator.com/item?id=3313234
- lukaseder 13y ago
  That seems more like a plea for LINQ than anything else ;-) One of LINQ's main visions is to unify querying regardless of the data source - and it does so quite well. At the time the author of that article conceived LINQ, he intended to unify querying RDBMS, XML, and objects. "NoSQL" (whatever that is) is a type of data store that was not part of LINQ's original master plan. Thus, LINQ has to be retrofitted / enhanced to accommodate new requirements. It would be all too nice if things were as easy as a simple duality, specifically given the fact that the author of that article has now also created a company called Applied Duality Inc. But history will teach us where these things go. I currently don't see a second E.F. Codd to solve the complexity introduced with the new abundance of NoSQL data stores - yet. (As always, when Erik Meijer writes something: Very interesting read, nonetheless)
brasetvik 13y ago
See also: What Goes Around Comes Around (Stonebraker and Hellerstein) http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.113.5640 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.113.... That one's from the time where XML-databases were all the rage. Unfortunately, its publication year is missing. (The youngest cited article is from 97)
- lukaseder 13y ago
  Gotta love Stonebraker!
SeanLuke 13y ago
> This is neat because, in my Artificial Intelligence class in college, this kind of “reasoning” is presented as something advanced. In fact there’s a whole language, Prolog, that’s whole job is to take assertions about the world and give you extra facts. I had to learn all that stuff in AI when mundane-old SQL was right under my nose giving me the tools to learn anything I want about the world my relations represented! [Sigh] I guess broken worldviews like this are why Datalog never really got off the ground in the DB community.
coldtea 13y ago
Which reminds me of Fabian Pascal, probably somewhere drinking heavily to forget the current state of the DB industry. (Context: for more than a decade he tried to educate people on the relational model, the ill-thought back-to-the-past ideas that are NoSQL DBs, and more besides. Unfortunately his old site was scrapped, it seems, and has as of 2012 turned it into this blog: http://www.dbdebunk.com/ http://www.dbdebunk.com/ )
m_mueller 13y ago
> My point is that it’s dangerous to just grab a database off the shelf because its fun and trendy. You might end up creating a lot of technical debt for yourself. Do you know you’re handling an extremely high volume of a single somewhat-well-defined “thing”? Then the hierarchical model might be an amazing bet for you. Do you have many different kinds of records that link together, but don’t necessarily contain parts of the whole? A relational data store might be the ideal solution. This seems to me the most important point of the article and I fully agree here. What I'd like to add is that a DBMS like CouchDB is IMO fully suitable to the use case you describe. ArangoDB, which was also presented at that conference, appears to be even easier for those kinds of datastores - I haven't been able to play around with it however. There is no such thing as a perfect DBMS - and that applies to RDBMS as well. With Couch you give up sharding, Ad-hoc querying and per-document security, in return for conditionless replication, relative fool proofness in terms of indexing, the ability to have fully normalized relational data (compared to other NoSQL) and mobile offline capabilities.
- lukaseder 13y ago
  One thing about ArangoDB. Why don't they implement SQL instead of their weird COBOLESQUE AQL language?
  - m_mueller 13y ago
    I have been asking myself the same question - they tried to explain it but I didn't really understand the point to be honest. From what I get SQL was what they first pursued, but gave up at some point when it became clear that it wouldn't be ideal for their use case of a multi paradigm database (document as well as graph database). In general I agree that when developing a new product, it's good to question everything rather than blindly repeat what everyone has done for 20+ years. I'd love to read a blog entry about this decision though.
    - lukaseder 13y ago
      I agree that one should question 1-2 things. But when you look at an example (from https://www.arangodb.org/manuals/current/Aql.html https://www.arangodb.org/manuals/current/Aql.html): FOR u IN users FOR f IN friends FILTER u.active == true && f.active == true && u.id == f.userId RETURN u.name What would keep anyone from writing this as SELECT u.name FROM users AS u JOIN friends AS f ON u.id = f.userId WHERE u.active = true AND f.active = true Writing an entirely new language may inhibit adoption, if anything... Their rationale is given here in the FAQ (https://www.arangodb.org/faq https://www.arangodb.org/faq) > For example, think of nested list structures inside a document, graph traversals etc. There is no way to query such structures in standard SQL, and deviating from standard SQL does not make much sense. I would have thought that PostgreSQL had solved this pretty well. So has Oracle with its OBJECT types. There must have been a complete SQL show-stopper in their use cases for them to invent yet another language. I guess the language was inspired by XQuery, even if they don't mention XQuery in the FAQ.
deleted 13y ago
[deleted]
Delphiza 13y ago
One thing that is never directly mentioned in the history of SQL is the available memory, cpu and storage at the time. When a mainframe has 192KB of core memory and a 300KB/s disk transfer speed, coming up with a way to access all that data was important. The OP example of data in a text file would be okay(ish) today because you could load up the whole in memory and be done with it. Yes, we have more data, more rows, and larger records but are solving different problems with different tools and significantly more horsepower at our disposal. This is not anti-SQL, but the SQL-as-a-solution in history needs more context. With that context, those lessons may be less relevant.

107 points by lukaseder 13y ago

yepguy 13y ago

Author thinks he is being contrarian by defending SQL... huh?

deleted 13y ago
[deleted]
- deleted 13y ago
  [deleted]
andrewflnr 13y ago
It's a NoSQL conference, so yes.
- yepguy 13y ago
  Ah, I missed that bit.
lafar6502 13y ago
This happens when someone is trying to actually use NoSQL instead of just praising it ;)
softwaredoug 13y ago
hey thanks for your comment (I'm the author) I must be biased by my peer group :). Maybe I'm less concerned about SQL being a default choice than Mongo or something being the default choice? I say this as somebody that helps people migrate to NoSQL databases as a job.

sowhatquestion 13y ago

Excellent article. As promised, the author brings a historical perspective that's typically lacking in this debate (at least for someone like me who has casually followed it on HN). Not to stray too far off topic, but this article raised a question that I've had since starting software development in earnest... Why do we have to choose between heavily "normalized" relational databases that structure all data & thus allow more arbitrary queries, OR un-structured databases that are fast and flexible but often slow to query in complex ways? Why can't there be a hybrid "smart" database that dynamically generates indexes (to use the term loosely) based on how it's queried, in order to speed up similar queries in the future? With some kind of additional weight given to the most frequent queries. Granted, it would need some time to "warm up" (not unlike a tracing JIT compiler?), and the implementation might be fairly complex, but other than that, I can't think of any downsides...?

lukaseder 13y ago
You might enjoy PostgreSQL, then. It supports a variety of "weakly" structured data types, such as JSON, hashmaps (key/value types, also called hstore in PostgreSQL). Many databases also support XML data types. And of course, you can always store unstructured data in CLOBs and BLOBs
- sowhatquestion 13y ago
  Actually, you read my mind! I'm currently working on a Rails project that uses PostgreSQL with hstore for semi-structured data such as user information, configuration, etc. Haven't gotten too far into it yet, but it looks promising. :) What I was trying to imagine in my post, though, was some hypothetical DB where all this would be abstracted away--i.e., it would expose a NoSQL-like interface for all data, while working behind the scenes to provide SQL-like speed for frequent queries. Hence the analogy that it would be the "tracing JIT" of databases.
  - mtdewcmu 13y ago
    I've been thinking along similar lines, I think. One problem I see with data in RDBMSs is that data gets transformed in the process of importing it to match the schema, and this transformation loses information, i.e. it's hard to recover the original meaning and context, because it's been contorted to fit the schema. Schemas tend to need to evolve, and so you'd like not to be stuck with the assumptions of schema 1.0 forever. Data isn't that big these days in relation to the speed of computers, so you could keep the data in its original form and leave open the ability to "reimport" to fit the current schema as needed. If you're building a website, for instance, then your web app is going to be doing a limited number of queries over and over. If you start from the queries, you can infer things about how the data should be organized, e.g. which indexes are needed. In fact, if you took a query and pre-generated all the possible results of that query and saved them, that's basically an index, the one that fits that query. The inputs to this hypothetical database would be the original unprocessed data, the set of queries you need, and mappings to transform the unprocessed data into queryable form, i.e. the equivalent of a schema. The inputs apart from the data proper would also be considered as data.

alok-g 13y ago

See also: SQL and NoSQL are two sides of the same coin, mathematically https://news.ycombinator.com/item?id=3313234 https://news.ycombinator.com/item?id=3313234

lukaseder 13y ago
That seems more like a plea for LINQ than anything else ;-) One of LINQ's main visions is to unify querying regardless of the data source - and it does so quite well. At the time the author of that article conceived LINQ, he intended to unify querying RDBMS, XML, and objects. "NoSQL" (whatever that is) is a type of data store that was not part of LINQ's original master plan. Thus, LINQ has to be retrofitted / enhanced to accommodate new requirements. It would be all too nice if things were as easy as a simple duality, specifically given the fact that the author of that article has now also created a company called Applied Duality Inc. But history will teach us where these things go. I currently don't see a second E.F. Codd to solve the complexity introduced with the new abundance of NoSQL data stores - yet. (As always, when Erik Meijer writes something: Very interesting read, nonetheless)

brasetvik 13y ago

See also: What Goes Around Comes Around (Stonebraker and Hellerstein) http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.113.5640 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.113.... That one's from the time where XML-databases were all the rage. Unfortunately, its publication year is missing. (The youngest cited article is from 97)

lukaseder 13y ago
Gotta love Stonebraker!

SeanLuke 13y ago

> This is neat because, in my Artificial Intelligence class in college, this kind of “reasoning” is presented as something advanced. In fact there’s a whole language, Prolog, that’s whole job is to take assertions about the world and give you extra facts. I had to learn all that stuff in AI when mundane-old SQL was right under my nose giving me the tools to learn anything I want about the world my relations represented! [Sigh] I guess broken worldviews like this are why Datalog never really got off the ground in the DB community.

coldtea 13y ago

Which reminds me of Fabian Pascal, probably somewhere drinking heavily to forget the current state of the DB industry. (Context: for more than a decade he tried to educate people on the relational model, the ill-thought back-to-the-past ideas that are NoSQL DBs, and more besides. Unfortunately his old site was scrapped, it seems, and has as of 2012 turned it into this blog: http://www.dbdebunk.com/ http://www.dbdebunk.com/ )

m_mueller 13y ago

> My point is that it’s dangerous to just grab a database off the shelf because its fun and trendy. You might end up creating a lot of technical debt for yourself. Do you know you’re handling an extremely high volume of a single somewhat-well-defined “thing”? Then the hierarchical model might be an amazing bet for you. Do you have many different kinds of records that link together, but don’t necessarily contain parts of the whole? A relational data store might be the ideal solution. This seems to me the most important point of the article and I fully agree here. What I'd like to add is that a DBMS like CouchDB is IMO fully suitable to the use case you describe. ArangoDB, which was also presented at that conference, appears to be even easier for those kinds of datastores - I haven't been able to play around with it however. There is no such thing as a perfect DBMS - and that applies to RDBMS as well. With Couch you give up sharding, Ad-hoc querying and per-document security, in return for conditionless replication, relative fool proofness in terms of indexing, the ability to have fully normalized relational data (compared to other NoSQL) and mobile offline capabilities.

lukaseder 13y ago
One thing about ArangoDB. Why don't they implement SQL instead of their weird COBOLESQUE AQL language?
- m_mueller 13y ago
  I have been asking myself the same question - they tried to explain it but I didn't really understand the point to be honest. From what I get SQL was what they first pursued, but gave up at some point when it became clear that it wouldn't be ideal for their use case of a multi paradigm database (document as well as graph database). In general I agree that when developing a new product, it's good to question everything rather than blindly repeat what everyone has done for 20+ years. I'd love to read a blog entry about this decision though.
  - lukaseder 13y ago
    I agree that one should question 1-2 things. But when you look at an example (from https://www.arangodb.org/manuals/current/Aql.html https://www.arangodb.org/manuals/current/Aql.html): FOR u IN users FOR f IN friends FILTER u.active == true && f.active == true && u.id == f.userId RETURN u.name What would keep anyone from writing this as SELECT u.name FROM users AS u JOIN friends AS f ON u.id = f.userId WHERE u.active = true AND f.active = true Writing an entirely new language may inhibit adoption, if anything... Their rationale is given here in the FAQ (https://www.arangodb.org/faq https://www.arangodb.org/faq) > For example, think of nested list structures inside a document, graph traversals etc. There is no way to query such structures in standard SQL, and deviating from standard SQL does not make much sense. I would have thought that PostgreSQL had solved this pretty well. So has Oracle with its OBJECT types. There must have been a complete SQL show-stopper in their use cases for them to invent yet another language. I guess the language was inspired by XQuery, even if they don't mention XQuery in the FAQ.

deleted 13y ago

[deleted]

Delphiza 13y ago

One thing that is never directly mentioned in the history of SQL is the available memory, cpu and storage at the time. When a mainframe has 192KB of core memory and a 300KB/s disk transfer speed, coming up with a way to access all that data was important. The OP example of data in a text file would be okay(ish) today because you could load up the whole in memory and be done with it. Yes, we have more data, more rows, and larger records but are solving different problems with different tools and significantly more horsepower at our disposal. This is not anti-SQL, but the SQL-as-a-solution in history needs more context. With that context, those lessons may be less relevant.