8 ms·

Relational databases performed a task that didn't need doing (1991)

91 points by blasdel 17y ago

deleted 17y ago
[deleted]
TheSOB88 17y ago
What?
jbarciauskas 17y ago
Every organization I've ever worked at has huge data quality issues. Also, a company's data is one of its most valuable assets, providing financial reporting, forecasting, market analysis, and transactional functions. The question I pose, then, is the data available within corporations today of a higher consistency and quality than it would be if all organizations used only key-value stores or a document stores or CSV files? I'm going to venture to say despite the generally abysmal quality of data in large organizations today, it'd be much worse without the tools made available through RDBMS's.
- protomyth 17y ago
  I'm not so sure. It seems that the shoehorning that is done to get data into the tables cause some problems. I am not sure if it is inherent in the model or just a function of the standard two team arrangement (dev & DBA). The rigid implementation requirements and mismatch with the languages developer use seems to be a problem.
  - mattmcknight 17y ago
    The two team arrangement has caused me no end of problems over the past 10 years of my life, beyond shoehorning. The separation of the database team from the development team introduces communication problems, priority mismatches, and fixes done on one side or the other that should have been made on the other side. If you are reading this and you have separate database and development teams, merge them now, or at the least, put control of the schema in the hands of the development team.
  - StrawberryFrog 17y ago
    I'm not so sure. It seems that the shoehorning that is done to get data into the tables cause some problems. Possibly. But I shudder to think of what it would be like if there was no powerful standard tool like SQL for essential add-hoc queries and fixes. That is what you'd get with custom file storage. Also the "shoehorning", if done right (normal forms and all that) prevents duplicating information, and all attendant problems.
bradgessler 17y ago
I think this would be more appropriately titled, "I think RDBMS has set the industry back by 10 years". Some of the more interesting tidbits: I can categorically state that relational databases set the commercial data processing industry back at least ten years and wasted many of the billions of dollars -- Virtually all commercial applications in the 1960's were based on files of fixed-length records of multiple fields, which were selected and merged. Codd's relational theory dressed up these concepts with the trappings of mathematics (wow, we lowly Cobol programmers are now mathematicians!) by calling files relations, records rows, fields domains, and merges joins. I would love to hear HN's thoughts on this.
- gchpaco 17y ago
  SQL being notoriously poor at hierarchal data has been known for quite a while; he's obviously very correct there. Oracle goes so far as to offer a non-standard CONNECT statement for tree traversal. And there's not a whole lot of sophistication to relational theory on the ground level. Optimizing it is hard, but that's always been the case. To a large degree we have designed things so that the data we try to store is as relational as possible, never mind the domain implications--then, the mismatch is covered up. A shopping cart is a perfect stupid problem easily solved by relational databases, until you start introducing the real world; special discounts, group packages, "buy these items and pay less $$ than you would getting them individually" is usually done through gross hackery, etc.
  - Retric 17y ago
    A databases job is to handle Data and a relational database can do that just fine. When you want to handle really complex relationships you need to write code because the rules are not abstract. You can create a horribly complex view / stored procedure and pretend it's the databases job, or you accept that some rules are stored on the database but not implemented using the database. Edit: None of the "database alternatives" help you solve the checkout problem. PS: It's like complaining that HS level Calculus does not let you solve a complex differential equation. Useful abstractions always have their limits because they are abstractions.
lsb 17y ago
The task they did do, that needed doing, was causing a shift in thinking that was tantamount to the shift you undergo when you learn Prolog, having only been exposed to Assembler. By abstracting away the storage model, you can reuse the same database calls in the browser (SQLite), on one machine (MySQL/Postgres), or on a cluster, and you're free to think about more business-specific logic.
xenoterracide 17y ago
"With the recent arrival of object-oriented databases, the industry may finally achieve some of the promises which were made 20 years ago about the capabilities of computers to automate and improve organizations." um... wow... this 1 sentence discredits the entire email. Object Oriented Databases were a HUGE failure. I can't be sure but PostgreSQL might be the only surviving one, and it survived because it went relational. just my 2 cents.
- jbooth 17y ago
  No it doesn't.. ok, early 90s, people thought object oriented everything was the way forward. But his main point, that blasting through a flat file without a ton of row-locking and transactional overhead could satisfy a number of needs better than the actual DBMS, is still true. If you need to be transactional, on the other hand, that's a different story.
  - deleted 17y ago
    [deleted]
- rapind 17y ago
  What's the difference between object oriented and document oriented databases? Aren't all of the recent key/value databases lately pretty much the same concept as an object database? Sorry if that's a dumb question...
  - wmf 17y ago
    OODBs tend to use navigation of object graphs (see http://c2.com/cgi/wiki?NavigationalDatabase http://c2.com/cgi/wiki?NavigationalDatabase ) as the key operation, while document or key-value DBs rely more on queries.
    - rapind 17y ago
      Thanks for the link. Dug around a bit and from what I can tell they are pretty similar. Object databases have been referred to as hierarchical databases (I.e. XML data), many of the key-value databases being touted in the past couple of years seem hierarchical to me, only with some additional indexing and query language packaged on top (which you could add yourself with Lucene or Sphinx I would assume). I must be missing something, but I'll keep digging.
davidw 17y ago
> Biological systems follow the rule "ontogeny recapitulates phylogeny", which states that every higher-level organism goes through a developmental history which mirrors the evolutionary development of the species itself. I learned that that was incorrect in high school. http://en.wikipedia.org/wiki/Recapitulation_theory http://en.wikipedia.org/wiki/Recapitulation_theory
- omouse 17y ago
  That theory was rejected one year after this letter.
- euccastro 17y ago
  True, but the phrase you quote has a merely rethoric use anyway. The OP wasn't basing any of his argumentation on that broken thesis.
  - davidw 17y ago
    No, but it still takes something away from the whole when you know it's flat out wrong. It's like starting out "Since the earth is flat, we believe files should be too".
    - euccastro 17y ago
      That's a bad illustration, since the false fact is used in the argumentation. I find bad spelling a better analogy. If the OP's spelling and grammar were insufferable, it would have "taken something away from the whole" and still not be worth commenting on. That said, no big deal. I only felt compelled to reply to your comment because I found it strange that it was the highest ranked.
amalcon 17y ago
People act like non-relational data storage is some kind of new idea, but it's not. Even if you discount flat files, there's still BerkelyDB (early 90's) and the things it was meant to improve on (earlier). Why, then, are relational databases popular? I find it hard to believe the author's claim that it's because people wanted to claim they were doing fancy math with their data storage, over all other considerations. If this were the case, Lisp would have been the programming lingua franca through the 70s and 80s, not C. Lisp is obviously closer to its mathematical roots. Maybe it's because relational databases got the nice consistency features first; Interbase certainly predates BerkeleyDB. I'm more inclined to think it's a combination of two factors: First, the database can do much of the heavy lifting for you. This is nice to application programmers who are stuck with the likes of C, FORTRAN, COBOL, and Pascal; it's less interesting with functional abstractions available. Second, you get the SQL prompt. It's a dangerous tool, and should almost never be used for updates, but it's really nice to be able to answer unanticipated questions by simply writing up a query. Think of what the Django admin buys you; this is like a miniature version of that.
- blasdel 17y ago
  His point is not that "people wanted to claim they were doing fancy math with their data storage, over all other considerations" -- but that they wanted to reify their extant degenerate patterns as theory.
  - amalcon 17y ago
    You're right, that does make more sense. It would still seem that Lisp would have been more prominent if the ability to claim good theory (regardless of what you're actually doing) was a primary consideration. The average rank-and-file programmer (or manager) today just doesn't care about theory.
    - rbranson 17y ago
      Agreed. Most development organizations, especially those in large enterprises, the primary users of RDBMS technology when this article was written, are more concerned with coming up with the correct solution at the fastest clip. As far as they're concerned, theory be damned. Relations provide a simple, easy-to-understand data modeling concept and the query language is relatively quick to pick up for most developers. RDBMS are like the Java of data containers. You'll never get fired for going with an RDBMS.
sfwc 17y ago
Seemingly the same Henry Baker who wrote all those papers on garbage collection. http://home.pipeline.com/~hbaker1/ http://home.pipeline.com/~hbaker1/ http://www.bakercapital.com/team_baker_g.html http://www.bakercapital.com/team_baker_g.html Also, http://c2.com/cgi/wiki?ResponseToBakersAntiRelationalPaper http://c2.com/cgi/wiki?ResponseToBakersAntiRelationalPaper (all via http://c2.com/cgi/wiki?HenryBaker http://c2.com/cgi/wiki?HenryBaker)
- mahmud 17y ago
  Henry Baker = Hans Boehm + Zed Shaw.
  - ThinkWriteMute 17y ago
    Oh SNAP
ars 17y ago
This: > Biological systems follow the rule "ontogeny recapitulates phylogeny", which states that every higher-level organism goes through a developmental history which mirrors the evolutionary development of the species itself. Is not true. And it had been known to not be true since 1922, and possibly even 1890.
- randallsquared 17y ago
  It was brought up as possibly-true in my 1980s high school, so I guess it was probably something he learned in high school or college in the 1950s or 1960s and never unlearned. People forget that so much more of what we thought we knew before google, wikipedia, and snopes was wrong. You have to work harder to be wrong on stuff like that, nowadays, since it's so easy to look up. In 1991, you had to go to or have a decent library to find out that that was well-known to be wrong, so a non-specialist might never question what they learned in long-ago school. Now, it's so easy to look up minor details as you write an essay that it seems willful if someone doesn't, but if you apply that standard to twentieth century essays, you'll come away with a view of the writers that's often unwarranted... Next up: spellcheck. ;)
edw519 17y ago
I had great difficulty in controlling my mirth while I read the self-congratulatory article "Database Systems: Achievements and Opportunities" in the October, 1991, issue of the Communications,... I had little difficult in controlling my mirth when I realized that in 18 years, some things haven't changed. Linkbait, drama, and trolling all still look the same. As a designer of commercial manufacturing applications on IBM mainframes in the late 1960's and early 1970's... If you're going to wave your resume, make sure it's "wavable". If I had had a hand as a designer of commercial manufacturing applications on IBM mainframes in the late 1960's and early 1970's, I sure wouldn't brag about it. They were an excellent example of what not to do: so expensive, so difficult to deploy and use, and so ineffective, that the whole world rushed out to write better apps on mini-computers, and eventually, PCs. Ironically, the one thing they did do well was their relational database storage systems. If you owned a multi-million dollar AMAPS installation in 1978, the COBOL apps were soon worthless. The only thing salvageable was the TOTAL DBMS. I can categorically state that relational databases set the commercial data processing industry back at least ten years and wasted many of the billions of dollars that were spent on data processing. This conclusion is based upon what data? Maybe in 1991, you could bullshit the ACM without supporting data, but 2009 readers demand citings. Wikis & google have exposed the posers. Unfortunately, relational databases performed a task that didn't need doing; e.g., these databases were orders of magnitude slower than the "flat files" they replaced, Again, based upon what data? From what planet? Just because someone overnormalized a commercial database doesn't make it the fault of the underlying technology. That would be like saying, "That webpage sucks, therefore HTML sucks." Why were relational databases such a Procrustean bed? Because organizations, budgets, products, etc., are hierarchical... Organizations, budgets, products, etc. are data sources and sinks and can be structured any number of ways, including hierarchical. But the lifeblood of any business is its order flow and business processes which are almost always ideally suited to be relational; they're "linked" to almost everything else. Not everything has to be in 4th normal form, but flat files and hirerarchical data bases are almost always a poor stepchild to RDBMS for business flow. These databases could also respond quickly to "real-time" requests for information, because the data was readily accessible through pointers and hash tables--without performing "joins". I guess it's not really fair to "debate" with an OP from a generation ago. Even with Moore's Law, he would have had a hard time wrapping his head around where the real bottlenecks would be today. But one thing really hasn't changed that much: throughput has rarely been on the critical path. Why sacrifice data integrity, adherance to business rules, and effective delivery of user needs for a few microseconds? I remember routinely witnessing subsecond intercontinental response time on massive relational database installations as early as 1981. Why didn't OP? Oh, and RDBMS with pointers and hash tables have been around since 1965: http://en.wikipedia.org/wiki/Pick_operating_system http://en.wikipedia.org/wiki/Pick_operating_system and now also support object oriented technology: http://en.wikipedia.org/wiki/InterSystems_Cach%C3%A9 http://en.wikipedia.org/wiki/InterSystems_Cach%C3%A9 I shudder to think about the large number of man-years that were devoted during the 1970's and 1980's to "optimizing" relational databases to the point where they could remotely compete in the marketplace. I shudder to think about the large number of man-years lost by PHBs who read drivel like this and waste the time of people who do real work with initiatives derived from these drama-based conclusions. Database research has produced a number of good results, but the relational database is not one of them. From someone who has built a career rescuing so many manufacturers and distributors from flat file systems with relational database technology to whoever posted this: thanks for the laugh. I really needed it on a tough Monday.
cpr 17y ago
People probably don't realize who Henry Baker is. http://en.wikipedia.org/wiki/Henry_Baker_(computer_scientist) http://en.wikipedia.org/wiki/Henry_Baker_(computer_scientist... Yes, he's trolling here a bit, but he's brilliant (if quirky). (Disclaimer: I knew him when I worked at MIT lo these many decades ago.)
47 17y ago
> With the recent arrival of object-oriented databases, the industry may finally achieve some of the promises which were made 20 years ago about the capabilities of computers to automate and improve organizations. Oh No! i thought NoSQL/Key-value databases are the future
JulianMorrison 17y ago
Seems to me, messing with immense flat files is going to be faster iff you have a Computer Scientist of an equal caliber to the ones who wrote <insert professional SQL DB here> to optimize your indexing and storage formats and disk access patterns and whatnot. That is, at the very least, going to be expensive. It may just be simpler to just throw Oracle/Hadoop and hardware at the problem.
chubbard 17y ago
I think this is a very poignant essay given the place in history we are at. In 1991 this essay looked like sour grapes over forgotten history, but I'm more interested in what lessons we can learn from previous system. It's interesting to hear about a time before SQL and relational systems, and how early developers dealt with persistence problems. The historical account on the controversy of relational technology rings true today given we are struggling with it as well. I heard someone say that when the industry was moving away from big iron to mini-computers and PCs the programs they wrote were considered worthless because they couldn't reuse them on the next platform. It was the data that was the jewel. Relational technology embraced separating data from the program to enable reusing data within another program or different hardware. Prior to that data and program were tied together. They were fixed and non-portable. Relational technology came to occupy separation of data and program to facilitate data portability. What's changed is the way in which we write programs. We aren't tied to machine architecture like we once were. We use python, ruby or Java which is portable among many platforms. What was once trash is now treasure because we aren't constrained to one architecture. Does that mean let's go back to flat files? No. I think it's combination of the two. The features like data separation is important. Network access to data is important. Query languages are important. Hierarchical has always been important, but we've probably down played how important. Scalability and speed are ever more important. What's not as important anymore is relational. Those other features can be separated.
giardini 17y ago
I wonder if relational databases would not be as popular had easy-to-use transactional filesystems been commonly available at an earlier time. To gain transactional integrity (ACID properties) in that day, one had to either purchase a database management system (usually hierarchical or network) and work within that system or be a heckuva programmer and write it oneself. Had transactional filesystems been commonly available, then file processing with those filesystems would have been akin to working with a relational database management system (RDBMS). But that leaves referential integrity (RI) as a programmer task. Unfortunately most sites I've worked at do not seem to believe that RI should be strictly enforced. So maybe it wouldn't make much difference.

91 points by blasdel 17y ago

deleted 17y ago

[deleted]

TheSOB88 17y ago

What?

jbarciauskas 17y ago

Every organization I've ever worked at has huge data quality issues. Also, a company's data is one of its most valuable assets, providing financial reporting, forecasting, market analysis, and transactional functions. The question I pose, then, is the data available within corporations today of a higher consistency and quality than it would be if all organizations used only key-value stores or a document stores or CSV files? I'm going to venture to say despite the generally abysmal quality of data in large organizations today, it'd be much worse without the tools made available through RDBMS's.

protomyth 17y ago
I'm not so sure. It seems that the shoehorning that is done to get data into the tables cause some problems. I am not sure if it is inherent in the model or just a function of the standard two team arrangement (dev & DBA). The rigid implementation requirements and mismatch with the languages developer use seems to be a problem.
- mattmcknight 17y ago
  The two team arrangement has caused me no end of problems over the past 10 years of my life, beyond shoehorning. The separation of the database team from the development team introduces communication problems, priority mismatches, and fixes done on one side or the other that should have been made on the other side. If you are reading this and you have separate database and development teams, merge them now, or at the least, put control of the schema in the hands of the development team.
- StrawberryFrog 17y ago
  I'm not so sure. It seems that the shoehorning that is done to get data into the tables cause some problems. Possibly. But I shudder to think of what it would be like if there was no powerful standard tool like SQL for essential add-hoc queries and fixes. That is what you'd get with custom file storage. Also the "shoehorning", if done right (normal forms and all that) prevents duplicating information, and all attendant problems.

bradgessler 17y ago

I think this would be more appropriately titled, "I think RDBMS has set the industry back by 10 years". Some of the more interesting tidbits: I can categorically state that relational databases set the commercial data processing industry back at least ten years and wasted many of the billions of dollars -- Virtually all commercial applications in the 1960's were based on files of fixed-length records of multiple fields, which were selected and merged. Codd's relational theory dressed up these concepts with the trappings of mathematics (wow, we lowly Cobol programmers are now mathematicians!) by calling files relations, records rows, fields domains, and merges joins. I would love to hear HN's thoughts on this.

gchpaco 17y ago
SQL being notoriously poor at hierarchal data has been known for quite a while; he's obviously very correct there. Oracle goes so far as to offer a non-standard CONNECT statement for tree traversal. And there's not a whole lot of sophistication to relational theory on the ground level. Optimizing it is hard, but that's always been the case. To a large degree we have designed things so that the data we try to store is as relational as possible, never mind the domain implications--then, the mismatch is covered up. A shopping cart is a perfect stupid problem easily solved by relational databases, until you start introducing the real world; special discounts, group packages, "buy these items and pay less $$ than you would getting them individually" is usually done through gross hackery, etc.
- Retric 17y ago
  A databases job is to handle Data and a relational database can do that just fine. When you want to handle really complex relationships you need to write code because the rules are not abstract. You can create a horribly complex view / stored procedure and pretend it's the databases job, or you accept that some rules are stored on the database but not implemented using the database. Edit: None of the "database alternatives" help you solve the checkout problem. PS: It's like complaining that HS level Calculus does not let you solve a complex differential equation. Useful abstractions always have their limits because they are abstractions.

lsb 17y ago

The task they did do, that needed doing, was causing a shift in thinking that was tantamount to the shift you undergo when you learn Prolog, having only been exposed to Assembler. By abstracting away the storage model, you can reuse the same database calls in the browser (SQLite), on one machine (MySQL/Postgres), or on a cluster, and you're free to think about more business-specific logic.

xenoterracide 17y ago

"With the recent arrival of object-oriented databases, the industry may finally achieve some of the promises which were made 20 years ago about the capabilities of computers to automate and improve organizations." um... wow... this 1 sentence discredits the entire email. Object Oriented Databases were a HUGE failure. I can't be sure but PostgreSQL might be the only surviving one, and it survived because it went relational. just my 2 cents.

jbooth 17y ago
No it doesn't.. ok, early 90s, people thought object oriented everything was the way forward. But his main point, that blasting through a flat file without a ton of row-locking and transactional overhead could satisfy a number of needs better than the actual DBMS, is still true. If you need to be transactional, on the other hand, that's a different story.
- deleted 17y ago
  [deleted]
rapind 17y ago
What's the difference between object oriented and document oriented databases? Aren't all of the recent key/value databases lately pretty much the same concept as an object database? Sorry if that's a dumb question...
- wmf 17y ago
  OODBs tend to use navigation of object graphs (see http://c2.com/cgi/wiki?NavigationalDatabase http://c2.com/cgi/wiki?NavigationalDatabase ) as the key operation, while document or key-value DBs rely more on queries.
  - rapind 17y ago
    Thanks for the link. Dug around a bit and from what I can tell they are pretty similar. Object databases have been referred to as hierarchical databases (I.e. XML data), many of the key-value databases being touted in the past couple of years seem hierarchical to me, only with some additional indexing and query language packaged on top (which you could add yourself with Lucene or Sphinx I would assume). I must be missing something, but I'll keep digging.

davidw 17y ago

> Biological systems follow the rule "ontogeny recapitulates phylogeny", which states that every higher-level organism goes through a developmental history which mirrors the evolutionary development of the species itself. I learned that that was incorrect in high school. http://en.wikipedia.org/wiki/Recapitulation_theory http://en.wikipedia.org/wiki/Recapitulation_theory

omouse 17y ago
That theory was rejected one year after this letter.
euccastro 17y ago
True, but the phrase you quote has a merely rethoric use anyway. The OP wasn't basing any of his argumentation on that broken thesis.
- davidw 17y ago
  No, but it still takes something away from the whole when you know it's flat out wrong. It's like starting out "Since the earth is flat, we believe files should be too".
  - euccastro 17y ago
    That's a bad illustration, since the false fact is used in the argumentation. I find bad spelling a better analogy. If the OP's spelling and grammar were insufferable, it would have "taken something away from the whole" and still not be worth commenting on. That said, no big deal. I only felt compelled to reply to your comment because I found it strange that it was the highest ranked.

amalcon 17y ago

People act like non-relational data storage is some kind of new idea, but it's not. Even if you discount flat files, there's still BerkelyDB (early 90's) and the things it was meant to improve on (earlier). Why, then, are relational databases popular? I find it hard to believe the author's claim that it's because people wanted to claim they were doing fancy math with their data storage, over all other considerations. If this were the case, Lisp would have been the programming lingua franca through the 70s and 80s, not C. Lisp is obviously closer to its mathematical roots. Maybe it's because relational databases got the nice consistency features first; Interbase certainly predates BerkeleyDB. I'm more inclined to think it's a combination of two factors: First, the database can do much of the heavy lifting for you. This is nice to application programmers who are stuck with the likes of C, FORTRAN, COBOL, and Pascal; it's less interesting with functional abstractions available. Second, you get the SQL prompt. It's a dangerous tool, and should almost never be used for updates, but it's really nice to be able to answer unanticipated questions by simply writing up a query. Think of what the Django admin buys you; this is like a miniature version of that.

blasdel 17y ago
His point is not that "people wanted to claim they were doing fancy math with their data storage, over all other considerations" -- but that they wanted to reify their extant degenerate patterns as theory.
- amalcon 17y ago
  You're right, that does make more sense. It would still seem that Lisp would have been more prominent if the ability to claim good theory (regardless of what you're actually doing) was a primary consideration. The average rank-and-file programmer (or manager) today just doesn't care about theory.
  - rbranson 17y ago
    Agreed. Most development organizations, especially those in large enterprises, the primary users of RDBMS technology when this article was written, are more concerned with coming up with the correct solution at the fastest clip. As far as they're concerned, theory be damned. Relations provide a simple, easy-to-understand data modeling concept and the query language is relatively quick to pick up for most developers. RDBMS are like the Java of data containers. You'll never get fired for going with an RDBMS.

sfwc 17y ago

Seemingly the same Henry Baker who wrote all those papers on garbage collection. http://home.pipeline.com/~hbaker1/ http://home.pipeline.com/~hbaker1/ http://www.bakercapital.com/team_baker_g.html http://www.bakercapital.com/team_baker_g.html Also, http://c2.com/cgi/wiki?ResponseToBakersAntiRelationalPaper http://c2.com/cgi/wiki?ResponseToBakersAntiRelationalPaper (all via http://c2.com/cgi/wiki?HenryBaker http://c2.com/cgi/wiki?HenryBaker)

mahmud 17y ago
Henry Baker = Hans Boehm + Zed Shaw.
- ThinkWriteMute 17y ago
  Oh SNAP

ars 17y ago

This: > Biological systems follow the rule "ontogeny recapitulates phylogeny", which states that every higher-level organism goes through a developmental history which mirrors the evolutionary development of the species itself. Is not true. And it had been known to not be true since 1922, and possibly even 1890.

randallsquared 17y ago
It was brought up as possibly-true in my 1980s high school, so I guess it was probably something he learned in high school or college in the 1950s or 1960s and never unlearned. People forget that so much more of what we thought we knew before google, wikipedia, and snopes was wrong. You have to work harder to be wrong on stuff like that, nowadays, since it's so easy to look up. In 1991, you had to go to or have a decent library to find out that that was well-known to be wrong, so a non-specialist might never question what they learned in long-ago school. Now, it's so easy to look up minor details as you write an essay that it seems willful if someone doesn't, but if you apply that standard to twentieth century essays, you'll come away with a view of the writers that's often unwarranted... Next up: spellcheck. ;)

edw519 17y ago

I had great difficulty in controlling my mirth while I read the self-congratulatory article "Database Systems: Achievements and Opportunities" in the October, 1991, issue of the Communications,... I had little difficult in controlling my mirth when I realized that in 18 years, some things haven't changed. Linkbait, drama, and trolling all still look the same. As a designer of commercial manufacturing applications on IBM mainframes in the late 1960's and early 1970's... If you're going to wave your resume, make sure it's "wavable". If I had had a hand as a designer of commercial manufacturing applications on IBM mainframes in the late 1960's and early 1970's, I sure wouldn't brag about it. They were an excellent example of what not to do: so expensive, so difficult to deploy and use, and so ineffective, that the whole world rushed out to write better apps on mini-computers, and eventually, PCs. Ironically, the one thing they did do well was their relational database storage systems. If you owned a multi-million dollar AMAPS installation in 1978, the COBOL apps were soon worthless. The only thing salvageable was the TOTAL DBMS. I can categorically state that relational databases set the commercial data processing industry back at least ten years and wasted many of the billions of dollars that were spent on data processing. This conclusion is based upon what data? Maybe in 1991, you could bullshit the ACM without supporting data, but 2009 readers demand citings. Wikis & google have exposed the posers. Unfortunately, relational databases performed a task that didn't need doing; e.g., these databases were orders of magnitude slower than the "flat files" they replaced, Again, based upon what data? From what planet? Just because someone overnormalized a commercial database doesn't make it the fault of the underlying technology. That would be like saying, "That webpage sucks, therefore HTML sucks." Why were relational databases such a Procrustean bed? Because organizations, budgets, products, etc., are hierarchical... Organizations, budgets, products, etc. are data sources and sinks and can be structured any number of ways, including hierarchical. But the lifeblood of any business is its order flow and business processes which are almost always ideally suited to be relational; they're "linked" to almost everything else. Not everything has to be in 4th normal form, but flat files and hirerarchical data bases are almost always a poor stepchild to RDBMS for business flow. These databases could also respond quickly to "real-time" requests for information, because the data was readily accessible through pointers and hash tables--without performing "joins". I guess it's not really fair to "debate" with an OP from a generation ago. Even with Moore's Law, he would have had a hard time wrapping his head around where the real bottlenecks would be today. But one thing really hasn't changed that much: throughput has rarely been on the critical path. Why sacrifice data integrity, adherance to business rules, and effective delivery of user needs for a few microseconds? I remember routinely witnessing subsecond intercontinental response time on massive relational database installations as early as 1981. Why didn't OP? Oh, and RDBMS with pointers and hash tables have been around since 1965: http://en.wikipedia.org/wiki/Pick_operating_system http://en.wikipedia.org/wiki/Pick_operating_system and now also support object oriented technology: http://en.wikipedia.org/wiki/InterSystems_Cach%C3%A9 http://en.wikipedia.org/wiki/InterSystems_Cach%C3%A9 I shudder to think about the large number of man-years that were devoted during the 1970's and 1980's to "optimizing" relational databases to the point where they could remotely compete in the marketplace. I shudder to think about the large number of man-years lost by PHBs who read drivel like this and waste the time of people who do real work with initiatives derived from these drama-based conclusions. Database research has produced a number of good results, but the relational database is not one of them. From someone who has built a career rescuing so many manufacturers and distributors from flat file systems with relational database technology to whoever posted this: thanks for the laugh. I really needed it on a tough Monday.

cpr 17y ago

People probably don't realize who Henry Baker is. http://en.wikipedia.org/wiki/Henry_Baker_(computer_scientist) http://en.wikipedia.org/wiki/Henry_Baker_(computer_scientist... Yes, he's trolling here a bit, but he's brilliant (if quirky). (Disclaimer: I knew him when I worked at MIT lo these many decades ago.)

47 17y ago

> With the recent arrival of object-oriented databases, the industry may finally achieve some of the promises which were made 20 years ago about the capabilities of computers to automate and improve organizations. Oh No! i thought NoSQL/Key-value databases are the future

JulianMorrison 17y ago

Seems to me, messing with immense flat files is going to be faster iff you have a Computer Scientist of an equal caliber to the ones who wrote <insert professional SQL DB here> to optimize your indexing and storage formats and disk access patterns and whatnot. That is, at the very least, going to be expensive. It may just be simpler to just throw Oracle/Hadoop and hardware at the problem.

chubbard 17y ago

I think this is a very poignant essay given the place in history we are at. In 1991 this essay looked like sour grapes over forgotten history, but I'm more interested in what lessons we can learn from previous system. It's interesting to hear about a time before SQL and relational systems, and how early developers dealt with persistence problems. The historical account on the controversy of relational technology rings true today given we are struggling with it as well. I heard someone say that when the industry was moving away from big iron to mini-computers and PCs the programs they wrote were considered worthless because they couldn't reuse them on the next platform. It was the data that was the jewel. Relational technology embraced separating data from the program to enable reusing data within another program or different hardware. Prior to that data and program were tied together. They were fixed and non-portable. Relational technology came to occupy separation of data and program to facilitate data portability. What's changed is the way in which we write programs. We aren't tied to machine architecture like we once were. We use python, ruby or Java which is portable among many platforms. What was once trash is now treasure because we aren't constrained to one architecture. Does that mean let's go back to flat files? No. I think it's combination of the two. The features like data separation is important. Network access to data is important. Query languages are important. Hierarchical has always been important, but we've probably down played how important. Scalability and speed are ever more important. What's not as important anymore is relational. Those other features can be separated.

giardini 17y ago

I wonder if relational databases would not be as popular had easy-to-use transactional filesystems been commonly available at an earlier time. To gain transactional integrity (ACID properties) in that day, one had to either purchase a database management system (usually hierarchical or network) and work within that system or be a heckuva programmer and write it oneself. Had transactional filesystems been commonly available, then file processing with those filesystems would have been akin to working with a relational database management system (RDBMS). But that leaves referential integrity (RI) as a programmer task. Unfortunately most sites I've worked at do not seem to believe that RI should be strictly enforced. So maybe it wouldn't make much difference.