4 ms·

SciDB – relational daddy answers Google, Hadoop, NoSQL

42 points by bensummers 16y ago

arethuza 16y ago
I've been designing and developing systems for over twenty years and in the last few months I've been using "NoSQL" databases a fair bit (CouchDB on the server and implementing my own lightweight one within JavaScript on the iPhone/iPad/...) and it's made me admit to a guilty secret: I hate SQL. In fact, I've always hated SQL - I just never admitted it to anyone. Very few applications I've ever designed ever seemed like a good fit when mapped to a relational data model - they always looked ugly to me and the resulting contortions required to map to and from in-memory objects pretty much always made me wish there was a better way (ORMs seemed to help initially but always became problematic in the long term). The most common reaction from colleagues when I mention document oriented databases is "they will become a mess over time". Sorry, every large commercial application that I have seen that uses SQL databases is already a mess. Look at your average production ERP data model - they are usually things of Lovecraftian nightmares. If he is amazed that Facebook can run with MySQL "40,000" ways, then I'm equally amazed that SAP can run at all! However, both seem to work fairly happily - just because you don't personally like a technology doesn't mean that others can't be perfectly happy with it.
- nodata 16y ago
  You compared 20 years of SQL use with a few months of NoSQL use. Do you think that NoSQL will become more of a mess than traditional databases over a long period of time, so much so that it becomes unusable, or do you think the app relying on good data will keep the data clean?
  - arethuza 16y ago
    I'm not sure if NoSQL will be any better, but they can't really be much worse that the reality of most relational designs that I have seen (especially for apps that have evolved over many years).
    - gfodor 16y ago
      The nice thing about NoSQL is you have no idea of what format the data in the database actually is. So, you'll never actually know how much worse it is. Ignorance is bliss!
      - arethuza 16y ago
        I suspect there will a healthy market for analysis tools for document oriented databases to allow people to find out what they actually have in their documents.
    - fauigerzigerk 16y ago
      I have seen horrible relational schemas as well. But they are islands of sanity in the ocean of custom data formats and encoding rules. Interestingly, the worst relational schemas are often justified using exactly the same arguments we're hearing out of the NoSQL camp today.
- rbanffy 16y ago
  Can you imagine an ERP running on a non-relational database? The complexity of the ERP SQL database does not reflect a failure of the relational model more than the insanely complicated nature of the problem being solved. Interestingly, I started my career on NoSQL databases in the 80's. We just didn't call them that at the time.
  - arethuza 16y ago
    I don't see why not - ERP concepts probably predate relational databases. The biggest problem would probably be reporting - but you could have a NoSQL database for operational entities and a link to either a relational or dedicated OLAP database for reporting.
- fauigerzigerk 16y ago
  "Very few applications I've ever designed ever seemed like a good fit when mapped to a relational data model" I think for many applications it's sign of bad design if the data model really fits the application in the sense that application data structures have 1:1 mappings in the database. It's similar to the relationship between the UI and the underlying object model (or whatever the paradigm may be). You need separation of concerns. Data lives longer and usually serves more than one application over time. Fitting it too closely to the needs of one particular application at one particular point in time causes too many dependencies. Of course, there are exceptions. Sometimes applications are trivial. Sometimes data isn't analyzed or used in other contexts. Sometimes performance dominates everything else. But for most applications, the infamaous impedance mismatch is an automatic consequence of acknowledging the different requirements we have for applications and data respectively.
- ergo98 16y ago
  As another poster mentioned, data lives on while applications come and go. This is a truism that has shown itself time and time again, in organization after organization. The data should be architected for its own consistency, usability (and re-usability), and robustness. Of course for very complex needs the data will intrinsically become incredibly complex, which is why you should be using decent design tools to organize and document at a higher level. Applications are layered on top (often many applications), and often there will be an impedance mismatch. However what I hear a lot from the NoSQL camp is essentially that by hiding the complexity away into unbounded documents, the problem doesn't exist. That is, pardon the bluntness, unbelievably naive. Try to model one of those ERP systems and seriously -- have fun. There's no big surprise that most of the success stories of NoSQL have been examples of intense simplicity scaled to large heights. Not every problem is so simple. Few problems are so simple. To your orginal comment -- while SQL is not the most elegant language, I like it enough. I love relational database design, however, as there is a purity and elegance in it that is very rewarding. In many ways it's like programming in assembly, and while it might not be for everyone, for some of us it's deeply enjoyable.
siculars 16y ago
Getting through that article was quite difficult, numerous typos. What happened to 'editors'? Basically Stonebraker, who should generally be read, is saying don't count ACID out just yet.
- rbanffy 16y ago
  > What happened to 'editors'? The ad blockers killed them...
  - planckscnst 16y ago
    And it was the abusive use of flickering, garish ads that brought them into popularity.
  - deleted 16y ago
    [deleted]
- ergo98 16y ago
  I'm still unsure how what they're describing differs from VoltDB. This initiative seems like it's primarily geared at churning out project names to continue to get industry attention.
  - jhugg 16y ago
    (From my comment on another thread) VoltDB is focused on mutating an in-memory dataset at ultra-high speed with full transactions and ACID. SciDB is focused on analyzing petabytes of scientific data in array form, where people care about things like data provenance and accumulated error bounds. Almost polar opposites in use, but similar in philosophy. Both believe specialized systems are the way to handle specialized workloads.
JonnieCache 16y ago
Single page view: http://www.theregister.co.uk/2010/09/13/michael_stonebraker_interview/print.html http://www.theregister.co.uk/2010/09/13/michael_stonebraker_...
serichsen 16y ago
That is an interesting article, but sloppy writing.

42 points by bensummers 16y ago

I've been designing and developing systems for over twenty years and in the last few months I've been using "NoSQL" databases a fair bit (CouchDB on the server and implementing my own lightweight one within JavaScript on the iPhone/iPad/...) and it's made me admit to a guilty secret: I hate SQL. In fact, I've always hated SQL - I just never admitted it to anyone. Very few applications I've ever designed ever seemed like a good fit when mapped to a relational data model - they always looked ugly to me and the resulting contortions required to map to and from in-memory objects pretty much always made me wish there was a better way (ORMs seemed to help initially but always became problematic in the long term). The most common reaction from colleagues when I mention document oriented databases is "they will become a mess over time". Sorry, every large commercial application that I have seen that uses SQL databases is already a mess. Look at your average production ERP data model - they are usually things of Lovecraftian nightmares. If he is amazed that Facebook can run with MySQL "40,000" ways, then I'm equally amazed that SAP can run at all! However, both seem to work fairly happily - just because you don't personally like a technology doesn't mean that others can't be perfectly happy with it.

nodata 16y ago
You compared 20 years of SQL use with a few months of NoSQL use. Do you think that NoSQL will become more of a mess than traditional databases over a long period of time, so much so that it becomes unusable, or do you think the app relying on good data will keep the data clean?
- arethuza 16y ago
  I'm not sure if NoSQL will be any better, but they can't really be much worse that the reality of most relational designs that I have seen (especially for apps that have evolved over many years).
  - gfodor 16y ago
    The nice thing about NoSQL is you have no idea of what format the data in the database actually is. So, you'll never actually know how much worse it is. Ignorance is bliss!
    - arethuza 16y ago
      I suspect there will a healthy market for analysis tools for document oriented databases to allow people to find out what they actually have in their documents.
  - fauigerzigerk 16y ago
    I have seen horrible relational schemas as well. But they are islands of sanity in the ocean of custom data formats and encoding rules. Interestingly, the worst relational schemas are often justified using exactly the same arguments we're hearing out of the NoSQL camp today.
rbanffy 16y ago
Can you imagine an ERP running on a non-relational database? The complexity of the ERP SQL database does not reflect a failure of the relational model more than the insanely complicated nature of the problem being solved. Interestingly, I started my career on NoSQL databases in the 80's. We just didn't call them that at the time.
- arethuza 16y ago
  I don't see why not - ERP concepts probably predate relational databases. The biggest problem would probably be reporting - but you could have a NoSQL database for operational entities and a link to either a relational or dedicated OLAP database for reporting.
fauigerzigerk 16y ago
"Very few applications I've ever designed ever seemed like a good fit when mapped to a relational data model" I think for many applications it's sign of bad design if the data model really fits the application in the sense that application data structures have 1:1 mappings in the database. It's similar to the relationship between the UI and the underlying object model (or whatever the paradigm may be). You need separation of concerns. Data lives longer and usually serves more than one application over time. Fitting it too closely to the needs of one particular application at one particular point in time causes too many dependencies. Of course, there are exceptions. Sometimes applications are trivial. Sometimes data isn't analyzed or used in other contexts. Sometimes performance dominates everything else. But for most applications, the infamaous impedance mismatch is an automatic consequence of acknowledging the different requirements we have for applications and data respectively.
ergo98 16y ago
As another poster mentioned, data lives on while applications come and go. This is a truism that has shown itself time and time again, in organization after organization. The data should be architected for its own consistency, usability (and re-usability), and robustness. Of course for very complex needs the data will intrinsically become incredibly complex, which is why you should be using decent design tools to organize and document at a higher level. Applications are layered on top (often many applications), and often there will be an impedance mismatch. However what I hear a lot from the NoSQL camp is essentially that by hiding the complexity away into unbounded documents, the problem doesn't exist. That is, pardon the bluntness, unbelievably naive. Try to model one of those ERP systems and seriously -- have fun. There's no big surprise that most of the success stories of NoSQL have been examples of intense simplicity scaled to large heights. Not every problem is so simple. Few problems are so simple. To your orginal comment -- while SQL is not the most elegant language, I like it enough. I love relational database design, however, as there is a purity and elegance in it that is very rewarding. In many ways it's like programming in assembly, and while it might not be for everyone, for some of us it's deeply enjoyable.

siculars 16y ago

Getting through that article was quite difficult, numerous typos. What happened to 'editors'? Basically Stonebraker, who should generally be read, is saying don't count ACID out just yet.

rbanffy 16y ago
> What happened to 'editors'? The ad blockers killed them...
- planckscnst 16y ago
  And it was the abusive use of flickering, garish ads that brought them into popularity.
- deleted 16y ago
  [deleted]
ergo98 16y ago
I'm still unsure how what they're describing differs from VoltDB. This initiative seems like it's primarily geared at churning out project names to continue to get industry attention.
- jhugg 16y ago
  (From my comment on another thread) VoltDB is focused on mutating an in-memory dataset at ultra-high speed with full transactions and ACID. SciDB is focused on analyzing petabytes of scientific data in array form, where people care about things like data provenance and accumulated error bounds. Almost polar opposites in use, but similar in philosophy. Both believe specialized systems are the way to handle specialized workloads.

JonnieCache 16y ago

Single page view: http://www.theregister.co.uk/2010/09/13/michael_stonebraker_interview/print.html http://www.theregister.co.uk/2010/09/13/michael_stonebraker_...

serichsen 16y ago

That is an interesting article, but sloppy writing.