5 ms·

NoSQL is the duck-billed platypus in the evolution of databases. Absolutely fascinating, provides great material for PhD students, but a total dead end. Might

NoSQL is the duck-billed platypus in the evolution of databases. Absolutely fascinating, provides great material for PhD students, but a total dead end. Might survive in some isolated corners of the world; not one to bet on to be the next dominant species.

DrJokepu 17y ago
A while ago I have held the same opinion; that is up until I have faced a database problem that genuinely wasn't very RDMBS-friendly so I was "forced" to have a look at the alternatives. Ever since then I have realized that there there are large domains of problems that can be solved a lot easier with "nosql" configurations than with RDBMS systems. I mean stuff that I've used to solve with SQL databases before. My point is: it is always a good idea to have a larger perspective.
- justinsb 17y ago
  Interesting. What was the scenario? Which NoSQL DB did you end up using?
  - DrJokepu 17y ago
    It was essentially a key-value store of 256-bit audio fingerprints for an audio recognition system not unlike Shazaam but used for different purposes. The system was required to work on real time audio input so low response times were essentiall. And we're talking about many billions of keys here and a lot of queries per second. We've tried PostgreSql initially but response times were a real bottleneck. Then we have made a proof of concept with BerkeleyDb and it worked really well but we couldn't use it due to licensing issues so we have ended up coding our own key-value store
    - justinsb 17y ago
      I presume you used B-Tree indexing in PostgreSQL vs hash-indexing in BDB & your own code? PostgreSQL does support hash-indexing, but the documentation suggests it's not faster than B-trees anyway (probably because it would involve a second disk access). Let's be honest though - this is an incredibly niche use-case. There's nothing in the relational model that precludes supporting this use case optimally, but in practice engineering resources get focused on more mainstream uses. You did your due-diligence and found that support for your scenario was lacking in existing RDBMS systems. That's great, but I think you should be qualifying your support for NoSQL lest others with traditional use cases (best served by RDBMS systems) simply copy your conclusions where they don't apply.
    - neilk 17y ago
      Where do you work, the NSA?
      - DrJokepu 17y ago
        Broadcasting (television).
    - mahmud 17y ago
      If you still have ownership of the project, port it to Tokyo Cabinet.
- spudlyo 17y ago
  I had a similar change of perspective in 1997 when I worked at a well known Internet retailer. The entire catalog of items we sold (as well as customer reviews and other data) were all stored in a number of key value stores (Berkeley DBs) that were routinely built and pushed out to each web front end. This was very fast and for our purposes was much better than storing this information in a centralized SQL database.
  - justinsb 17y ago
    If you'd had viable open source SQL databases in 1997, would you have spent the engineering time on BerkeleyDB, or would you simply have replicated the master database onto each server? In 1997, you weren't choosing NoSQL vs SQL, you were choosing open-source vs commercial. Anyway, you're essentially storing pre-generated pages, which isn't a use case that I think anyone considers particularly database-appropriate (SQL or NoSQL). Using memcached to cache the data that is in active use seems more efficient, faster, and gives you the option to force through out-of-sequence updates (though again, that wasn't available to you off-the-shelf in 1997.) Complete re-generation of data might have worked for Amazon in 1997, but is this what they're using today?
    - spudlyo 17y ago
      With open source SQL databases there is no "simply" when it comes to replication. Even today, MySQL replication is brittle, and master/slave inconsistencies are the rule rather than the exception. Slave crashes often cause replayed transactions due to lack of atomicity in writing master.info and relay-log.info. The replication landscape with PostgeSQL is varied and essentially a bag on the side. Last I counted there were more than 10 different ways of doing it, a number involving trigger based log shipping. It wasn't about open-source vs commercial, it was about scaling the reads. The detail pages weren't pre-generated, they were based on read-only catalog data, which I think is entirely database appropriate. I imagine that complete re-generation of data is no longer done, but I'd be willing to bet that Berkeley DBs are still used in production somewhere.
      - justinsb 17y ago
gte910h 17y ago
I contend NoSQL is the introduction of nailguns to scaling up large projects with few consistancy constraints where priorly there were only screwdrivers and hammers. It's useful in it's arena. As are SQL based solutions in their arena. And there are tons of projects that can be done in either. Anyone who contends either is the only way hasn't spent enough time using the other. IMO, its the equivalent of panting as opposed to sweating: a functional way to get something done that fits some things quite well.
adamt 17y ago
I was initially sceptical over the NOSQL approach, but I have since changed my mind. Let me be very clear SQL is not dead, and it's the right tool for many jobs. However, if you want a cheap, replicated, multi-master, write heavy system - there is no simple way to this. Oracle is too expensive, I've used mysql-clsuter and pgpool-II etc, but in my experience they tend to introduce more failure points than they remove (at least in projects requiring a short development cycle). NOSQL pushes a degree of the complexity of the data model back to the application (e.g. you end up building some of your own indexes), but the data model is often no more complex in the end (less normalised, and greater flexibility of columns). The reward for this is massively simpler scaling. Something like Cassandra is so much easier to replicate to a multi-master multi-site system than PostgreSQL or MySQL. If you're building any form of semi-mission critical site (where you don't want to rely on or be tied to a single hosting provider) and you need to be multi-site & replicated from an early point, then NOSQL philosophy has a lot going for it. That said - I strongly hope that the MySQL/Postgres camp will fight back with a simple replication strategy that just works rather than is an bolt-on/afterthought. I find it helps to think of NOSQL being about simplifying the datastore to make it easier to replicate. So far I think only Cassandra does a persistent multi-master solution properly, but others will catch up.
- strlen 17y ago
  Cassandra isn't just multi-master, it's symmetric and masterless (peer-to-peer). It also isn't the only system with that property: other Dynamo "implementations" (Dynomite, Riak, Voldemort) are the same way. There are also multi-master systems that employ 3-PC like fault tolerant distributed commit protocols (e.g., Paxos with Leases, ZAB) and leader elections, allowing transactions to go through even if one of the coordinators fails.
daeken 17y ago
While I agree on the NoSQL movement in general (really, how many websites don't fit well on an RDBMS?), there are many useful cases for it. In my case, I'm storing massive binary analysis databases in a key-value store that allows quick graph traversal. Traversing code paths in a standard database or KV-store is insanely, insanely expensive, so a custom solution was really the only route to take.
- JustAGeek 17y ago
  Have you heard of neo4j? http://neo4j.org/ http://neo4j.org/ It's a graph database. I haven't used it but only attended a presentation by one of its developers, yet it seems to make a lot of sense. :)

DrJokepu 17y ago

A while ago I have held the same opinion; that is up until I have faced a database problem that genuinely wasn't very RDMBS-friendly so I was "forced" to have a look at the alternatives. Ever since then I have realized that there there are large domains of problems that can be solved a lot easier with "nosql" configurations than with RDBMS systems. I mean stuff that I've used to solve with SQL databases before. My point is: it is always a good idea to have a larger perspective.

justinsb 17y ago
Interesting. What was the scenario? Which NoSQL DB did you end up using?
- DrJokepu 17y ago
  It was essentially a key-value store of 256-bit audio fingerprints for an audio recognition system not unlike Shazaam but used for different purposes. The system was required to work on real time audio input so low response times were essentiall. And we're talking about many billions of keys here and a lot of queries per second. We've tried PostgreSql initially but response times were a real bottleneck. Then we have made a proof of concept with BerkeleyDb and it worked really well but we couldn't use it due to licensing issues so we have ended up coding our own key-value store
  - justinsb 17y ago
    I presume you used B-Tree indexing in PostgreSQL vs hash-indexing in BDB & your own code? PostgreSQL does support hash-indexing, but the documentation suggests it's not faster than B-trees anyway (probably because it would involve a second disk access). Let's be honest though - this is an incredibly niche use-case. There's nothing in the relational model that precludes supporting this use case optimally, but in practice engineering resources get focused on more mainstream uses. You did your due-diligence and found that support for your scenario was lacking in existing RDBMS systems. That's great, but I think you should be qualifying your support for NoSQL lest others with traditional use cases (best served by RDBMS systems) simply copy your conclusions where they don't apply.
  - neilk 17y ago
    Where do you work, the NSA?
    - DrJokepu 17y ago
      Broadcasting (television).
  - mahmud 17y ago
    If you still have ownership of the project, port it to Tokyo Cabinet.
spudlyo 17y ago
I had a similar change of perspective in 1997 when I worked at a well known Internet retailer. The entire catalog of items we sold (as well as customer reviews and other data) were all stored in a number of key value stores (Berkeley DBs) that were routinely built and pushed out to each web front end. This was very fast and for our purposes was much better than storing this information in a centralized SQL database.
- justinsb 17y ago
  If you'd had viable open source SQL databases in 1997, would you have spent the engineering time on BerkeleyDB, or would you simply have replicated the master database onto each server? In 1997, you weren't choosing NoSQL vs SQL, you were choosing open-source vs commercial. Anyway, you're essentially storing pre-generated pages, which isn't a use case that I think anyone considers particularly database-appropriate (SQL or NoSQL). Using memcached to cache the data that is in active use seems more efficient, faster, and gives you the option to force through out-of-sequence updates (though again, that wasn't available to you off-the-shelf in 1997.) Complete re-generation of data might have worked for Amazon in 1997, but is this what they're using today?
  - spudlyo 17y ago
    With open source SQL databases there is no "simply" when it comes to replication. Even today, MySQL replication is brittle, and master/slave inconsistencies are the rule rather than the exception. Slave crashes often cause replayed transactions due to lack of atomicity in writing master.info and relay-log.info. The replication landscape with PostgeSQL is varied and essentially a bag on the side. Last I counted there were more than 10 different ways of doing it, a number involving trigger based log shipping. It wasn't about open-source vs commercial, it was about scaling the reads. The detail pages weren't pre-generated, they were based on read-only catalog data, which I think is entirely database appropriate. I imagine that complete re-generation of data is no longer done, but I'd be willing to bet that Berkeley DBs are still used in production somewhere.
    - justinsb 17y ago

gte910h 17y ago

I contend NoSQL is the introduction of nailguns to scaling up large projects with few consistancy constraints where priorly there were only screwdrivers and hammers. It's useful in it's arena. As are SQL based solutions in their arena. And there are tons of projects that can be done in either. Anyone who contends either is the only way hasn't spent enough time using the other. IMO, its the equivalent of panting as opposed to sweating: a functional way to get something done that fits some things quite well.

adamt 17y ago

I was initially sceptical over the NOSQL approach, but I have since changed my mind. Let me be very clear SQL is not dead, and it's the right tool for many jobs. However, if you want a cheap, replicated, multi-master, write heavy system - there is no simple way to this. Oracle is too expensive, I've used mysql-clsuter and pgpool-II etc, but in my experience they tend to introduce more failure points than they remove (at least in projects requiring a short development cycle). NOSQL pushes a degree of the complexity of the data model back to the application (e.g. you end up building some of your own indexes), but the data model is often no more complex in the end (less normalised, and greater flexibility of columns). The reward for this is massively simpler scaling. Something like Cassandra is so much easier to replicate to a multi-master multi-site system than PostgreSQL or MySQL. If you're building any form of semi-mission critical site (where you don't want to rely on or be tied to a single hosting provider) and you need to be multi-site & replicated from an early point, then NOSQL philosophy has a lot going for it. That said - I strongly hope that the MySQL/Postgres camp will fight back with a simple replication strategy that just works rather than is an bolt-on/afterthought. I find it helps to think of NOSQL being about simplifying the datastore to make it easier to replicate. So far I think only Cassandra does a persistent multi-master solution properly, but others will catch up.

strlen 17y ago
Cassandra isn't just multi-master, it's symmetric and masterless (peer-to-peer). It also isn't the only system with that property: other Dynamo "implementations" (Dynomite, Riak, Voldemort) are the same way. There are also multi-master systems that employ 3-PC like fault tolerant distributed commit protocols (e.g., Paxos with Leases, ZAB) and leader elections, allowing transactions to go through even if one of the coordinators fails.

daeken 17y ago

While I agree on the NoSQL movement in general (really, how many websites don't fit well on an RDBMS?), there are many useful cases for it. In my case, I'm storing massive binary analysis databases in a key-value store that allows quick graph traversal. Traversing code paths in a standard database or KV-store is insanely, insanely expensive, so a custom solution was really the only route to take.

JustAGeek 17y ago
Have you heard of neo4j? http://neo4j.org/ http://neo4j.org/ It's a graph database. I haven't used it but only attended a presentation by one of its developers, yet it seems to make a lot of sense. :)