6 ms·

When non-traditional databases became popular, I thought it was primarily driven by people short-sightedly prioritizing development time over all of the good re

When non-traditional databases became popular, I thought it was primarily driven by people short-sightedly prioritizing development time over all of the good relational database features. Now I see things differently -- the non-traditional databases are just better at scaling horizontally, eventual consistency, and running in cloud environments than the traditional databases. They are easier to set up and use. Some of them now have pretty good relational features and schemas. During the past decade the Oracles of the world have continued to think dismissively of new non-traditional databases, as I did at first. The non-traditional databases got better and better while Oracle kept doing a lot of the same stuff. Oracle just didn't take the competition seriously, and that's fair enough because the competition didn't deserve to be taken seriously at first -- but now it does, and it's becoming a threat to Oracle's business. It's obvious that Amazon would want to use their own database products -- but the impressive thing is that those products, which are not very old, are already good enough to replace Oracle for a lot of use cases at very high scale.

AareyBaba 8y ago
Can you mention the popular non-traditional databases that are competitive with a traditional relational database ?
Derek_MK 8y ago
There are places where relational DBs are inherently better than noSQL (namely when you need fully up-to-date information on every query, no matter what) - But those situations are becoming more rare compared to situations where potentially stale data is an okay tradeoff for performance gains. Honestly, IMO the biggest issue holding noSQL databases back is lack of good documentation/support a lot of the time. Technically, though, they're going to become the standard for most use cases soon.
- lykr0n 8y ago
  The problem with noSQL is that there are too many players out there making duplicated efforts. You have MySQL & MariaDB and PostgreSQL as the two main SQL platforms, but you have too may to list in the noSQL field- most of them providing the same amount of functionality.
  - blihp 8y ago
    This always happens and the noSQL market will eventually consolidate / settle down to a handful of players just as the SQL market did.
    - DaiPlusPlus 8y ago
      That depends on the functionality requirements of noSQL customers. I posit that writing a new, indepedent RDBMS that implements the latest SQL spec with the same maganement and replication capabilities as the incumbents is on-par with trying to compete with WebKit and Gecko. The SQL Server engineering team at Microsoft is about 1,000 people according to my friends on campus. Whereas implementing a Key-Value Store is orders of magnitude simpler. There is plenty of room for innovation, of course - but you don’t need 1,000 people to build-on some new replication scheme or distributed backend. If you take a simple NoSQL system then tack on things like object scheme support, JSON blob value indexes, node graph support, etc you’ll end up with an RDBMS analogue. So the barrier-to-entry to building a NoSQL system is much lower than a RDBMS - and for many companies they might decide to build their own rather than extend an existing system - leading to more completion but not necessarily a better product.
      - blihp 8y ago
        It always starts that way. In time, the legacy baggage that gets added will turn the noSQL databases into the same bloated monsters that the SQL databases have become. There was a time when many of the SQL database options were the (relatively) simpler and more powerful options vs the wide variety of home-grown databases they eventually replaced. (i.e. there was a lot of baked in business logic etc. that the move to SQL databases forced companies to disentangle from the database engine itself) SQL Server was originally just a fork of Sybase SQL Server (a much smaller, simpler version) with some FoxPro tech bolted on to it...
        DaiPlusPlus 8y ago
        Right - and FoxPro was just another dBase clone. NoSQL today is where dBase was 30 years ago.
- just_myles 8y ago
  Agreed. I think for temporal data (accounting) reporting non-relational databases are fine.
  - just_myles 8y ago
    To clarify further. Of course, you will need a relational data repository.
- jefftk 8y ago
  > when you need fully up-to-date information on every query, no matter what This doesn't require something relational, just something centralized. And not even that; consider Spanner.
- amf12 8y ago
  > namely when you need fully up-to-date information on every query, no matter what So you basically want full consistency (vs eventual consistency) in a distributed environment. You can set up you noSQL that way. I don't see how relational DB's are "better" than noSQL in consistency.
  - ric2b 8y ago
    And if you do that you go back to the performance of one instance or (probably) worse.
    - amf12 8y ago
      Absolutely, but isn't that the case for SQL too? My point being, noSQL is not worse than SQL if all you care about is storing and retrieving documents vs joins, etc.
- threeseed 8y ago
  Why do people continue to use the term NoSQL ? It's ridiculous. There are hundreds maybe thousands of NoSQL databases ranging from Redis to MongoDB to Cassandra to InfluxDB to Druid. They pretty much have nothing in common other than storing data. Many of them support SQL directly, others via Spark and almost all have a SQL equivalent.
  - endymi0n 8y ago
    Exactly this, „non-traditional“ and noSQL is as inexact as it gets. It includes an incredibly wide range of architectures, from Spanner and CockroachDB which deliver ACID guarantees equal and even surpassing relational systems to pure performance tuned systems such as Aerospike which have few guarantees and features but are blazingly fast. There are pretty much all shades of consistency in between, like Cassandra and Scylla with tunable consistency amd LWT. Lumping them all together into a single category will yield poor results.
emmelaich 8y ago
My hot take is that K/V stores solved a problem they didn't know they had. To whit, de-normalised tables for the sake of "efficiency". It's pretty difficult to under normalise a two column table -- which is what K/V is.
cronix 8y ago
> The non-traditional databases got better and better while Oracle kept doing a lot of the same stuff. Oracle just didn't take the competition seriously, and that's fair enough because the competition didn't deserve to be taken seriously at first -- but now it does, and it's becoming a threat to Oracle's business. Yes, like traditional car companies not taking EV's too seriously, or Intel not taking mobile/ARM too seriously, etc. It seems the older and more established a company is, the thicker their forward-thinking blinders are. "We're king, no one can EVER touch us."
- __s 8y ago
  Part of this is because successful companies have the survivorship bias of yesterday
- rrcaptain 8y ago
  What do you mean traditional car companies not taking EVERYTHING IS seriously? Volzwagen is switching all their cars to electric.
  - gbear605 8y ago
    They could (should?) have been doing this a decade ago, but they didn’t. It took the relative success of Tesla to convince them.
jacques_chester 8y ago
This isn't really about relational vs non-relational. Redshift is still a relational database of the MPP genre.
hyperpallium 8y ago
"[N]on-traditional databases" with far better performance actually pre-date relational databases. Relational databases solved the problem they had, of requiring much developer effort to evolve for new application needs, and re-organization of the datastore for changing performance needs. Basically, everything is a table, and you can combine/separate tables, from whatever form it currently is, to whatever form you currently need. This is also helpful for reporting (as opposed to operational applications). But relational databases, like dynamic languages, pay a performance price for their flexibility. Of course, every other innovation has since been added, and there's Oracle's licensing too. So it's much more complex than the base technology of relational algebra.
- mathattack 8y ago
  Yes. Many people leave Oracle’s technology footprint out of frustration with their Salesforce.
jchanimal 8y ago
Shameless plug, but I think the new math that underlies global consistency is super interesting, I did a webcast with Professor Daniel Abadi about how Google Spanner differs from FaunaDB (my employer, which doesn’t require atomic clocks), you can get a link to the recording here https://www2.fauna.com/wcspannercalvin https://www2.fauna.com/wcspannercalvin
dgemm 8y ago
That is a hallmark of a disruptive technology, to not be as good at what the incumbent values but better in some other dimension that they don't.
- twblalock 8y ago
  The other interesting thing is that the latest database trends reflect the general trend in the industry away from monolithic solutions. Most of the new databases are not one-size-fits-all, but are optimized for narrower use cases.
grumpydba 8y ago
Most of those oracle databases are moving off to Aurora, which is pretty much traditional.
- repolfx 8y ago
  Traditional yes, and not even necessarily better. Amazon had a big outage in recent times that their own post-mortem linked to trying to move off Oracle to Aurora. Interestingly it was posted here to HN and repeatedly and immediately flagged off the page - it appears that some HN readers are very keen on suppressing anything that paints Oracle in a good light. It makes you wonder what other stories might be going missing that would alter people's impressions of this company.
  - grumpydba 8y ago
    This article was essentially Oracle PR. Andy Pavlo, database professor at CMU, has debunked it and gave details on the journalist's dishonesty. Also, anything is better than oracle. If not for technical reasons, price wise it makes a lot of sense.
    - repolfx 8y ago
      That sounds completely delusional I'm afraid. The article(s) were summaries of an internal post-mortem written by one of Oracle's competitors - the exact opposite of Oracle PR. And how exactly is some random professor going to debunk the conclusions of Amazon's own staff about their own infrastructure? Your last sentence appears to be typical of the problem I'm describing.
      - grumpydba 8y ago
        First: Andy Pavlo was contacted by the author of the article. He's all but random, he's one of the references in the field. The outage was quite minor, not user visible and cost something around 100k. I've seen such outages after an Oracle version upgrade so it's all but major. For your information I've been managing critical databases for more than a decade so I know that it might look unusual to you, but it's quite clear to anyone barely knowledgeable about database management that the article really overblown the issue. It was also mocked by Amazon's CTO. So please, manage databases for a few years. Then I might hear your opinion and maybe consider it.
gaius 8y ago
The non-traditional databases got better and better while Oracle kept doing a lot of the same stuff I'm not sure this is true, you can look at the new features in 8i, 9i, 10g, 11g, 12c and see that Oracle is adding a lot of features, and some of them are very impressive. The problem is most people don't need most of them and if you want any of them you have to pay for all of them. And the "non-traditional" databases had a pretty low bar. MongoDB just had to stay up for more than 5 minutes and not lose your data. Do they even do that yet?
- amf12 8y ago
  > MongoDB just had to stay up for more than 5 minutes and not lose your data. Do they even do that yet? What are you implying? Do you have any source for that? Some of the top tech companies out there use non-traditional databases, including MongoDB. They won't if the nodes crash every 5 minutes and/or there is data loss.
  - tluyben2 8y ago
    The parent was exaggerating but there are countless articles about how bad mongo is. These might be in the past, I do not know. Anecdotally I had horrible experiences with dataloss and instability, both of them incredibly hard to debug compared to say, MySql or Posgres. And when you see a hip company data leak it is often Mongo that was compromised (to be fair, usually because of it’s weak default settings and stupidity on the admin side, but still, it is a datapoint against mongo). Persobally I do not find it easier to use either: our ORM is a optionally schema’d layer; it works as flexible as Mongo and in naive setups, it is much much faster and easier to use (the latter is an opinion), without the risk of dataloss and with flexibility of db engine (it even supports Mongo because I was bored on a sunday).
    - grumpydba 8y ago
      Mongodb's storage engine is now wiredtiger. It is a bit young, but it's a really beautiful C codebase written by database veterans.
unscaled 8y ago
Of the three DB products mentioned in the tweet, only one is a non-traditional database: DynamoDB. Both Redshift and Aurora are traditional RDBMs, which are forked from very old codebases. Redshift is a fork of Postgres 8.0 that was adapted for OLAP use-cases, while Aurora is a fork of MySQL (or PostgreSQL in the PostgreSQL-compatible versions) that runs on a custom storage engine. The main reason technically-capable companies like Amazon or Netflix are moving away Oracle has nothing to do with horizontal scaling. It's the licensing costs. Oracle is just too expensive for them, and even if you are not Amazon yourself, RDS (or any other Cloud RDBMS) would be much cheaper for you.

AareyBaba 8y ago

Can you mention the popular non-traditional databases that are competitive with a traditional relational database ?

Derek_MK 8y ago

There are places where relational DBs are inherently better than noSQL (namely when you need fully up-to-date information on every query, no matter what) - But those situations are becoming more rare compared to situations where potentially stale data is an okay tradeoff for performance gains. Honestly, IMO the biggest issue holding noSQL databases back is lack of good documentation/support a lot of the time. Technically, though, they're going to become the standard for most use cases soon.

lykr0n 8y ago
The problem with noSQL is that there are too many players out there making duplicated efforts. You have MySQL & MariaDB and PostgreSQL as the two main SQL platforms, but you have too may to list in the noSQL field- most of them providing the same amount of functionality.
- blihp 8y ago
  This always happens and the noSQL market will eventually consolidate / settle down to a handful of players just as the SQL market did.
  - DaiPlusPlus 8y ago
    That depends on the functionality requirements of noSQL customers. I posit that writing a new, indepedent RDBMS that implements the latest SQL spec with the same maganement and replication capabilities as the incumbents is on-par with trying to compete with WebKit and Gecko. The SQL Server engineering team at Microsoft is about 1,000 people according to my friends on campus. Whereas implementing a Key-Value Store is orders of magnitude simpler. There is plenty of room for innovation, of course - but you don’t need 1,000 people to build-on some new replication scheme or distributed backend. If you take a simple NoSQL system then tack on things like object scheme support, JSON blob value indexes, node graph support, etc you’ll end up with an RDBMS analogue. So the barrier-to-entry to building a NoSQL system is much lower than a RDBMS - and for many companies they might decide to build their own rather than extend an existing system - leading to more completion but not necessarily a better product.
    - blihp 8y ago
      It always starts that way. In time, the legacy baggage that gets added will turn the noSQL databases into the same bloated monsters that the SQL databases have become. There was a time when many of the SQL database options were the (relatively) simpler and more powerful options vs the wide variety of home-grown databases they eventually replaced. (i.e. there was a lot of baked in business logic etc. that the move to SQL databases forced companies to disentangle from the database engine itself) SQL Server was originally just a fork of Sybase SQL Server (a much smaller, simpler version) with some FoxPro tech bolted on to it...
      - DaiPlusPlus 8y ago
        Right - and FoxPro was just another dBase clone. NoSQL today is where dBase was 30 years ago.
just_myles 8y ago
Agreed. I think for temporal data (accounting) reporting non-relational databases are fine.
- just_myles 8y ago
  To clarify further. Of course, you will need a relational data repository.
jefftk 8y ago
> when you need fully up-to-date information on every query, no matter what This doesn't require something relational, just something centralized. And not even that; consider Spanner.
amf12 8y ago
> namely when you need fully up-to-date information on every query, no matter what So you basically want full consistency (vs eventual consistency) in a distributed environment. You can set up you noSQL that way. I don't see how relational DB's are "better" than noSQL in consistency.
- ric2b 8y ago
  And if you do that you go back to the performance of one instance or (probably) worse.
  - amf12 8y ago
    Absolutely, but isn't that the case for SQL too? My point being, noSQL is not worse than SQL if all you care about is storing and retrieving documents vs joins, etc.
threeseed 8y ago
Why do people continue to use the term NoSQL ? It's ridiculous. There are hundreds maybe thousands of NoSQL databases ranging from Redis to MongoDB to Cassandra to InfluxDB to Druid. They pretty much have nothing in common other than storing data. Many of them support SQL directly, others via Spark and almost all have a SQL equivalent.
- endymi0n 8y ago
  Exactly this, „non-traditional“ and noSQL is as inexact as it gets. It includes an incredibly wide range of architectures, from Spanner and CockroachDB which deliver ACID guarantees equal and even surpassing relational systems to pure performance tuned systems such as Aerospike which have few guarantees and features but are blazingly fast. There are pretty much all shades of consistency in between, like Cassandra and Scylla with tunable consistency amd LWT. Lumping them all together into a single category will yield poor results.

emmelaich 8y ago

My hot take is that K/V stores solved a problem they didn't know they had. To whit, de-normalised tables for the sake of "efficiency". It's pretty difficult to under normalise a two column table -- which is what K/V is.

cronix 8y ago

> The non-traditional databases got better and better while Oracle kept doing a lot of the same stuff. Oracle just didn't take the competition seriously, and that's fair enough because the competition didn't deserve to be taken seriously at first -- but now it does, and it's becoming a threat to Oracle's business. Yes, like traditional car companies not taking EV's too seriously, or Intel not taking mobile/ARM too seriously, etc. It seems the older and more established a company is, the thicker their forward-thinking blinders are. "We're king, no one can EVER touch us."

__s 8y ago
Part of this is because successful companies have the survivorship bias of yesterday
rrcaptain 8y ago
What do you mean traditional car companies not taking EVERYTHING IS seriously? Volzwagen is switching all their cars to electric.
- gbear605 8y ago
  They could (should?) have been doing this a decade ago, but they didn’t. It took the relative success of Tesla to convince them.

jacques_chester 8y ago

This isn't really about relational vs non-relational. Redshift is still a relational database of the MPP genre.

hyperpallium 8y ago

"[N]on-traditional databases" with far better performance actually pre-date relational databases. Relational databases solved the problem they had, of requiring much developer effort to evolve for new application needs, and re-organization of the datastore for changing performance needs. Basically, everything is a table, and you can combine/separate tables, from whatever form it currently is, to whatever form you currently need. This is also helpful for reporting (as opposed to operational applications). But relational databases, like dynamic languages, pay a performance price for their flexibility. Of course, every other innovation has since been added, and there's Oracle's licensing too. So it's much more complex than the base technology of relational algebra.

mathattack 8y ago
Yes. Many people leave Oracle’s technology footprint out of frustration with their Salesforce.

jchanimal 8y ago

Shameless plug, but I think the new math that underlies global consistency is super interesting, I did a webcast with Professor Daniel Abadi about how Google Spanner differs from FaunaDB (my employer, which doesn’t require atomic clocks), you can get a link to the recording here https://www2.fauna.com/wcspannercalvin https://www2.fauna.com/wcspannercalvin

dgemm 8y ago

That is a hallmark of a disruptive technology, to not be as good at what the incumbent values but better in some other dimension that they don't.

twblalock 8y ago
The other interesting thing is that the latest database trends reflect the general trend in the industry away from monolithic solutions. Most of the new databases are not one-size-fits-all, but are optimized for narrower use cases.

grumpydba 8y ago

Most of those oracle databases are moving off to Aurora, which is pretty much traditional.

repolfx 8y ago
Traditional yes, and not even necessarily better. Amazon had a big outage in recent times that their own post-mortem linked to trying to move off Oracle to Aurora. Interestingly it was posted here to HN and repeatedly and immediately flagged off the page - it appears that some HN readers are very keen on suppressing anything that paints Oracle in a good light. It makes you wonder what other stories might be going missing that would alter people's impressions of this company.
- grumpydba 8y ago
  This article was essentially Oracle PR. Andy Pavlo, database professor at CMU, has debunked it and gave details on the journalist's dishonesty. Also, anything is better than oracle. If not for technical reasons, price wise it makes a lot of sense.
  - repolfx 8y ago
    That sounds completely delusional I'm afraid. The article(s) were summaries of an internal post-mortem written by one of Oracle's competitors - the exact opposite of Oracle PR. And how exactly is some random professor going to debunk the conclusions of Amazon's own staff about their own infrastructure? Your last sentence appears to be typical of the problem I'm describing.
    - grumpydba 8y ago
      First: Andy Pavlo was contacted by the author of the article. He's all but random, he's one of the references in the field. The outage was quite minor, not user visible and cost something around 100k. I've seen such outages after an Oracle version upgrade so it's all but major. For your information I've been managing critical databases for more than a decade so I know that it might look unusual to you, but it's quite clear to anyone barely knowledgeable about database management that the article really overblown the issue. It was also mocked by Amazon's CTO. So please, manage databases for a few years. Then I might hear your opinion and maybe consider it.

gaius 8y ago

The non-traditional databases got better and better while Oracle kept doing a lot of the same stuff I'm not sure this is true, you can look at the new features in 8i, 9i, 10g, 11g, 12c and see that Oracle is adding a lot of features, and some of them are very impressive. The problem is most people don't need most of them and if you want any of them you have to pay for all of them. And the "non-traditional" databases had a pretty low bar. MongoDB just had to stay up for more than 5 minutes and not lose your data. Do they even do that yet?

amf12 8y ago
> MongoDB just had to stay up for more than 5 minutes and not lose your data. Do they even do that yet? What are you implying? Do you have any source for that? Some of the top tech companies out there use non-traditional databases, including MongoDB. They won't if the nodes crash every 5 minutes and/or there is data loss.
- tluyben2 8y ago
  The parent was exaggerating but there are countless articles about how bad mongo is. These might be in the past, I do not know. Anecdotally I had horrible experiences with dataloss and instability, both of them incredibly hard to debug compared to say, MySql or Posgres. And when you see a hip company data leak it is often Mongo that was compromised (to be fair, usually because of it’s weak default settings and stupidity on the admin side, but still, it is a datapoint against mongo). Persobally I do not find it easier to use either: our ORM is a optionally schema’d layer; it works as flexible as Mongo and in naive setups, it is much much faster and easier to use (the latter is an opinion), without the risk of dataloss and with flexibility of db engine (it even supports Mongo because I was bored on a sunday).
  - grumpydba 8y ago
    Mongodb's storage engine is now wiredtiger. It is a bit young, but it's a really beautiful C codebase written by database veterans.

unscaled 8y ago

Of the three DB products mentioned in the tweet, only one is a non-traditional database: DynamoDB. Both Redshift and Aurora are traditional RDBMs, which are forked from very old codebases. Redshift is a fork of Postgres 8.0 that was adapted for OLAP use-cases, while Aurora is a fork of MySQL (or PostgreSQL in the PostgreSQL-compatible versions) that runs on a custom storage engine. The main reason technically-capable companies like Amazon or Netflix are moving away Oracle has nothing to do with horizontal scaling. It's the licensing costs. Oracle is just too expensive for them, and even if you are not Amazon yourself, RDS (or any other Cloud RDBMS) would be much cheaper for you.