6 ms·

Actually they're even older. When I talked to my dad about RDBMS he was like "weeeelll, sure, there's things like DB2 UDB that do relational but performance wi

Actually they're even older. When I talked to my dad about RDBMS he was like "weeeelll, sure, there's things like DB2 UDB that do relational but performance wise, nothing beats reading the data straight by key in exactly the format you need.". DB2 UDB: "Initial release: 1987; 34 years ago". I.e. what he would rather use and is sort of the NoSQL equivalent is _even older than that_. Background: he started off with 360 assembler and worked all his working life on IBM Mainframes and the various technologies in and around it. They had it all and they had it before it came to "us". We're just re-inventing most of these things on much cheaper and more open hardware and software.

xarope 5y ago
I worked on DB2 UDB in the 90s, and learnt a lot about DBs through all the improvements in isolation levels. Those were the days when you'd actually have to make some judgement calls about setting specific isolation levels to improve performance. Nowadays you just chuck an SSD and several TBs of memory into the works (or talk to someone about some massively horizontally scaled solution, and distract from the real problem at hand).
dd444fgdfg 5y ago
what I fail to fathom today is, when I think back to the late 90s, I had various production rdms, SQL Server 6 (or 7, can't remember), running on something like a Pentium 4, with millions of rows of data in various tables... and the performance was... fine. not an issue.
- foota 5y ago
  Single machine postgres is mostly fine when you're working with less than hundreds of million of rows, less than a few hundred gigs of data, don't need HA, and don't need more than a few thousand QPS. Outside of those constraints is why new things exist imo.
  - crooked-v 5y ago
    Of course, the overwhelming majority of all businesses (including web-centric businesses) work fine within those constraints.
    - foota 5y ago
      Sure, small businesses are fine using postgres. I would argue that by developer market share though most (or perhaps just many) developers have to worry about these issues, since the companies that have scale have to higher many more developers. There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.
      - jmchuster 5y ago
        Computers are very fast and fast storage is very cheap these days. Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
        jjav 5y ago
        > Your Data Isn't That Big This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies. Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it". I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to). A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.
        foota 5y ago
        "But my data is more than 5TB! Your life now sucks - you are stuck with Hadoop. You don't have many other choices (big servers with many hard drives might still be in play), and most of your other choices are considerably more expensive." My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US. [1]: https://www.daxx.com/blog/development-trends/number-software-developers-world https://www.daxx.com/blog/development-trends/number-software...
- dikei 5y ago
  Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows. Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.
  - da39a3ee 5y ago
    > but the data generated today is mind-boggling large compared to the 90s. For what percentage of companies would you say your statement is true?
  - nine_k 5y ago
    If you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF. In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS. It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
    - dikei 5y ago
      Yup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs. My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
  - smallerfish 5y ago
    :s/these days/at your particular job/g
- jjav 5y ago
  Today you need a Cassandra cluster fronted by a Kubernetes cluster to get similar performance. The good news is that you don't need a couple boxes under your desk, you can pay AWS tens of thousands a month for it.
- jeltz 5y ago
  It is not and issue today either. Modern RDBMs have become better optimized and hardware has become faster so a lot of NoSQL is premature optimization where people optimize for having 100+ TB databases when their database actually is 100 GB.
Clubber 5y ago
Oracle introduced RDBMS in 1979 based off Codd's paper in 1969. Mainframes were all flat file not object-relational, but you could finagle them to be sort of that. Object-Relational didn't come into play until the 90s. The main issue with object relational as the "main" database is its hard to run reports on. Reports (for biz) is half the reason the system exists. I'm not saying ORDBMS don't have a place, but RDBMS is much better suited for real-time data analysis / reports. I think ORDBMS is popular because it's easy for non-database people (who don't know 3rd normal form or SQL) to store stuff in.
- inopinatus 5y ago
  Let's not erase the work of Stonebraker et al whose efforts through the 70s were a lot closer to Codd than Oracle's. Ingres's query language, Quel, springs to mind: being directly based on the relational calculus it was far more readily composable than SQL (which might be charitably described as "borrowing ideas from the relational model") was or is.
- throwaway81523 5y ago
  You may be overlooking System R from 1974: https://en.wikipedia.org/wiki/IBM_System_R https://en.wikipedia.org/wiki/IBM_System_R
eru 5y ago
Well, even plain old filesystems are some weird hybrid between key-value stores and graph data bases, too.
- inopinatus 5y ago
  it's sometimes very helpful to think of filesystems as a streamable blob store with a hierarchical key
  - eru 5y ago
    That's one valid perspective. But don't forget that you can also seek and mmap etc. And hard-links are a thing, too. Things like fuse also show that it's easy to view other databases as filesystems, too. As a silly example, you can 'explode' /etc/passwd into individual files per entry. Or you could expose even a relational database as a filesystem. For the latter, you could do something simple and expose tables as directories and rows as files. Or you could do something dynamic, and eg represent (read-only or idempotent) queries as file-names, and their results as file contents. That wouldn't be so much a traditional filesystem, but would be accessible via a filesystem interface.
  - tharkun__ 5y ago
    I hear AWS DynamoDB is basically just a "key value store interface" to S3 file storage.
tootie 5y ago
The original Unix DBM file was basically NoSQL in a SQLite type package and it was developed in 1979. https://en.wikipedia.org/wiki/DBM_(computing) https://en.wikipedia.org/wiki/DBM_(computing)

xarope 5y ago

I worked on DB2 UDB in the 90s, and learnt a lot about DBs through all the improvements in isolation levels. Those were the days when you'd actually have to make some judgement calls about setting specific isolation levels to improve performance. Nowadays you just chuck an SSD and several TBs of memory into the works (or talk to someone about some massively horizontally scaled solution, and distract from the real problem at hand).

dd444fgdfg 5y ago

what I fail to fathom today is, when I think back to the late 90s, I had various production rdms, SQL Server 6 (or 7, can't remember), running on something like a Pentium 4, with millions of rows of data in various tables... and the performance was... fine. not an issue.

foota 5y ago
Single machine postgres is mostly fine when you're working with less than hundreds of million of rows, less than a few hundred gigs of data, don't need HA, and don't need more than a few thousand QPS. Outside of those constraints is why new things exist imo.
- crooked-v 5y ago
  Of course, the overwhelming majority of all businesses (including web-centric businesses) work fine within those constraints.
  - foota 5y ago
    Sure, small businesses are fine using postgres. I would argue that by developer market share though most (or perhaps just many) developers have to worry about these issues, since the companies that have scale have to higher many more developers. There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.
    - jmchuster 5y ago
      Computers are very fast and fast storage is very cheap these days. Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
      - jjav 5y ago
        > Your Data Isn't That Big This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies. Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it". I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to). A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.
      - foota 5y ago
        "But my data is more than 5TB! Your life now sucks - you are stuck with Hadoop. You don't have many other choices (big servers with many hard drives might still be in play), and most of your other choices are considerably more expensive." My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US. [1]: https://www.daxx.com/blog/development-trends/number-software-developers-world https://www.daxx.com/blog/development-trends/number-software...
dikei 5y ago
Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows. Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.
- da39a3ee 5y ago
  > but the data generated today is mind-boggling large compared to the 90s. For what percentage of companies would you say your statement is true?
- nine_k 5y ago
  If you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF. In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS. It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
  - dikei 5y ago
    Yup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs. My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
- smallerfish 5y ago
  :s/these days/at your particular job/g
jjav 5y ago
Today you need a Cassandra cluster fronted by a Kubernetes cluster to get similar performance. The good news is that you don't need a couple boxes under your desk, you can pay AWS tens of thousands a month for it.
jeltz 5y ago
It is not and issue today either. Modern RDBMs have become better optimized and hardware has become faster so a lot of NoSQL is premature optimization where people optimize for having 100+ TB databases when their database actually is 100 GB.

Clubber 5y ago

Oracle introduced RDBMS in 1979 based off Codd's paper in 1969. Mainframes were all flat file not object-relational, but you could finagle them to be sort of that. Object-Relational didn't come into play until the 90s. The main issue with object relational as the "main" database is its hard to run reports on. Reports (for biz) is half the reason the system exists. I'm not saying ORDBMS don't have a place, but RDBMS is much better suited for real-time data analysis / reports. I think ORDBMS is popular because it's easy for non-database people (who don't know 3rd normal form or SQL) to store stuff in.

inopinatus 5y ago
Let's not erase the work of Stonebraker et al whose efforts through the 70s were a lot closer to Codd than Oracle's. Ingres's query language, Quel, springs to mind: being directly based on the relational calculus it was far more readily composable than SQL (which might be charitably described as "borrowing ideas from the relational model") was or is.
throwaway81523 5y ago
You may be overlooking System R from 1974: https://en.wikipedia.org/wiki/IBM_System_R https://en.wikipedia.org/wiki/IBM_System_R

eru 5y ago

Well, even plain old filesystems are some weird hybrid between key-value stores and graph data bases, too.

inopinatus 5y ago
it's sometimes very helpful to think of filesystems as a streamable blob store with a hierarchical key
- eru 5y ago
  That's one valid perspective. But don't forget that you can also seek and mmap etc. And hard-links are a thing, too. Things like fuse also show that it's easy to view other databases as filesystems, too. As a silly example, you can 'explode' /etc/passwd into individual files per entry. Or you could expose even a relational database as a filesystem. For the latter, you could do something simple and expose tables as directories and rows as files. Or you could do something dynamic, and eg represent (read-only or idempotent) queries as file-names, and their results as file contents. That wouldn't be so much a traditional filesystem, but would be accessible via a filesystem interface.
- tharkun__ 5y ago
  I hear AWS DynamoDB is basically just a "key value store interface" to S3 file storage.

tootie 5y ago

The original Unix DBM file was basically NoSQL in a SQLite type package and it was developed in 1979. https://en.wikipedia.org/wiki/DBM_(computing) https://en.wikipedia.org/wiki/DBM_(computing)