6 ms·

what I fail to fathom today is, when I think back to the late 90s, I had various production rdms, SQL Server 6 (or 7, can't remember), running on something like

what I fail to fathom today is, when I think back to the late 90s, I had various production rdms, SQL Server 6 (or 7, can't remember), running on something like a Pentium 4, with millions of rows of data in various tables... and the performance was... fine. not an issue.

foota 5y ago
Single machine postgres is mostly fine when you're working with less than hundreds of million of rows, less than a few hundred gigs of data, don't need HA, and don't need more than a few thousand QPS. Outside of those constraints is why new things exist imo.
- crooked-v 5y ago
  Of course, the overwhelming majority of all businesses (including web-centric businesses) work fine within those constraints.
  - foota 5y ago
    Sure, small businesses are fine using postgres. I would argue that by developer market share though most (or perhaps just many) developers have to worry about these issues, since the companies that have scale have to higher many more developers. There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.
    - jmchuster 5y ago
      Computers are very fast and fast storage is very cheap these days. Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
      - jjav 5y ago
        > Your Data Isn't That Big This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies. Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it". I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to). A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.
      - foota 5y ago
        "But my data is more than 5TB! Your life now sucks - you are stuck with Hadoop. You don't have many other choices (big servers with many hard drives might still be in play), and most of your other choices are considerably more expensive." My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US. [1]: https://www.daxx.com/blog/development-trends/number-software-developers-world https://www.daxx.com/blog/development-trends/number-software...
        tomnipotent 5y ago
        Even combined, FAANG is probably only around 150k programmers. A vast majority of the worlds 24-26M working software programmers do not work at them, or even at the top 100 such companies.
- deleted 5y ago
  [deleted]
- jeltz 5y ago
  PostgreSQL handles 100k QPS without any issues unless your queries are very complex and I know many people who run multi-TB databases in PostgreSQL without any issues either. But it all depends on your workload of course.
  - foota 5y ago
    Yeah, I probably could have bumped both those numbers an order of magnitude.
dikei 5y ago
Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows. Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.
- da39a3ee 5y ago
  > but the data generated today is mind-boggling large compared to the 90s. For what percentage of companies would you say your statement is true?
- nine_k 5y ago
  If you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF. In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS. It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
  - dikei 5y ago
    Yup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs. My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
- smallerfish 5y ago
  :s/these days/at your particular job/g
jjav 5y ago
Today you need a Cassandra cluster fronted by a Kubernetes cluster to get similar performance. The good news is that you don't need a couple boxes under your desk, you can pay AWS tens of thousands a month for it.
jeltz 5y ago
It is not and issue today either. Modern RDBMs have become better optimized and hardware has become faster so a lot of NoSQL is premature optimization where people optimize for having 100+ TB databases when their database actually is 100 GB.

foota 5y ago

Single machine postgres is mostly fine when you're working with less than hundreds of million of rows, less than a few hundred gigs of data, don't need HA, and don't need more than a few thousand QPS. Outside of those constraints is why new things exist imo.

crooked-v 5y ago
Of course, the overwhelming majority of all businesses (including web-centric businesses) work fine within those constraints.
- foota 5y ago
  Sure, small businesses are fine using postgres. I would argue that by developer market share though most (or perhaps just many) developers have to worry about these issues, since the companies that have scale have to higher many more developers. There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.
  - jmchuster 5y ago
    Computers are very fast and fast storage is very cheap these days. Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
    - jjav 5y ago
      > Your Data Isn't That Big This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies. Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it". I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to). A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.
    - foota 5y ago
      "But my data is more than 5TB! Your life now sucks - you are stuck with Hadoop. You don't have many other choices (big servers with many hard drives might still be in play), and most of your other choices are considerably more expensive." My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US. [1]: https://www.daxx.com/blog/development-trends/number-software-developers-world https://www.daxx.com/blog/development-trends/number-software...
      - tomnipotent 5y ago
        Even combined, FAANG is probably only around 150k programmers. A vast majority of the worlds 24-26M working software programmers do not work at them, or even at the top 100 such companies.
deleted 5y ago
[deleted]
jeltz 5y ago
PostgreSQL handles 100k QPS without any issues unless your queries are very complex and I know many people who run multi-TB databases in PostgreSQL without any issues either. But it all depends on your workload of course.
- foota 5y ago
  Yeah, I probably could have bumped both those numbers an order of magnitude.

dikei 5y ago

Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows. Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.

da39a3ee 5y ago
> but the data generated today is mind-boggling large compared to the 90s. For what percentage of companies would you say your statement is true?
nine_k 5y ago
If you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF. In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS. It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.
- dikei 5y ago
  Yup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs. My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.
smallerfish 5y ago
:s/these days/at your particular job/g

jjav 5y ago

Today you need a Cassandra cluster fronted by a Kubernetes cluster to get similar performance. The good news is that you don't need a couple boxes under your desk, you can pay AWS tens of thousands a month for it.

jeltz 5y ago

It is not and issue today either. Modern RDBMs have become better optimized and hardware has become faster so a lot of NoSQL is premature optimization where people optimize for having 100+ TB databases when their database actually is 100 GB.