4 ms·

Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe

Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe 4.0 x4 SSD on my machine can do large I/Os at 6.8 GB/s rate, so 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. You may need clusters, duplicated systems, replication, etc for resiliency reasons of course, but a single modern machine with lots of memory channels per CPU and PCIe 4.0 can achieve ridiculous throughput... edit: Here's an example of doing 11M IOPS with 10x Samsung Pro 980 PCIe 4.0 SSDs (it's from an upcoming blog entry): https://twitter.com/TanelPoder/status/1352329243070504964 https://twitter.com/TanelPoder/status/1352329243070504964

riku_iki 6y ago
> 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. It is more interesting if actual CPU can handle such traffic in context of DB load: encode/decide records, sort, search, merge etc.
- tanelpoder 6y ago
  Yes, with modern storage, throughput is a CPU problem. And CPU problem for OLTP databases is largely a memory access latency problem. For columnar analytics & complex calculations it's more about CPU itself. When doing 1 MB sized I/Os for scanning, my 16c/32t (AMD Ryzen Threadripper Pro WX) CPUs were just about 10% busy. So, with a 64 core single socket ThreadRipper workstation (or 128-core dual socket EPYC server), there should be plenty of horsepower left.
  - tanelpoder 6y ago
    As I mentioned memory access latency - I just posted my old article series about measuring RAM access performance (using different database workloads) to HN and looks like it even made it to the front page (nice): https://news.ycombinator.com/item?id=25863093 https://news.ycombinator.com/item?id=25863093
    - gpsar 6y ago
      If the problem involves independent traversals, interleaving with coroutines is a practical way to hide latency https://dl.acm.org/doi/10.1145/3329785.3329917 https://dl.acm.org/doi/10.1145/3329785.3329917 https://www.linkedin.com/pulse/dont-stall-multitask-georgios-psaropoulos https://www.linkedin.com/pulse/dont-stall-multitask-georgios...
      - PeterCorless 6y ago
        We've made Scylla as async, shared nothing as possible, and we've also started adding C++20 coroutines (to replace futures/promises). We'll be doing more of that in 2021. https://www.scylladb.com/2021/01/12/making-scylla-a-monstrous-database-introducing-project-circe/ https://www.scylladb.com/2021/01/12/making-scylla-a-monstrou...
        tanelpoder 6y ago
        I plan to test Scylla (and Postgres with TimescaleDB) out soon after done with basic Linux tests :)
        ddorian43 6y ago
        Just note that TimescaleDB is olap (columnar compression), scylladb is oltp with async-fsync and Postgresql is disk-fsync-on by default.
jdsully 6y ago
It can burst to millions of IOPS, but you get killed on the sustained write workload. Even a high end enterprise NVMe drive will be limited to around 60k IOPs once you exceed its write cache.
- tanelpoder 6y ago
  Yup indeed it's an issue with NAND SSDs - and it heavily depends on the type (SLC, MLC, TLC, QLC), vendor (controller, memory, "write buffer") sizes etc. I'm doing mostly read tests right now and will move to writes after. The Samsung PRO 980 I have, are TLC for main storage, but apparently are using some of that TLC storage as a faster write buffer (TurboWrite buffer) - I'm not an expert, but apparently the controller can decide to program the TLC NAND with only 1-bit "depth", they call it "simulated SLC" or something like that. On the 1 TB SSD, the turbowrite buffer can dynamically extend to ~100 GB, if there's unused NAND space on the disk. Btw, the 3DXpoint storage (Intel Optane SSDs & Micron X1) should be able to sustain crazy write rates too.
zerd 6y ago
My mind exploded when realizing we can read random IO from disk at 40GB/s, which is faster than my laptop can read from RAM. https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ Btw. using SPDK or io_uring?

riku_iki 6y ago

> 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. It is more interesting if actual CPU can handle such traffic in context of DB load: encode/decide records, sort, search, merge etc.

tanelpoder 6y ago
Yes, with modern storage, throughput is a CPU problem. And CPU problem for OLTP databases is largely a memory access latency problem. For columnar analytics & complex calculations it's more about CPU itself. When doing 1 MB sized I/Os for scanning, my 16c/32t (AMD Ryzen Threadripper Pro WX) CPUs were just about 10% busy. So, with a 64 core single socket ThreadRipper workstation (or 128-core dual socket EPYC server), there should be plenty of horsepower left.
- tanelpoder 6y ago
  As I mentioned memory access latency - I just posted my old article series about measuring RAM access performance (using different database workloads) to HN and looks like it even made it to the front page (nice): https://news.ycombinator.com/item?id=25863093 https://news.ycombinator.com/item?id=25863093
  - gpsar 6y ago
    If the problem involves independent traversals, interleaving with coroutines is a practical way to hide latency https://dl.acm.org/doi/10.1145/3329785.3329917 https://dl.acm.org/doi/10.1145/3329785.3329917 https://www.linkedin.com/pulse/dont-stall-multitask-georgios-psaropoulos https://www.linkedin.com/pulse/dont-stall-multitask-georgios...
    - PeterCorless 6y ago
      We've made Scylla as async, shared nothing as possible, and we've also started adding C++20 coroutines (to replace futures/promises). We'll be doing more of that in 2021. https://www.scylladb.com/2021/01/12/making-scylla-a-monstrous-database-introducing-project-circe/ https://www.scylladb.com/2021/01/12/making-scylla-a-monstrou...
      - tanelpoder 6y ago
        I plan to test Scylla (and Postgres with TimescaleDB) out soon after done with basic Linux tests :)
        ddorian43 6y ago
        Just note that TimescaleDB is olap (columnar compression), scylladb is oltp with async-fsync and Postgresql is disk-fsync-on by default.

jdsully 6y ago

It can burst to millions of IOPS, but you get killed on the sustained write workload. Even a high end enterprise NVMe drive will be limited to around 60k IOPs once you exceed its write cache.

tanelpoder 6y ago
Yup indeed it's an issue with NAND SSDs - and it heavily depends on the type (SLC, MLC, TLC, QLC), vendor (controller, memory, "write buffer") sizes etc. I'm doing mostly read tests right now and will move to writes after. The Samsung PRO 980 I have, are TLC for main storage, but apparently are using some of that TLC storage as a faster write buffer (TurboWrite buffer) - I'm not an expert, but apparently the controller can decide to program the TLC NAND with only 1-bit "depth", they call it "simulated SLC" or something like that. On the 1 TB SSD, the turbowrite buffer can dynamically extend to ~100 GB, if there's unused NAND space on the disk. Btw, the 3DXpoint storage (Intel Optane SSDs & Micron X1) should be able to sustain crazy write rates too.

zerd 6y ago

My mind exploded when realizing we can read random IO from disk at 40GB/s, which is faster than my laptop can read from RAM. https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ Btw. using SPDK or io_uring?