3 ms·

> The separation of storage and compute is controversial I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the

> The separation of storage and compute is controversial I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the recent RocksDB meetup on their vision for the next 3 years of RocksDB's future: https://twitter.com/GavinRayDev/status/1600666127025156096 https://twitter.com/GavinRayDev/status/1600666127025156096 Disaggregated storage is becoming quite a hot topic, and there are several technologies fueling this. There are two major advances happening in terms of disk I/O right now (that aren't to do with async IO uring stuff) 1. NVMe Zones + NVMe FDP specs, which are ways for userland applications to control placement of writes on drive (like a software Flash-Translation-Layer) 2. NVMe-over-Fabrics or similar technologies (with rdma/tcp) enables the access of a remote disk with 'comparable' latency to that of a local one. E.g. if local disk access is 100us, AWS can give you 110us with NVMe-OF. These two advances support the growing trend of disaggregated storage --- Full credit to Arun George from the Samsung Memory group for explaining this to me.

jandrewrogers 4y ago
The storage-over-fabric latency improvements allow it to work well for software that was not particularly optimized for storage performance, which has obvious benefits that I am not arguing. There is a good reason storage-over-fabric has been a popular database deployment model for decades, it does have advantages, but this emerging trend is nothing new nor are the issues that it has. For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot. To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.
- gavinray 4y ago
  I didn't realize who I was talking to, just checked the username Would encourage other folks to do the same/read post history
  - sitkack 4y ago
    The truthiness of your comments shouldn't change upon who hears them?
    - gavinray 4y ago
      No, I still stand by my opinion that I don't find disaggregated storage controversial But if you're going to take an opinion on databases from someone, jandrewrogers is probably a good person to take them from. Me, on the other hand, I'm just a hobbyist/enthusiast, I've no accolades under my belt and my experience on the engineering end is shy of a year.
      - sitkack 4y ago
        > controversial You and jandrewrogers are using different connotations for this word. Separating compute and storage makes sense when you have already done this like the hyperscalers have. It is good for them but not necessarily for the end user. That it is surprising to you are not doesn't say whether it is a good idea or not. If you make a statement regarding your own opinion vs an expert, then that statement is not referentially transparent, why? When disaggregated storage gets predicate pushdown will it still be disaggregated?

jandrewrogers 4y ago

The storage-over-fabric latency improvements allow it to work well for software that was not particularly optimized for storage performance, which has obvious benefits that I am not arguing. There is a good reason storage-over-fabric has been a popular database deployment model for decades, it does have advantages, but this emerging trend is nothing new nor are the issues that it has. For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot. To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.

gavinray 4y ago
I didn't realize who I was talking to, just checked the username Would encourage other folks to do the same/read post history
- sitkack 4y ago
  The truthiness of your comments shouldn't change upon who hears them?
  - gavinray 4y ago
    No, I still stand by my opinion that I don't find disaggregated storage controversial But if you're going to take an opinion on databases from someone, jandrewrogers is probably a good person to take them from. Me, on the other hand, I'm just a hobbyist/enthusiast, I've no accolades under my belt and my experience on the engineering end is shy of a year.
    - sitkack 4y ago
      > controversial You and jandrewrogers are using different connotations for this word. Separating compute and storage makes sense when you have already done this like the hyperscalers have. It is good for them but not necessarily for the end user. That it is surprising to you are not doesn't say whether it is a good idea or not. If you make a statement regarding your own opinion vs an expert, then that statement is not referentially transparent, why? When disaggregated storage gets predicate pushdown will it still be disaggregated?