3 ms·

The separation of storage and compute is controversial, though it is definitely convenient for cloud operators. It is literally going backward in terms of the a

The separation of storage and compute is controversial, though it is definitely convenient for cloud operators. It is literally going backward in terms of the available storage bandwidth, and many modern database architectures can very effectively use all the storage bandwidth available in modern systems. There is not enough network bandwidth, and won’t be for the foreseeable future, for running storage over networks to be efficient. Database workloads are also starting to move to edge environments, which make it even worse. The resource mismatch currently leaves a lot of compute sitting idle in practice. Another issue with this model is that it tends to offer poor ratios of bandwidth to storage size. This is at odds with the desire to run mixed workloads in a single system.

gavinray 4y ago
> The separation of storage and compute is controversial I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the recent RocksDB meetup on their vision for the next 3 years of RocksDB's future: https://twitter.com/GavinRayDev/status/1600666127025156096 https://twitter.com/GavinRayDev/status/1600666127025156096 Disaggregated storage is becoming quite a hot topic, and there are several technologies fueling this. There are two major advances happening in terms of disk I/O right now (that aren't to do with async IO uring stuff) 1. NVMe Zones + NVMe FDP specs, which are ways for userland applications to control placement of writes on drive (like a software Flash-Translation-Layer) 2. NVMe-over-Fabrics or similar technologies (with rdma/tcp) enables the access of a remote disk with 'comparable' latency to that of a local one. E.g. if local disk access is 100us, AWS can give you 110us with NVMe-OF. These two advances support the growing trend of disaggregated storage --- Full credit to Arun George from the Samsung Memory group for explaining this to me.
- jandrewrogers 4y ago
  The storage-over-fabric latency improvements allow it to work well for software that was not particularly optimized for storage performance, which has obvious benefits that I am not arguing. There is a good reason storage-over-fabric has been a popular database deployment model for decades, it does have advantages, but this emerging trend is nothing new nor are the issues that it has. For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot. To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.
  - gavinray 4y ago
    I didn't realize who I was talking to, just checked the username Would encourage other folks to do the same/read post history
    - sitkack 4y ago
      The truthiness of your comments shouldn't change upon who hears them?
      - gavinray 4y ago
        No, I still stand by my opinion that I don't find disaggregated storage controversial But if you're going to take an opinion on databases from someone, jandrewrogers is probably a good person to take them from. Me, on the other hand, I'm just a hobbyist/enthusiast, I've no accolades under my belt and my experience on the engineering end is shy of a year.
        sitkack 4y ago
        > controversial You and jandrewrogers are using different connotations for this word. Separating compute and storage makes sense when you have already done this like the hyperscalers have. It is good for them but not necessarily for the end user. That it is surprising to you are not doesn't say whether it is a good idea or not. If you make a statement regarding your own opinion vs an expert, then that statement is not referentially transparent, why? When disaggregated storage gets predicate pushdown will it still be disaggregated?
glogla 4y ago
The separation of compute and storage is most of all convenient for whoever operates the database. Need more compute, or scale it down when not in use? You can add it in minutes, no re-balancing, no careful moving of data. Need more storage? You have it on tap. Want to give some group of users their dedicated compute? Or run differently configured computes on top of one data, including even using different compute engines for batch and for interactive? You can. Compare Snowflake "create warehouse", which can give you compute that can immediately access any data, with a dance you need to do to resize Clickhouse and decide which tables are replicated in what way. At the same time, everything you say is true, and that is why system without storage-compute separation can give you order of magnitude more performance for the same money than a separated system can. If someone could solve the fragility and hassle of it, it would be quite a winner.
AdamProut 4y ago
There are hybrid designs for separation of storage and compute that are aimed at mixed workloads[1]. They avoid writes to remote storage on transaction commit (i.e., act like a shared nothing databases for commits, but still push data asynchronously to a shared remote disk that can be used for scale up/point in time restores/branching). [1] https://www.singlestore.com/blog/separating-storage-and-compute-for-transaction-and-analytics/ https://www.singlestore.com/blog/separating-storage-and-comp... (disclosure: SingleStoreDB CTO)
- nikita 4y ago
  Adam, you still synchronously replicate each log record to 2 places, right? Technically this should be roughly equivalent to writing into a consensus. And use write through cache for reads.
  - AdamProut 4y ago
    Yep, for writes network bandwidth usage is independent of separation of storage of compute in some sense. Any database that provides high availability is writing over the network somewhere before it acks a transaction committed. It matters who is on the other end of that network write though. Take the typical Cloud DW (i.e., Snowflake) design of forcing writes to the blob storage (shared remote disks) before commit. That is a much higher latency write then what a high performance transaction log replication protocol will do to replicate a write to another host.
- zX41ZdbW 4y ago
  SingleStore is a kind of uninteresting example because it is not open-source. It makes potential customers feel non-confident. Why move large data volumes to SingleStore if the company can cease to exist in a year?
  - AdamProut 4y ago
    Yep, SingleStoreDB (formerly MemSQL) is not open source (probably never will be), but it does have many paying customers who have had workloads in production for over a decade at this point. Also, isn't taking a bet on a very recently launched database as a service based around an open source database also pretty risky? Say for example the very recently launched Clickhouse Inc. service (which your a co-founder of?).
nikita 4y ago
> The resource mismatch currently leaves a lot of compute sitting idle in practice. When separate storage is multi tenant and compute is elastic you can really squeeze more efficiency. With that you can keep utilization extremely high, b/c storage is effectively spread across the whole fleet. We really see this now running 10K databases on a single (single per region really) storage deployment. Compute is also elastic, so you again can drive utilization up. So my argument is that separation of storage and compute is not necessarily for performance but rather for utilization, convenience to operate, and additional capabilities (branching, serverless).
- jandrewrogers 4y ago
  Sure, for aggregating many small databases this makes sense. Disaggregation of storage and compute has been popular at many times in database history, the tradeoffs are well-understood. Some databases are not particularly performance sensitive and it eliminates resource waste from resource overhang when databases only need fractions of server. For single large databases that occupy many servers, the economics tend to follow the size of the cluster which can be significantly smaller with direct-attached storage. Scaling storage independent of compute introduces some interesting edge cases operationally because they are intrinsically coupled to some extent — scaling one creates resource pressures and bottlenecks elsewhere. An alternative model to disaggregation that can produce a similar dynamic balancing of compute to storage without sacrificing bandwidth in scale-out systems is to use a dynamic mix of heterogeneous nodes. The storage is still tightly coupled to the compute but the ratio of compute to storage of the aggregate system can be quickly adjusted on the fly by adjusting the mix of server types. I haven’t seen a lot of work on heterogeneous cluster architectures in a long time — it was impractical when database clusters were capex — but it is a proven model albeit complex to implement, and easier to deploy today thanks to the cloud. Many of the emerging data problems at the edge look like heterogeneous cluster architecture problems too if you squint, so it likely has value beyond the data center. The edge has a lot of surprisingly-shaped problems that don’t fit any of our current tools e.g. some of the most extreme scale-up database problems I’ve seen in any context.
unusualmonkey 4y ago
I've used BigQuery extensively in the past. Have to say network bandwidth didn't seem a problem when querying terabytes of arbitrary data in seconds. What problems do you forsee?

gavinray 4y ago

> The separation of storage and compute is controversial I don't think it's a controversial topic, it's an emerging trend. See for example this slide from the recent RocksDB meetup on their vision for the next 3 years of RocksDB's future: https://twitter.com/GavinRayDev/status/1600666127025156096 https://twitter.com/GavinRayDev/status/1600666127025156096 Disaggregated storage is becoming quite a hot topic, and there are several technologies fueling this. There are two major advances happening in terms of disk I/O right now (that aren't to do with async IO uring stuff) 1. NVMe Zones + NVMe FDP specs, which are ways for userland applications to control placement of writes on drive (like a software Flash-Translation-Layer) 2. NVMe-over-Fabrics or similar technologies (with rdma/tcp) enables the access of a remote disk with 'comparable' latency to that of a local one. E.g. if local disk access is 100us, AWS can give you 110us with NVMe-OF. These two advances support the growing trend of disaggregated storage --- Full credit to Arun George from the Samsung Memory group for explaining this to me.

jandrewrogers 4y ago
The storage-over-fabric latency improvements allow it to work well for software that was not particularly optimized for storage performance, which has obvious benefits that I am not arguing. There is a good reason storage-over-fabric has been a popular database deployment model for decades, it does have advantages, but this emerging trend is nothing new nor are the issues that it has. For systems that are highly optimized for storage performance, storage bandwidth tends to be more important than storage latency and that is what is being sacrificed for the sake of disaggregating storage. I see integer factor reductions in workload throughput when deployed on disaggregated storage versus the usually NVMe JBOD, and that factor tracks real-world storage bandwidth pretty well, particularly at high storage densities. At scale, these economics start to matter quite a lot. To be clear, disaggregated storage has no effect on database design generally, seeing as how it has been widely used for decades. It does put an artificial floor on achievable throughput for many workloads due to the bandwidth bottleneck. When people realize this then the pendulum swings the other way (again), so I wouldn't want to overfit for disaggregated storage.
- gavinray 4y ago
  I didn't realize who I was talking to, just checked the username Would encourage other folks to do the same/read post history
  - sitkack 4y ago
    The truthiness of your comments shouldn't change upon who hears them?
    - gavinray 4y ago
      No, I still stand by my opinion that I don't find disaggregated storage controversial But if you're going to take an opinion on databases from someone, jandrewrogers is probably a good person to take them from. Me, on the other hand, I'm just a hobbyist/enthusiast, I've no accolades under my belt and my experience on the engineering end is shy of a year.
      - sitkack 4y ago
        > controversial You and jandrewrogers are using different connotations for this word. Separating compute and storage makes sense when you have already done this like the hyperscalers have. It is good for them but not necessarily for the end user. That it is surprising to you are not doesn't say whether it is a good idea or not. If you make a statement regarding your own opinion vs an expert, then that statement is not referentially transparent, why? When disaggregated storage gets predicate pushdown will it still be disaggregated?

glogla 4y ago

The separation of compute and storage is most of all convenient for whoever operates the database. Need more compute, or scale it down when not in use? You can add it in minutes, no re-balancing, no careful moving of data. Need more storage? You have it on tap. Want to give some group of users their dedicated compute? Or run differently configured computes on top of one data, including even using different compute engines for batch and for interactive? You can. Compare Snowflake "create warehouse", which can give you compute that can immediately access any data, with a dance you need to do to resize Clickhouse and decide which tables are replicated in what way. At the same time, everything you say is true, and that is why system without storage-compute separation can give you order of magnitude more performance for the same money than a separated system can. If someone could solve the fragility and hassle of it, it would be quite a winner.

AdamProut 4y ago

There are hybrid designs for separation of storage and compute that are aimed at mixed workloads[1]. They avoid writes to remote storage on transaction commit (i.e., act like a shared nothing databases for commits, but still push data asynchronously to a shared remote disk that can be used for scale up/point in time restores/branching). [1] https://www.singlestore.com/blog/separating-storage-and-compute-for-transaction-and-analytics/ https://www.singlestore.com/blog/separating-storage-and-comp... (disclosure: SingleStoreDB CTO)

nikita 4y ago
Adam, you still synchronously replicate each log record to 2 places, right? Technically this should be roughly equivalent to writing into a consensus. And use write through cache for reads.
- AdamProut 4y ago
  Yep, for writes network bandwidth usage is independent of separation of storage of compute in some sense. Any database that provides high availability is writing over the network somewhere before it acks a transaction committed. It matters who is on the other end of that network write though. Take the typical Cloud DW (i.e., Snowflake) design of forcing writes to the blob storage (shared remote disks) before commit. That is a much higher latency write then what a high performance transaction log replication protocol will do to replicate a write to another host.
zX41ZdbW 4y ago
SingleStore is a kind of uninteresting example because it is not open-source. It makes potential customers feel non-confident. Why move large data volumes to SingleStore if the company can cease to exist in a year?
- AdamProut 4y ago
  Yep, SingleStoreDB (formerly MemSQL) is not open source (probably never will be), but it does have many paying customers who have had workloads in production for over a decade at this point. Also, isn't taking a bet on a very recently launched database as a service based around an open source database also pretty risky? Say for example the very recently launched Clickhouse Inc. service (which your a co-founder of?).

nikita 4y ago

> The resource mismatch currently leaves a lot of compute sitting idle in practice. When separate storage is multi tenant and compute is elastic you can really squeeze more efficiency. With that you can keep utilization extremely high, b/c storage is effectively spread across the whole fleet. We really see this now running 10K databases on a single (single per region really) storage deployment. Compute is also elastic, so you again can drive utilization up. So my argument is that separation of storage and compute is not necessarily for performance but rather for utilization, convenience to operate, and additional capabilities (branching, serverless).

jandrewrogers 4y ago
Sure, for aggregating many small databases this makes sense. Disaggregation of storage and compute has been popular at many times in database history, the tradeoffs are well-understood. Some databases are not particularly performance sensitive and it eliminates resource waste from resource overhang when databases only need fractions of server. For single large databases that occupy many servers, the economics tend to follow the size of the cluster which can be significantly smaller with direct-attached storage. Scaling storage independent of compute introduces some interesting edge cases operationally because they are intrinsically coupled to some extent — scaling one creates resource pressures and bottlenecks elsewhere. An alternative model to disaggregation that can produce a similar dynamic balancing of compute to storage without sacrificing bandwidth in scale-out systems is to use a dynamic mix of heterogeneous nodes. The storage is still tightly coupled to the compute but the ratio of compute to storage of the aggregate system can be quickly adjusted on the fly by adjusting the mix of server types. I haven’t seen a lot of work on heterogeneous cluster architectures in a long time — it was impractical when database clusters were capex — but it is a proven model albeit complex to implement, and easier to deploy today thanks to the cloud. Many of the emerging data problems at the edge look like heterogeneous cluster architecture problems too if you squint, so it likely has value beyond the data center. The edge has a lot of surprisingly-shaped problems that don’t fit any of our current tools e.g. some of the most extreme scale-up database problems I’ve seen in any context.

unusualmonkey 4y ago

I've used BigQuery extensively in the past. Have to say network bandwidth didn't seem a problem when querying terabytes of arbitrary data in seconds. What problems do you forsee?