4 ms·

Sparse block storage is one of the cloud features I miss most. What I'd expect: * You can create a volume, but pay only for in-use blocks. * You create a volu

Sparse block storage is one of the cloud features I miss most. What I'd expect: * You can create a volume, but pay only for in-use blocks. * You create a volume bigger than what you'll ever need => no resizing required (At least 16 TiB, but possibly even something crazy like an exabyte) * There is some kind of "trim" operation that marks a block as free * Ideally it's possible to choose the maximum block index (size of block-volume) independently from the maximum number of in-use blocks (for cost control)

lifty 4y ago
Great feature, but cloud providers are incentivised to make it difficult for people to save money. This would instantly vaporise a big chunk of revenue, because people would not have to pay for pre-provisioned storage.
- advisedwang 4y ago
  Maybe one of the lower tier providers should use it as a differentiator.
  - djbusby 4y ago
    Why add a low margin feature to your product/service?
    - bo0tzz 4y ago
      Because it draws in customers that will then also use your higher margin features.
    - discodave 4y ago
      Amazon historically has gone after 'low margin' businesses. The cloud, at 25% margins (for AWS) is is 'low margin' compared to traditional software sales.
- pgwhalen 4y ago
  > Great feature, but cloud providers are incentivised to make it difficult for people to save money. Every business is incentivized this way. Competition is what pushes prices down, of which there is plenty in the cloud industry (though maybe still not as much as we would like).
- thinkmassive 4y ago
  AWS EBS volumes used by a PVC can be expanded without downtime by increasing the request on the PVC. The only prerequisite is that the StorageClass has allowVolumeExpansion=true Edit: Now that I actually read TFA I see it mentions most CSI drivers already provide this functionality. They provide a workaround similar to what I’m sure most people use, and I agree this functionality seems like it could be handled by StatefulSet.
- waynesonfire 4y ago
  Yep, the cloud long term is a terrible investment. Due to its sticky nature innovation in technology is seldom passed down to consumers and instead is pocketed by the provider. I guarantee you internally cloud providers over provision their storage.
dpedu 4y ago
It's not at the block level, but Amazon's EFS checks most of these boxes.
- chrsig 4y ago
  i read a story on HN recently about someone having left a `nohup yes > /some/efs/mount/file.txt` type command running, and not noticing until the bill came due... since then, I'm totally on board with pre-provisioned storage. at the very least, some way to put an upper bound on volume size (maybe efs has that? not sure.)
  - switch007 4y ago
    Oh wow that's horrific. That will write gigabytes in a few seconds on an SSD. Unsurprisingly, AWS EFS does not support NFS quotas.
  - kqr 4y ago
    For production software, you should always have an upper limit on everything. There is always a point where you can tell in advance that it doesn't make sense to keep going. It is never sensible to literally go on forever with anything -- it can only break stuff in annoying ways when it runs into real world limitations (like financial ones in that case.) I sound aggressive about this because it's such a common mistake. It always goes something like, "Why does this list have to be unbounded?" "Well, we don't want to give the user an error because it's full." "Okay, but does it really need to support 35,246,953 instances?" "Sure, why not?" "How long would the main interaction with the system take if you stress it to that level?" "Oh, I don't know, at that level it might well take 20 minutes." "And the clients usually timeout after?..." "5 seconds." "Would the user rather wait for 20 minutes and then get a response that might be outdated by that time, or get an error right away?" "They may well prefer the error at that point." "So let's go backwards from that. Will it ever make sense to support more than 250,000 instances?" "That corresponds to the five second timeout and then some. I guess that's fine in practise..." It's not that hard!
    - chrsig 4y ago
dmitryminkovsky 4y ago
Where is such block storage available? Seems fantastical...?
- Nullabillity 4y ago
  Ceph's RBD basically works like this (but it's a self-hosted software package, not IaaS), so you still need to manage the storage pool yourself.
- topspin 4y ago
  "Where is such block storage available? Seems fantastical...?" SAN vendors have been selling devices that deliver thin provisioned network block devices you can snapshot, clone, resize dynamically and replicate in a cluster for about 20 years now. Third parties can do all of this in cloud environments as well. See Netapp Cloud Volumes. You could roll your own using Ceph, or even a ghetto version with a subset of these capabilities with thin provisioned LVM+iSCSI, DRBD etc. You should be able to point and click a thin provisioned, dynamically expandable, snapshot-able, clone-able, replicate-able pay as you go network block device with whatever degree of performance, availability and capacity you wish to pay for. The fact that you can't is a function of the business model and mentality of IaaS cloud operators; they don't want your old fashioned 'stateful' workloads. Neither does kubernetes. So you're swimming against the current.
AtlasBarfed 4y ago
AWS has elastic filesystem, but the perf is crap from what I understand. I haven't used it, so your mileage is a total unknown, but our "big data" folks like it. AWS won't change to your desired model, they mint money on underutilized resources (and hidden I/O charges/network)
benlivengood 4y ago
Sparse block storage is achievable with s3backer and similar tools which treat bucket/object storage as arbitrarily large volumes. Practically, sizing thin-provisioned filesystems too large wastes a lot of space on filesystem metadata. All operating systems and clouds that I've worked with support online volume and filesystem resizing which is as close to the semantics of thin-provisioning as I've ever needed. I wasn't actually aware that StatefulSets don't resize volumes as smoothly as desired but I haven't had a problem manually deleting pods to get the filesystem on a PVC to grow (the way it works for Deployments).

lifty 4y ago

Great feature, but cloud providers are incentivised to make it difficult for people to save money. This would instantly vaporise a big chunk of revenue, because people would not have to pay for pre-provisioned storage.

advisedwang 4y ago
Maybe one of the lower tier providers should use it as a differentiator.
- djbusby 4y ago
  Why add a low margin feature to your product/service?
  - bo0tzz 4y ago
    Because it draws in customers that will then also use your higher margin features.
  - discodave 4y ago
    Amazon historically has gone after 'low margin' businesses. The cloud, at 25% margins (for AWS) is is 'low margin' compared to traditional software sales.
pgwhalen 4y ago
> Great feature, but cloud providers are incentivised to make it difficult for people to save money. Every business is incentivized this way. Competition is what pushes prices down, of which there is plenty in the cloud industry (though maybe still not as much as we would like).
thinkmassive 4y ago
AWS EBS volumes used by a PVC can be expanded without downtime by increasing the request on the PVC. The only prerequisite is that the StorageClass has allowVolumeExpansion=true Edit: Now that I actually read TFA I see it mentions most CSI drivers already provide this functionality. They provide a workaround similar to what I’m sure most people use, and I agree this functionality seems like it could be handled by StatefulSet.
waynesonfire 4y ago
Yep, the cloud long term is a terrible investment. Due to its sticky nature innovation in technology is seldom passed down to consumers and instead is pocketed by the provider. I guarantee you internally cloud providers over provision their storage.

dpedu 4y ago

It's not at the block level, but Amazon's EFS checks most of these boxes.

chrsig 4y ago
i read a story on HN recently about someone having left a `nohup yes > /some/efs/mount/file.txt` type command running, and not noticing until the bill came due... since then, I'm totally on board with pre-provisioned storage. at the very least, some way to put an upper bound on volume size (maybe efs has that? not sure.)
- switch007 4y ago
  Oh wow that's horrific. That will write gigabytes in a few seconds on an SSD. Unsurprisingly, AWS EFS does not support NFS quotas.
- kqr 4y ago
  For production software, you should always have an upper limit on everything. There is always a point where you can tell in advance that it doesn't make sense to keep going. It is never sensible to literally go on forever with anything -- it can only break stuff in annoying ways when it runs into real world limitations (like financial ones in that case.) I sound aggressive about this because it's such a common mistake. It always goes something like, "Why does this list have to be unbounded?" "Well, we don't want to give the user an error because it's full." "Okay, but does it really need to support 35,246,953 instances?" "Sure, why not?" "How long would the main interaction with the system take if you stress it to that level?" "Oh, I don't know, at that level it might well take 20 minutes." "And the clients usually timeout after?..." "5 seconds." "Would the user rather wait for 20 minutes and then get a response that might be outdated by that time, or get an error right away?" "They may well prefer the error at that point." "So let's go backwards from that. Will it ever make sense to support more than 250,000 instances?" "That corresponds to the five second timeout and then some. I guess that's fine in practise..." It's not that hard!
  - chrsig 4y ago

dmitryminkovsky 4y ago

Where is such block storage available? Seems fantastical...?

Nullabillity 4y ago
Ceph's RBD basically works like this (but it's a self-hosted software package, not IaaS), so you still need to manage the storage pool yourself.
topspin 4y ago
"Where is such block storage available? Seems fantastical...?" SAN vendors have been selling devices that deliver thin provisioned network block devices you can snapshot, clone, resize dynamically and replicate in a cluster for about 20 years now. Third parties can do all of this in cloud environments as well. See Netapp Cloud Volumes. You could roll your own using Ceph, or even a ghetto version with a subset of these capabilities with thin provisioned LVM+iSCSI, DRBD etc. You should be able to point and click a thin provisioned, dynamically expandable, snapshot-able, clone-able, replicate-able pay as you go network block device with whatever degree of performance, availability and capacity you wish to pay for. The fact that you can't is a function of the business model and mentality of IaaS cloud operators; they don't want your old fashioned 'stateful' workloads. Neither does kubernetes. So you're swimming against the current.

AtlasBarfed 4y ago

AWS has elastic filesystem, but the perf is crap from what I understand. I haven't used it, so your mileage is a total unknown, but our "big data" folks like it. AWS won't change to your desired model, they mint money on underutilized resources (and hidden I/O charges/network)

benlivengood 4y ago

Sparse block storage is achievable with s3backer and similar tools which treat bucket/object storage as arbitrarily large volumes. Practically, sizing thin-provisioned filesystems too large wastes a lot of space on filesystem metadata. All operating systems and clouds that I've worked with support online volume and filesystem resizing which is as close to the semantics of thin-provisioning as I've ever needed. I wasn't actually aware that StatefulSets don't resize volumes as smoothly as desired but I haven't had a problem manually deleting pods to get the filesystem on a PVC to grow (the way it works for Deployments).