3 ms·

EFS is a great addition to AWS. We have SAN as a service via EBS, now we get NFS as a service. Great. The question (for me) now becomes "where do we go from he

EFS is a great addition to AWS. We have SAN as a service via EBS, now we get NFS as a service. Great. The question (for me) now becomes "where do we go from here?" Infinite NFS is great, but what I've always wanted is infinite EBS that is fully integrated from file system to SAN. In other words, something that behaves like a local file system (without the gotchas of NFS like a lack of delete on close), but I don't have to snapshot and create new volumes and issue file system expansion commands to grow a volume. I want seamless and automatic growth. Furthermore, there's so much local SSD just sitting around when using EBS. I want to make full use of local SSD inside of an EC2 instance to do write-back or write-through caching. I could do this in software, but maybe there's an abstraction begging to be made at the service level. Throw in things like snapshots, and this would make for a fairly powerful solution, and it would certainly remove a lot of operational concerns around growing database nodes and such. Don't get me wrong, you can pull together a few things and write some automation to do this today. You could use LVM to stitch together many EBS volumes, add in caching middleware (dm-cache, flashcache, etc.), and then automate the addition of volumes and file system growth. However, it's clunky, and there's an opportunity to make this much easier. I recognize that what I'm describing doesn't serve the same purpose as NFS - for example, EBS isn't mountable in multiple locations at once - but I'd really like to see the "seamless infinite storage" idea applied to EBS.

jordanthoms 12y ago
That would be awesome. A huge engineering effort - I imagine that would require building a radically different filesystem more or less from scratch - but AWS has the resources to do that sort of thing.
deleted 12y ago
[deleted]
rgbrenner 12y ago
something that behaves like a local file system (without the gotchas of NFS like a lack of delete on close), but I don't have to snapshot and create new volumes and issue file system expansion commands to grow a volume. I want seamless and automatic growth. Wouldn't EBS w/ thin provisioning get you most of this? Just create a massive volume, and you get billed for the space actually used. (and the volume size could also function as a limit on your bill.)
- andrew311 12y ago
  Yes, good point. This would provide effectively "infinite" backing storage. There might be some hurdles to overcome, though. For example, when you delete a file, will EBS know that the blocks are now free and thus can be decommissioned? This might mean the whole stack needs to support things like TRIM. I'm not sure the rest of the stack is smart enough yet. I'd love to hear from a storage/FS expert on this. Edit: coincidentally, I just saw this article about XFS which observes the following: "Over the next five or more years, XFS needs to have better integration with the block devices it sits on top of. Information needs to pass back and forth between XFS and the block device, [says Dave Chinner]. That will allow better support of thin provisioning." https://lwn.net/Articles/638546/ https://lwn.net/Articles/638546/
  - rgbrenner 12y ago
    The reason I suggested it is because thin volumes are well understood, so the issues are straightforward... and it's been implemented in many products (lvm; virtually every san; xenserver & vmware; etc). So there really shouldn't be many surprises if amazon were to implement it. And yes, trim is used to mark blocks as free. Honestly, it's so widespread, I would be surprised if Amazon weren't already using it to over-commit ebs.
kondro 12y ago
http://aws.amazon.com/storagegateway/ http://aws.amazon.com/storagegateway/ is a little like what you require (if you're unfamiliar). It's not unlimited, but currently its 32TB/volume and you're charged for the storage you use, rather than what is provisioned. It supports encryption at rest.

jordanthoms 12y ago

That would be awesome. A huge engineering effort - I imagine that would require building a radically different filesystem more or less from scratch - but AWS has the resources to do that sort of thing.

deleted 12y ago

[deleted]

rgbrenner 12y ago

something that behaves like a local file system (without the gotchas of NFS like a lack of delete on close), but I don't have to snapshot and create new volumes and issue file system expansion commands to grow a volume. I want seamless and automatic growth. Wouldn't EBS w/ thin provisioning get you most of this? Just create a massive volume, and you get billed for the space actually used. (and the volume size could also function as a limit on your bill.)

andrew311 12y ago
Yes, good point. This would provide effectively "infinite" backing storage. There might be some hurdles to overcome, though. For example, when you delete a file, will EBS know that the blocks are now free and thus can be decommissioned? This might mean the whole stack needs to support things like TRIM. I'm not sure the rest of the stack is smart enough yet. I'd love to hear from a storage/FS expert on this. Edit: coincidentally, I just saw this article about XFS which observes the following: "Over the next five or more years, XFS needs to have better integration with the block devices it sits on top of. Information needs to pass back and forth between XFS and the block device, [says Dave Chinner]. That will allow better support of thin provisioning." https://lwn.net/Articles/638546/ https://lwn.net/Articles/638546/
- rgbrenner 12y ago
  The reason I suggested it is because thin volumes are well understood, so the issues are straightforward... and it's been implemented in many products (lvm; virtually every san; xenserver & vmware; etc). So there really shouldn't be many surprises if amazon were to implement it. And yes, trim is used to mark blocks as free. Honestly, it's so widespread, I would be surprised if Amazon weren't already using it to over-commit ebs.

kondro 12y ago

http://aws.amazon.com/storagegateway/ http://aws.amazon.com/storagegateway/ is a little like what you require (if you're unfamiliar). It's not unlimited, but currently its 32TB/volume and you're charged for the storage you use, rather than what is provisioned. It supports encryption at rest.