24 ms·
Amazon Elastic File System – Production-Ready in Three Regions
- madbiz 10y agoCan EFS be used as a volume for ECS tasks?
- bbgm 10y agoHere's a post we wrote earlier in the year about EFS and ECS: https://aws.amazon.com/blogs/compute/using-amazon-efs-to-persist-data-from-amazon-ecs-containers/ https://aws.amazon.com/blogs/compute/using-amazon-efs-to-per...
- madbiz 10y agoThanks!
- madbiz 10y agoIn the blog post the EFS is mounted by the EC2 instances from the start but i expected to be dynamic depending on the scheduled tasks on the instance. I dont want to couple EFS volumes which might be used by the containers if they might get scheduled on an instance. I would like to have a general purpose cluster and let the scheduler instrument the EC2 instance to mount an EFS volume if its needed by a scheduled task dynamically.
- illumin8 10y agoIf you use the Convoy NFS volume driver with EFS, your containers can use any NFS server (including EFS) as highly available persistent volume storage: https://github.com/rancher/convoy https://github.com/rancher/convoy
- atrilumen 10y agoCan EFS be mounted from the Lambda environment?
- jeffbarr 10y agoNot at present. I would be interested in learning about use cases for this. Post here or find me online, as you wish.
- markonen 10y agoYou mention HPC and Big Data processing as use cases for EFS; Lambda is also very well suited for big processing tasks you want to execute quickly with a large amount of concurrency. You often need persistence across executions, for example when breaking tasks into multi-function workflows, and EFS would seem to simplify this over manually using S3 for it.
- jedberg 10y agoOne use case would be one that came up a while back here on HN -- The person had a large binary blob that they needed to load up every time their Lambda function ran. If they could mount EFS from Lambda, they could put the large binary blob there. Depending on how they wrote their code, if the Lambda could seek to the right part of the file then this would work, but if it has to load the entire file into RAM anyway, then it wouldn't gain them much except perhaps a slightly easier way to get the file into RAM.
- boucher 10y agoCouldn't you just download the portion of the one file you want from S3? Unless EFS is significantly faster... From a caching perspective, you wouldn't seem to gain much in lamda. You can cache in memory/tmpfs until your container eventually dies, and then you likely aren't on the same machine so there's no EFS level caching to take advantage of.
- jedberg 10y agoRight that was my point. You most likely have to load the whole thing into memory, but in the rare case that you can seek directly to the right spot in the file, ECS might have an advantage for you in Lambda.
- buremba 10y agoWe use a columnar database and we insert data to it periodically (30s to 90s) using Kinesis as middleware. Files are immutable and each insert creates [count of columns] * [count of tables] files. The database also periodically merges the batch files in background. Since EFS was not available, I was trying to develop a FUSE filesystem that writes data to both S3 and local filesystem (EBS) for durability and read only from local filesystem for performance. Even though it's cheaper than EFS, it's slow since each file needs to be sent to S3 individually. Our requirements are not that much because the files are immutable, there are not many files (I heard that EFS doesn't work well with many small files.), no need for concurrent access. I think I will give it a shot since it also fits our use-case.
- takeda 10y agoIf files are immutable, isn't s3 much better fit for it? s3 main weaknesses is that you can't modify part of a file, which makes it unsuitable for a filesystem, but looks like it satisfies your use case.
- buremba 10y agoMaybe but unfortunately the database doesn't support "backup store" that we can use directly S3, it just writes the data to specified directory in a filesystem. We should either fork the database or implement "S3 backed" filesystem and we picked the latter one. Unfortunately the performance FUSE is not that good for read and writing data to S3 is somehow expensive.
- jedberg 10y agoThis is really cool but also kind of a shame. I feel like it will encourage bad behavior. I can see people using this to deploy code by deploying onto the shared volume so that all their instances get an "instant update". Which is great when it works, but lord help you when EFS goes down and every app server you have is hung on a broken NFS connection.
- oconnore 10y agoIsn't the whole point of AWS that you can blame AWS engineers for downtime instead of your own (in this case) code deployment thingy?
- jedberg 10y agoI'm not sure if you're being sarcastic or not, but no, it isn't. It's to provide building blocks for infrastructure. Down time is still on you. Only a lazy engineer blames their tools for their failures.
- dunkelheit 10y ago"Down time is still on you" - well put. And it is actually great when you can only blame yourself because then you can fix the issue right away. If someone other is to blame, you need to a) convince that person that there is a problem and b) wait for the issue to rise to the top of their queue. Which can be painful.
- dexterdog 10y agoBut in the case of Amazon they will likely know about the problem before you would have anyway. Sure, their problems tend to have a much larger scale, but their ops teams are pretty good.
- Johnny555 10y agoJust because it's your fault, that doesn't mean that you can quickly and easily fix the problem yourself. Amazon has a team of people to keep EFS up and running, and AWS owns the entire storage, server, and network stack. When your own NFS server starts timing out, you may have only a few people to diagnose the problem.
- Cidan 10y agoWe've been in the EFS preview for quite a while now, and we are incredibly pleased with it. We do a few different things using EFS, including using it as an output for some of our Spark jobs. It's really a top notch product; couldn't be happier.
- kldavenport 10y agoPricing looks competitive https://aws.amazon.com/efs/pricing/ https://aws.amazon.com/efs/pricing/ I wonder if this helps those who perceive S3 as vendor lock-in since it replicates a traditional file system.
- LeoPanthera 10y agoDoes it? It looks significantly more expensive than, say, rsync.net.
- fulafel 10y agoAny ideas what they are using for NFS servers software?
- boraturan 10y agoSeems like no Windows support?
- Corrado 10y agoThat's a good question. Originally it was Linux only as the EFS client was locked to NFS 4.0 and the Windows Server NFS client was 4.1 (why 4.1 is not backwards compatible, I don't know). However, the announcement mentioned using the PowerShell cmdlets to attach EFS volumes, and in fact the PowerShell help pages have specific commands for working with EFS. So, my guess is that it is now available for Windows and Jeff didn't do a very good job of bringing that to light.
- rincebrain 10y agoThe EC2 example shows mounting with nfsvers=4.1, so I'm guessing it got resolved by bumping the version supported.
- objectivefs 10y agoCool! If you are looking at shared filesystems on EC2, you might also want to take a look at ObjectiveFS[1]. Works in all regions and on GCS, and you can mount your filesystem securely (end-to-end encryption) between regions. [1]: https://objectivefs.com https://objectivefs.com
- arqq 10y agoObjectiveFS looks awesome and I've been wanting to use it, but using non-S3 storage is locked to "enterprise, I can't afford it". :( But I also can't afford S3's bandwidth pricing.. I could, however afford to run an S3-API compatible service myself with bandwidth pricing I could afford. But then you wouldn't let me use it because then I'd be "enterprise".
- objectivefs 10y agoGreat that you have been wanting to use ObjectiveFS. We are happy to talk with you about your non-S3 storage use case and see if we can find a plan that works for you.
- merb 10y agoI wonder if some can run a postgresql database on top of that. This would make it relativly easy to have a master without downtime and be way cheaper than RDS.
- Sanddancer 10y agoIt claims to present itself as a full filesystem so I don't see why it wouldn't, however, you're probably going to run into pretty hardcore performance issues, because the bandwidth amazon provides in general is on the stingy side, but under a terabyte, they're downright miserly, with a 10 gig filesystem only able to provide 0.5 MB/s sustained. This product is problematic at best.
- Someone 10y agoAlso, it is a full-fledged NFS-mounted file system. I would be very wary of running a database from that.
- MrMullen 10y agoIt sounds like a disaster waiting to happen. The latency of NFS is just too high (Even if it is very fast for NFS) for it to be safe for a database.
- SteveNuts 10y agoWe're running Oracle on NFS volumes, and some MySQL databases also with no problems whatsoever. It'll never be as fast as local disk but the flexibility it provides is very nice.
- Sanddancer 10y agoThat's normal NFS. Amazon's offerings, if you have a 10 gigabyte database, you can only get 500KB/sec bandwidth -- http://docs.aws.amazon.com/efs/latest/ug/performance.html http://docs.aws.amazon.com/efs/latest/ug/performance.html
- Sanddancer 10y agoThis feels rather...primitive for a network filesystem introduced in 2016. Where are the snapshots? Where are the subvolumes? Where's the ability to send/receive volumes/subvolumes to filesystems in other availability zones? There are also I/O limits if you have a smaller filesystem with frequently accessed data, an inability to mount on machines through vpn gateways, and a lot of other seriously rough edges. It's an interesting start to a project, but the public features feel very much lacking.
- rincebrain 10y agoIt seems like this is more of a glue layer between a more interesting backend service (EFS) and clients, but I don't know that I'd expect Amazon to do something like write a native client, versus just probably improving the NFS client behavior on Windows/Linux/... The neat trick would be if NFS 4.X grew support for communicating interesting operations like snapshotting.
- KaiserPro 10y agoYES Finally we can start using standard posix semantics for things. Yes people might say its primitive, but currently all I can see is people trying to re-create shared files systems over protocols not designed for it cough cough HTTP Finally I can have a shared home, with a shared environment. A single readonly binary source (great for docker by the way.) also great for ensuring one version of scripts, without having to ansible everything.
- jdub 10y agoCongratulations, you've won a single point of failure!
- willejs 10y agoHAHAHA. YES! So much this. THere are legit use cases for this, but there are many use cases that will cause you much pain.
- omni 10y agoI suspect you're being downvoted because you've described every piece of technology ever created.
- MOARDONGZPLZ 10y agoOr the "HAHAHA. YES! So much this." part. Though I can't down vote so I wouldn't know.
- willejs 10y agoI just think that some people are getting excited about reviving the SAN. I don't think everyone realises or is familiar with the pain running NFS on distributed or non distributed infrastructure can cause, when used for the wrong use cases... Yes my comment was a bit of a joke too.
- illumin8 10y ago
- ngrilly 10y agoTo store small files (10 to 1000 KB) uploaded by users in a web app, what is the best tool, S3 or EFS?
- gedrap 10y agoA benefit (usually) of S3 is that since it's operating over HTTP, you don't need any additional things to make the files accessible. In case of EFS you would need a HTTP server running. Maintaining a HTTP work is extra work, extra configuration but might give you additional benefits. Thinking of small file size, EFS would probably work a bit faster (since you'd have some HTTP overhead) but I doubt that it would be significant, unless users upload hundreds of files at once. EFS advantage in this case would be that you'd no longer need a S3 library to do it, since it's POSIX compliant. Given that most languages have simple, solid libraries for interacting with it, it's not a huge difference but still :) All in all, I'd go with S3 unless there are some special requirements that are hard to satisfy using current S3 features.
- RossM 10y agoObject storage (i.e. s3+cloudfront) is still the correct way to store immutable files. that said, I know this will be used when the options are to rewrite filesystem access to an abstract (i.e. fs/s3/azure/etc) or just use nfs. People already use s3fs-fuse[1] which is dubious at best. [1]: https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse
- ngrilly 10y agoWould you say that the main choice criteria is immutability? Immutable files => Object storage => S3 Mutable files => File storage with POSIX semantics => EFS/NFS
- cshimmin 10y agoI have an application where users upload hundreds of small (~10kB) files every day. We quickly realized that S3 was not an option, since you get charged per request. As a workaround we batched the file uploads by archiving uploads each hour, and uploading a single .tar.gz to S3 at the end of the hour. The aggregated data gets processed offline later. We have N upload endpoints (as part of an autoscaling group), it's a bit of a pain to reaggregate the data since each server uploads its own .tgz to S3. I'm very happy to have EFS now! It makes scaling out our upload endpoints much simpler, as they can just dump their data into a common directory (which can still get archived and persisted to S3 later).
- s17tnet 10y agoFinally! My client was unable to use AWS for lacking of real POSIX fs.
- jakozaur 10y agoPrice is a bit high $0.3 GB / month vs. $0.1 GB / month for SSD EBS and even less for local storage. S3 is even cheaper $0.03 GB / month. So EFS is 10 times more expensive than S3. On I/O front also EBS (gp2) and (st1) are several times cheaper. So most of EFS advantages are in flexibility and sharing same file system across many instances. This may be great e.g. for sharing readonly binaries, but large scale data pipeline likely can be done much cheaper using different technologies.
- KaiserPro 10y agoYou are right for small scale things. I'll have to double check my working out, but you're only paying for what you use, not provision, which is one big things. Also, if you have > 3 machines working from the same dataset (ie docker image/video/large binaryblob) there is an instant saving, without factoring in provisioned vs useable space with EBS but, your original point, as a 1-1 replacement of a properly sized EBS mount, is correct, its more expensive.
- Corrado 10y agoI think EFS is meant for large read-heavy patterns, at least that's what we used it for. For example, I help run an e-commerce site that has a lot of content (JS, CSS, images, etc.) that we serve up from 10 dedicated servers as well as 30 general servers. When we release new content we have to update each of the servers and make sure that everything got installed properly everywhere. With EFS, we just have to update one thing and everyone gets it automatically.
- Corrado 10y agoI am excited by EFS but during the beta it didn't perform as well as I would have liked. I had a small Wordpress install that I wanted to make elastic in that I could have multiple web servers and all of the data was stored apart from the AMI. After testing my setup I found that latency was way too high and the performance of my site was terrible. I'm guessing it's because the amount of data was really low (100MB) and the requests were infrequent. I ended up going with a shared volume, which works fine and performance is great, but I can only attach the volume to a single instance at a time which prevents me from running multiple instances. That said, it looks like there has been some performance tweaks to EFS so maybe it might be better this time.
- lsaferite 10y agoWas it not possible to use the S3 plugin for WP to offload all the content files to S3 + CloudFront?
- jeffbarr 10y agoWe addressed this use case during the preview. Could you try it again now?
- jtrtoo 10y agoI think you might be going about solving this problem in a sub-optiimal way. WordPress data consists of db + asset files. Db gets addressed outside of this and asset files either go in s3 or are synced (rsync, etc)
- TheGuyWhoCodes 10y agoDoes EFS provide better throughput and latency than S3?
- bhouston 10y agoAre there any third party performance metrics yet? Read/write ops per second, and read/write bytes per second from one or many machines? Would love to see this in comparison to a 10Gbps (or 1Gpbs) attached spinning rust or SSD NSF drive. That would really help me understand what tradeoffs are happening here.
- deskamess 10y agoCan you expose the EFS file systems to CloudFront? For a future migration (local to cloud) I would like a backend application running on EC2 to use standard path names, but public access would be through CloudFront. It would be easier to migrate existing apps to the cloud with that approach, and then slowly work on changing the apps to use S3 (for cost reasons).
- sheraz 10y agoI think it is worth mentioning that Azure offer something similar with Samba / CIFS/ SMB protocol. [1] This is the underlying tech that powered their Docker volume plugin as well [2] I'm using in production for serving up small images to some web servers, and I'm currently in playground with docker stuff. So far, so good. I'm impressed. [1] https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-files/ https://azure.microsoft.com/en-us/documentation/articles/sto... [2] https://azure.microsoft.com/en-us/blog/persistent-docker-volumes-with-azure-file-storage/ https://azure.microsoft.com/en-us/blog/persistent-docker-vol... edit: footnotes
- aluskuiuc 10y agoA notable difference is that Azure file shares have a 5 TB limit for the whole share, and a 1 TB limit on the size of any given file [1], while EFS has no limit on the size of the file system, and a 52 TB limit on the size of any given file. [2] Disclosure, work for AWS. [1] https://azure.microsoft.com/en-us/documentation/articles/storage-scalability-targets/#scalability-targets-for-blobs-queues-tables-and-files https://azure.microsoft.com/en-us/documentation/articles/sto... [2] http://docs.aws.amazon.com/efs/latest/ug/limits.html http://docs.aws.amazon.com/efs/latest/ug/limits.html
- agperson 10y agoGlad to finally see this in production, wish it was available across VPC peering, VPNs, etc.
- gtirloni 10y agoI don't have a lot of experience with pNFS but it's supposed to be an extension to NFS 4.1, which Amazon EFS is supporting. However, pNFS is not mentioned in this blog post, their website or the official docs. That would be an interesting feature.
- api 10y agoThis is pretty awesome. Even though we don't use AWS much due to lack of IPv6 support, we've wanted to have a giant bottomless FS for various purposes. It'd be easy to set that up in here and export it over a virtual network.
- spullara 10y agoI really want to be able to access EFS file systems from Lambda. Has anyone figured out a way to access NFS?
- rymohr 10y agoStill no support for incremental snapshots though right?
- mike503 10y agoWhy hasn't anyone mentioned ObjectiveFS? When EFS was in preview just a month or so ago it was absolutely abysmal with performance for basic smaller web stuff. OFS feels nearly like an attached disk, and uses S3. You only pay per mount. Their support has been awesome and very personal. It provides a fuse-based full POSIX filesystem backed by S3 and can be mounted from multiple places in multiple regions. Some basic things like snapshotting they said are coming soon too. I gave up on EFS and even though it's finally out of preview, I think I still am going to prefer OFS based on what I've seen so far...