8 ms·
Amazon Elastic File System Update – Sub-Millisecond Read Latency
- nickjj 5y agoOne of the most impressive parts of the post is this: > We "flipped the switch" and enabled this performance boost for all existing EFS General Purpose mode file systems over the course of the last few weeks Surely it wasn't as simple as flipping "[x] Make filesystem slower than it could be" to unchecked. It would be cool to read up on how they did this. I'm guessing lots of folks are writing a lot of data to the existing old set up, does anyone have any guesses on how they hot swapped that to the new set up with no downtime or data inconsistencies?
- deltaci 5y agomost of the latency comes from network layer. my naiive guess is they probably switched from a standard ethernet setup to a infiniband setup to achieve 600us of total latency.
- geertj 5y agoI'm the PMT for this project in the EFS team. The "flip the switch" part was indeed one of the harder parts to get right. Happy to share some limited details. The performance improvement builds on a distributed consistent cache. You can enable such a cache in multiple steps. First you deploy the software across the entire stack that supports the caching protocol but it's disabled by configuration. Then you turn it for the multiple components that are involved in the right order. Another thing that was hard to get right was to ensure that there are no performance regressions due to the consistency protocol. Shameless plug, only one I promise. If you think this is cool, we are hiring for multiple PMT and SDE positions, which can be fully remote at the Sr. level and above. DM me for details, see [1], or see amazon.jobs and search for EFS. [1] https://www.amazon.jobs/en/jobs/1935130/senior-software-development-engineer-fully-remote-us-canada-efs-distributed-storage https://www.amazon.jobs/en/jobs/1935130/senior-software-deve... EDIT: public link
- staticassertion 5y agoThis sounds a lot more interesting than I was expecting. I assumed you just swapped to better network hardware. Will there be more info at some point?
- lillecarl 5y agoI thought they're already running quite top dog stuff in their whole network, it's the only way to scale big isn't it?
- zoover2020 5y agoYou shared an internal link, aside from this, great work! CDO salutes you.
- ryanworl 5y agoI watched the SNIA presentation from SDC2020 on EFS and it described each extent in the file system as a state machine replicated via multi-paxos. It seems possible to implement this feature via a time-based leader lease on the extent where a read request goes to a read-through cache. The cache will store some metadata (e.g. a hash or a verison number) of the block it is trying to validate from cache and send that to the leader along with the read request. If the leader has the same version number for a block as was requested by the cache, the block itself does not need to be transferred and the other replicas don't need to be contacted. If you place the cache in the same AZ as the EC2 instance reading the file system, 600ms sounds viable. Am I on the right track? :)
- geertj 5y ago> Am I on the right track? :) The caches are local to each AZ so you get the low latency in each AZ, the other details are different. Unfortunately I can't share additional details at this moment, but we are looking to do a technical update on EFS at some point soon, maybe at a similar venue!
- ryanworl 5y ago
- lpapez 5y agoThis reminds me of a 10x-developer coworker at my previous company. Whenever more performance was needed from the system he would just say "sure, just give me a week to remove the sleep() calls from the code".
- Cthulhu_ 5y agoSee it's one of those things where you can't be sure if it's sarcasm or not; I've read plenty of anecdotes where it was in fact a case of putting in sleep() on purpose (either to remove it later to show off, or because customers were complaining things went too fast), or leaving it in by accident. Another variant is that older games (think cartridges, some CD's) had a block of unused data, so that when the game approached completion and they had to crunch to make it fit on the cartridge, they had some space left over. Probably the same for RAM memory.
- musingsole 5y agoI've done this. I had to add sleep calls because the client had a simple resource consumption monitor that would complain if anything used more than 80% of the machine's CPU. Arguably a stupid policy; arguably simple enough to work. Some time later, the client wanted the script to finish faster...so the sleep calls were deleted, and they changed their resource monitor.
- Spivak 5y agoI mean as an infra person I can kinda agree with this. I wouldn’t put it on the app, and would have used cgroups but I don’t want to starve OS services for cycles.
- infocollector 5y agoDoes anyone have any benchmarks with AWS FSX (LustreFS) vs EFS - how do they compare in random/sequential access?
- bushbaba 5y agoThere’s a durability difference. “Amazon EFS is designed to provide 99.999999999% (11 9’s) of durability over a given year” Where-as FSx for lustre’s durability and EBS single volume’s durability isn’t anywhere near as high. I suspect but cannot confirm that lustre can support higher single instance and in aggregate throughput speeds.
- mprovost 5y agoQuick plug for my open source project nfsping for measuring NFS latency: https://github.com/mprovost/NFStash/blob/master/md/nfsping.md https://github.com/mprovost/NFStash/blob/master/md/nfsping.m...
- geertj 5y agoHi, thanks for the link! I just tried this on EFS but it doesn't give the expected answer. The program uses NFS NULL calls. I'm guessing we haven't optimized those yet given that they are typically only used at mount time, but we'll have a look to see what's going on. One of the most common operations on NFS are OPEN and GETATTR. The following example measures the latency of OPEN (GETATTR would be similar): $ python3 -m timeit -s 'import os' 'os.open("/efs/test.bin", os.O_CREAT|os.O_RDONLY)' 500 loops, best of 5: 554 usec per loop Results will vary a bit depending on network routing and the region.
- mprovost 5y agoYes the NULL RPC does nothing and should give a baseline for measuring the round trip time. I'm surprised that it doesn't give the expected results. Your test case is going through the kernel, not sending raw RPCs across the wire. There are a lot of moving parts in the filesystem layer. I implemented direct RPC calls for other RPCs like GETATTR but only for NFS version 3, and EFS uses version 4. Unfortunately the design of version 4 is a disaster from a protocol perspective. The designers wanted to enable multiple operations in a single RPC, but that's not supported by SUNRPC. So to maintain backwards compatibility they only implemented 2 procedures, NULL and COMPOUND. Everything inside a COMPOUND RPC is opaque. Well it's possible to construct but not with the standard RPC toolchain, so I never got around to it.
- staticassertion 5y agoIs there an updated version of latency.txt? Would be interesting to compare against.
- geertj 5y ago> Is there an updated version of latency.txt? Would be interesting to compare against. Our updated docs [1] contains a latency table. The official answer is that previously our read latencies were low single-digit millisecond and now they are as low as 600 microseconds. [1] https://docs.aws.amazon.com/efs/latest/ug/performance.html https://docs.aws.amazon.com/efs/latest/ug/performance.html
- londons_explore 5y agoIs it me, or is "sub millisecond" actually quite unimpressive for an 'all-in-the-same-building-on-ssd' filesystem? When my consumer flash ssd can typically provide data to a read request inside 50 microseconds, I really expect a big hosted filesystem to have a median latency in the 100 microsecond range. Ie. I expect amazon to have the majority of filesystem metadata in RAM, such that a single read request from the user translates to a single SSD read request at a known block address. That data is then sent back across the 10GBit ethernet network to the machine doing the request, adding 1.3 microseconds per network switch.
- kylegordon 5y agoIt's not all in the same building though
- bushbaba 5y agoDo a ping test between two machines in your local network. Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds. the kernel will buffer and queue packets vs sending them out asap. So there’s delay added at your host level and networking layer. The. You’ve got the speed of light traversing that distance. All this adds latency.
- londons_explore 5y agoBut something like a filesystem should probably be built on RDMA to get the kernel mostly out-of-the-way. Then you should see performance like this: https://www.microway.com/knowledge-center-articles/performance-characteristics-of-common-network-fabrics/ https://www.microway.com/knowledge-center-articles/performan... And the speed of light is 1000 feet per microsecond, so even the largest data centers won't add more than a handful of microseconds.
- jeffbee 5y agoI just put a crossover cable between a Mac mini and a random Linux box and with no tuning the median ping is 180 microseconds.
- ZeroCool2u 5y agoThis is good to see. I didn't realize just how much this latency could effect me until I recently was doing some fine tuning on an NLP model on Sagemaker and noticed my training loss was looking pretty rough when I used a working directory that was in EFS as opposed to on my EC2 instance. Testing it out, performance was dramatically improved when I moved everything to the directly attached storage, so I've just been avoiding EFS since then during training.
- geertj 5y agoFor ML training, where you repeatedly open the same files over and over again, the NFS 'close to open' coherency protocol (where every open() and close() is a round trip) is not a good fit. Once we support NFS delegations, this will become a lot faster, but local training data set will very likely continue to be faster (we have ideas on this though!) Given that ML training uses very expensive instance types, a local file system is the way to go for now. ML inference on the other hand is a use case that we've specifically targeted with this update, and it should work very well. Many of our customers do inference using AWS Lambda, directly connected to their EFS file system.
- ZeroCool2u 5y agoAppreciate the insight, that's really good to know! It's one of those things that's been bugging me every once in a while when I think back to it. Also, does today's update apply to GovCloud regions as well or just Commercial for now?
- geertj 5y ago> Also, does today's update apply to GovCloud regions as well or just Commercial for now? It also applies to GovCloud.
- ZeroCool2u 5y agoThat's awesome, thank you!
- mabbo 5y ago> EFS file systems now provide average latency as low as 600 microseconds for the majority of read operations on data and metadata. I do not want to detract from how awesome this is at all. It's a great improvement and worth of celebration. That said, I spent long enough at Amazon to know that they have the metrics around this, before and after, to a surprising level of detail. When Amazon gives a single piece of data like that, it makes me worry me there is some nuance to the improvement. Show me the p50, p90, p99, and p99.99 as well as the average. Remember, if reading 1 byte is improved 10X on average, but one in every 10,000 bytes is worsened by 10X, then 10kb file reads might now take ~10X longer (if all reads were in parallel, etc).
- geertj 5y ago(PMT on the EFS team). Yes, the wordings are carefully formulated as they have to be signed off by the AWS legal team for obvious reasons. With that said, this update was driven by profiling real applications and addressing the most common operations, so the benefits are real. For example, a simple WordPress "hello world" is now about 2x as fast as before.
- remus 5y ago> For example, a simple WordPress "hello world" is now about 2x as fast as before. Before seeing this announcement, someone at work today literally said "the wordpress site is feeling very fast today, I wonder if anything has changed?" Great work!
- remram 5y agoIf you don't mind me asking, why do you run Wordpress off of EFS rather than EBS?
- JStanton617 5y agoEFS allows you to connect to the filesystem from multiple web front-end machines. Doing the same with EBS would require you to manage some sort of clustered filesystem, or run a separate storage tier server.
- joelbondurant1 5y ago
- TrispusAttucks 5y agoRecently moved a system off of EFS because the latency. This update might have prevented that. We had to make extensive use of fscache to get reasonable performance for shared data for web applications. I wonder if fscache is part of the solution on the backend.
- lysecret 5y agoI wonder what you all are using for EFS? I guess for larger models in ML (like NLP transformer stuff) it makes sense. I have observed myself moving everything i could to a DB and if i cant mostly S3. But for both there will be a latency pulling the data.
- mschuster91 5y ago> I wonder what you all are using for EFS? Quick and easy way to store the user uploads of a CMS like Drupal. Deploy the application itself via either Beanstalk, EKS/ECS or on an EC2 server, point the database to RDS and mount the sites/xxx/files directory from EFS.
- mwcampbell 5y agoI ran a Dovecot IMAP server using EFS for a while, before moving it to a leased dedicated server (i.e. away from AWS) with local NVMe storage. And that move did make it noticeably faster in addition to reducing the cost.
- electroly 5y agoI think there's very little "proper" usage for EFS. That is to say: if you had infinite time and dev resources, you'd pick something else because other options are faster and cheaper. But it is very easy to set up EFS and start using it since it's just a filesystem. The time it takes to migrate from "my code deals with local files" to "my code deals with networked files" with EFS is about 15 minutes regardless of what you're doing with the files. S3 can't say that. I think EFS's value proposition is more about that convenience than it is about actually being an appropriate tech choice when considered in a vacuum.
- the8472 5y agoOn a project we had to deal with some properietary software that was highly optimized for its CPU-bound steady-state workload but suffered from terrible startup times due to single-threaded, blocking IO to EFS. Part of the workaround was a wrapper that preopened files on multiple threads, issued readaheads and then rewrote the data to point to /proc/<pid>/fd/<number> paths instead of the paths on EFS. That alone already cut load times from 6 minutes to < 30 seconds. Lower latency storage helps, but applications not doing things the most naive way possible would be great too. Spawn a few threads if you have thousands of files to open. Issueing IO operations in batches to io_uring would help too but libraries need to catch up first.
- RyanHamilton 5y ago"Spawn a few threads if you have thousands of files to open." You've gone so deep into the trees that you can't see the forest. Creating a workaround wrapper or spawning unnecessary threads is a bad idea. You want async io or io_uring.
- otterley 5y ago“Proprietary software” suggests the commenter did not have the source code or resources to do that. Sometimes you just have to make the best hack you can.
- secondcoming 5y agoI was under the impression that Linux does not support async file access and so people have to fake it with threads.
- coder543 5y agoThis is no longer true since the arrival of io_uring, and that's why the person you replied to referenced it.
- ryeguy 5y agoCould you use EFS to host a horizontally scalable relational database by using finely sharded SQLite dbs? Like if you had 1 db per user, for example.
- sudhirj 5y agoYeah, think you could. They have a lambda adapter as well, so you’d get nearly infinite scale straight away.
- stephen 5y agoI've wanted to use "SQLite db per user" for awhile now but haven't had the right problem domain for it yet. https://engineering.backtrace.io/2021-12-02-verneuil-s3-backed-asynchronous-replication-for-sqlite/ https://engineering.backtrace.io/2021-12-02-verneuil-s3-back... looks really interesting.
- mboerwink 5y agoI actually recently delivered a project which uses almost this exact technique. Python lambdas that operate on SQLite files have the benefit of being much simpler and cheaper than most other scalable database solutions (like Aurora) for very light loads. That said, accessing SQLite databases is surprisingly disk IO heavy. I haven't gone too deeply into measuring the effect, but it seems the core issue is that traditional RDBS wire protocols are better than SQLite's disk accesses wrapped over NFS (or whatever connection the lambda/EFS join is). Stuff especially starts to break down when you need any sort of concurrent access. The small overhead for locking/unlocking files can quickly become awful when multiplied by the EFS latency, so you really do need extremely fine sharding.
- electroly 5y agoI do something similar, but test out the performance before you commit to it. There is a massive chasm between how EFS is marketed and how it actually performs. EFS is the slowest possible way to store data in AWS, with painfully low per-client and aggregate throughput. We implemented EFS because it was easy and then immediately commenced the project to replace it with S3. The bare EFS solution didn't survive long enough to make it to the S3 project's finish; we had to add a layer of caching instances in EC2 to bring EFS throughput up to an acceptable level. We almost needed two layers of caching instances; i.e. EFS fans out to N caching instances, which then fan out to M caching instances, which then fan out to P workers, P >> M >> N. Because the EFS throughput is so poor, it could barely cope with the fan-out to the one layer of caching instances. Fortunately the S3 migration project finished before we got that far. This announcement is, of course, the result of adding a caching layer on top of EFS. Naturally. But because they don't mention "throughput" and only mention "latency" I'm betting they have not used the cache layer to increase throughput. I'll only ever recommend EFS if your data is very small and the throughput requirements are negligible.
- wsaintx 5y agoFor people thinking about EFS - the backup story is currently not that great. You may see a few days of recovery time for a volume with a few Tbs. If you have a reasonable number of clients connecting (<20) and do not need access from Fargate or Lambda, then FsX (Ontap, openZFS) is a more well rounded offering.
- harshaw 5y agoSorry to hear that. We don't see this internally but if you put your email in your profile I'd be happy to reach out to you. (not sure else how to start the communication chain unfortunately). SDM on the team that owns backup.
- orf 5y agoI’m not sure how come you don’t see this internally. Before we moved off EFS we simply where never able to back up our drive - it would timeout after 7 days for an incremental backup, and we reported this several times in tickets. I think we had about 300tb in that drive. Which is a lot, sure, but never being able to take a backup made us pretty sad.
- wsaintx 5y ago
- isignal 5y agoDefinitely impressive. However, EFS perf oriented pricing is confusing. It seems to be quite expensive if we ask for throughput comparable with EBS. 100 MB/s costs $600/month. I’m not sure why the pricing for disitributed file system is so much higher than the block storage side. For ex, this makes running a db on top of EFS very expensive for non trivial workloads.
- geertj 5y agoHi! Unless you need to drive that 100 MB/s continuously, you can use the bursting throughput mode, which means you don't have to separately provision and pay for throughput. If you do need to drive 100 MB/s continuously, then Provisioned Throughput at the price that you mention would be appropriate. But note that you get 50 MB/s of throughput included with every TB of storage, and you only pay for the provisioned throughput in excess of what's included in your storage.
- isignal 5y agoThanks for the reply. Relying on bursting for a core service like a db would be too much risk - it would make sense for batch compute or similar workloads. From a systems viewpoint, I can’t understand if there’s an actual technical challenge involved or if this pricing is just a lack of competitive pressure. Is there some corner case about the POSIX guarantees of EFS that makes higher throughput challenging to achieve? (Otherwise one could just spray file blocks across many servers and maintain a metadata mapping of file to blocks to achieve good throughput)
- rmbyrro 5y agoYou should probably not run a database off of EFS. It's not best tailored for the type of concurrent access that DBs will throw at it.
- otterley 5y agoWhat kind of database do you want to run on EFS? Have you considered built-for-purpose managed services instead, such as RDS?