3 ms·

Not at present. I would be interested in learning about use cases for this. Post here or find me online, as you wish.

markonen 10y ago
You mention HPC and Big Data processing as use cases for EFS; Lambda is also very well suited for big processing tasks you want to execute quickly with a large amount of concurrency. You often need persistence across executions, for example when breaking tasks into multi-function workflows, and EFS would seem to simplify this over manually using S3 for it.
jedberg 10y ago
One use case would be one that came up a while back here on HN -- The person had a large binary blob that they needed to load up every time their Lambda function ran. If they could mount EFS from Lambda, they could put the large binary blob there. Depending on how they wrote their code, if the Lambda could seek to the right part of the file then this would work, but if it has to load the entire file into RAM anyway, then it wouldn't gain them much except perhaps a slightly easier way to get the file into RAM.
- boucher 10y ago
  Couldn't you just download the portion of the one file you want from S3? Unless EFS is significantly faster... From a caching perspective, you wouldn't seem to gain much in lamda. You can cache in memory/tmpfs until your container eventually dies, and then you likely aren't on the same machine so there's no EFS level caching to take advantage of.
  - jedberg 10y ago
    Right that was my point. You most likely have to load the whole thing into memory, but in the rare case that you can seek directly to the right spot in the file, ECS might have an advantage for you in Lambda.
    - boucher 10y ago
      Yeah, I just meant that S3 has partial downloads, so you can still seek. EFS/NFS makes a lot of sense I think when you have a lot of random/unpredictable access across large numbers of files. You could do the same thing with FUSE/S3 but across lots of smaller files/accesses I would think the overhead adds up faster.
atrilumen 10y ago
I want to run the Elm compiler in a Lambda, and maintain persistent environments so that subsequent builds can benefit from keeping the libraries and intermediate files available.
shatgupt 10y ago
I would like to have a media pipeline where a media goes through QC check, thumbnail, preview generation and some other processing which are independent of each other. My media can range from 1-15GB, so if I can store it in EFS which can be mounted in Lambda, I can use various Lambda functions which can simply read from a common storage without downloading every time from S3.

You mention HPC and Big Data processing as use cases for EFS; Lambda is also very well suited for big processing tasks you want to execute quickly with a large amount of concurrency. You often need persistence across executions, for example when breaking tasks into multi-function workflows, and EFS would seem to simplify this over manually using S3 for it.

jedberg 10y ago

One use case would be one that came up a while back here on HN -- The person had a large binary blob that they needed to load up every time their Lambda function ran. If they could mount EFS from Lambda, they could put the large binary blob there. Depending on how they wrote their code, if the Lambda could seek to the right part of the file then this would work, but if it has to load the entire file into RAM anyway, then it wouldn't gain them much except perhaps a slightly easier way to get the file into RAM.

boucher 10y ago
Couldn't you just download the portion of the one file you want from S3? Unless EFS is significantly faster... From a caching perspective, you wouldn't seem to gain much in lamda. You can cache in memory/tmpfs until your container eventually dies, and then you likely aren't on the same machine so there's no EFS level caching to take advantage of.
- jedberg 10y ago
  Right that was my point. You most likely have to load the whole thing into memory, but in the rare case that you can seek directly to the right spot in the file, ECS might have an advantage for you in Lambda.
  - boucher 10y ago
    Yeah, I just meant that S3 has partial downloads, so you can still seek. EFS/NFS makes a lot of sense I think when you have a lot of random/unpredictable access across large numbers of files. You could do the same thing with FUSE/S3 but across lots of smaller files/accesses I would think the overhead adds up faster.

atrilumen 10y ago

I want to run the Elm compiler in a Lambda, and maintain persistent environments so that subsequent builds can benefit from keeping the libraries and intermediate files available.

shatgupt 10y ago

I would like to have a media pipeline where a media goes through QC check, thumbnail, preview generation and some other processing which are independent of each other. My media can range from 1-15GB, so if I can store it in EFS which can be mounted in Lambda, I can use various Lambda functions which can simply read from a common storage without downloading every time from S3.