3 ms·

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (

by jupp0r 6y ago

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

Havoc 6y ago
It certainly has some potential for shooting oneself in the foot in some scenarios, but still nice to have the option
scarface74 6y ago
I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb. We were just about to put the script in Fargate (Serverless Docker) and run a ECS task as part of the state machine. Now we don’t have to.
- hansef 6y ago
  This - if you have to fetch data from or output data to outside of the AWS ecosystem, the 512mb /tmp limit pushes you into the additional (relative) complexity of having to run on Fargate pretty quickly. Just had to deal with this for a content ingest job involving pulling a couple GB of data from an FTP server, processing it and pushing it into an RDS database on an hourly basis. Would have been super simple if the file was on S3 already.
  - jupp0r 6y ago
    Where do you need a distributed file system in this use case? Sounds like all you need is some local scratch space?
    - scarface74 6y ago
      Lambda only gives you 512MB pf local “scratch space”. If it had to provision and de provision gigs of space at each invocation, it would probably cause longer start times and shut down times.
- jupp0r 6y ago
  Why do you need a distributed file system to solve that problem? It sounds like all you need is some scratch space?
riquito 6y ago
"practicality beats purity" Depending on the use case it can be a good option (some are listed in the article). If the dev doesn't get the disadvantages the rest of the architecture will hardly be correct anyway
KaiserPro 6y ago
It can, but... As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.
- hamandcheese 6y ago
  S3 allows you to specify a byte range.
- deleted 6y ago
  [deleted]
giorgioz 6y ago
time will prove you very wrong

by jupp0r 6y ago

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

Havoc 6y ago

It certainly has some potential for shooting oneself in the foot in some scenarios, but still nice to have the option

scarface74 6y ago

I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb. We were just about to put the script in Fargate (Serverless Docker) and run a ECS task as part of the state machine. Now we don’t have to.

hansef 6y ago
This - if you have to fetch data from or output data to outside of the AWS ecosystem, the 512mb /tmp limit pushes you into the additional (relative) complexity of having to run on Fargate pretty quickly. Just had to deal with this for a content ingest job involving pulling a couple GB of data from an FTP server, processing it and pushing it into an RDS database on an hourly basis. Would have been super simple if the file was on S3 already.
- jupp0r 6y ago
  Where do you need a distributed file system in this use case? Sounds like all you need is some local scratch space?
  - scarface74 6y ago
    Lambda only gives you 512MB pf local “scratch space”. If it had to provision and de provision gigs of space at each invocation, it would probably cause longer start times and shut down times.
jupp0r 6y ago
Why do you need a distributed file system to solve that problem? It sounds like all you need is some scratch space?

riquito 6y ago

"practicality beats purity" Depending on the use case it can be a good option (some are listed in the article). If the dev doesn't get the disadvantages the rest of the architecture will hardly be correct anyway

KaiserPro 6y ago

It can, but... As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.

hamandcheese 6y ago
S3 allows you to specify a byte range.
deleted 6y ago
[deleted]

giorgioz 6y ago

time will prove you very wrong