4 ms·

Uploads to an S3 bucket can trigger a lambda… don’t complicate things. The upload trigger can tell the system about the upload and the client can continue on th

Uploads to an S3 bucket can trigger a lambda… don’t complicate things. The upload trigger can tell the system about the upload and the client can continue on their day. Uploader on the client uses presigned url. S3 triggers lambda. Lambda function takes file path and tells background workers about it either via queue, mq, rest, gRPC, or doing the lift in workflow etl functions. Easy peasy. /s

stuartjohnson12 1y ago
> Uploads to an S3 bucket can trigger a lambda… don’t complicate things. I read this and was getting ready to angrily start beating my keyboard. The best satire is hard to detect.
- Dylan16807 1y ago
  I don't really get the joke. S3 triggering a lambda doesn't sound meaningfully more complicated than using a lambda by itself. What am I missing?
  - reactordev 1y ago
    Solving a serverless limitation with more serverless so you can continue doing serverless when you can’t FormUpload a simple 101mb zip file as an application/octet-stream. Doubling down on it for a triple beat.
    - Dylan16807 1y ago
      I wouldn't really call it "more" severless to rearrange the order a bit. Which makes it "solving a serverless limitation so you can continue doing severless". And that's just a deliberately awkward way of saying "solving a serverless limitation" because if you can solve it easily why would you not continue? Spite? So I still don't see how it's notably worse than the idea of using serverless at all.
      - reactordev 1y ago
        The controversy here is the fact that the API Gateway limits the upload resulting in having to engineer a workaround workflow using s3 and triggers (even if this is the serverless way) when all you want to do is upload a file. A POST call with an octet-octet stream. Let http handle resume. But you can’t and you end up going around the side door, when all you really want is client_body_max_size The sarcasm of correctness yet playing down its complexity is entirely my own. We used to be able to do things easily.
  - akdev1l 1y ago
    It gets really complex in this workflow to even achieve something like “file coprocessor successfully” on the client side with this approach how will your client know if you backend lambda crashed or whatever? All it knows is the upload to s3 succeeded Basically you’re turning a synchronous process into asynchronous
- fragmede 1y ago
  unfortunately they ruined it at the end with that /s
  - reactordev 1y ago
    Did I? I don’t think I did.
isoprophlex 1y ago
Every day we stray further from the light
themafia 1y ago
> Easy peasy. /s It actually is though. I don't need to build a custom upload client, I don't need to manage restart behavior, I get automatic restarts if any of the background workers fail, I have a dead letter queue built in to catch unusual failures, I can tie it all together with a common API that's a first class component of the system. Working in the cloud forces you to address the hard problems first. If you actually take the time to do this everything else becomes _absurdly_ easy. I want to write programs. I don't want to manage failures and fix bad data in the DB directly. I personally love the cloud and this separation of concerns.
- sunrunner 1y ago
  > Working in the cloud forces you to address the hard problems first. It also forces you to address all the non-existent problems first, the ones you just wish you had like all the larger companies that genuinely have to deal with thousands of file upload per second. And don't forget all the new infrastructure you added to do the job of just receiving the file in your app server and putting it into the place it was going to go anyway but via separate components that all always seem to end up with individual repositories, separate deployment pipelines, and that can't be effectively tested in isolation without going into their target environment. And all the additional monitoring you need on each of the individual components that were added, particularly on those helpful background workers to make sure they're actually getting triggered (you won't know they're failing if they never got called in the first place due to misconfiguration). And you're now likely locked into your upload system being directly coupled to your cloud vendor. Oh wait, you used Minio to provide a backend-agnostic intermediate layer? Great, that's another layer that needs managing. Is a content delivery network better suited to handling concurrent file uploads from millions of concurrent users than your app server? I'd honestly hope so, that's what it's designed for. Was it necessary? I'd like to see the numbers first. At the end of the day, every system design decision is a trade off and almost always involves some kind of additional complexity for some benefit. It might be worth the cost, but a lot of these system designs don't need this many moving parts to achieve the same results and this only serves to add complexity without solving a direct problem. If you're actually that company, good for you and genuinely congratulations on the business success. The problem is that companies that don't currently and may never need that are being sold system designs that, while technically more than capable, are over-designed for the problem they're solving.
MrDarcy 1y ago
If you don’t do it this way you fail the system design interview.
- reactordev 1y ago
  Nope, you didn’t use terracottax so you failed anyway. 6 months before you can reapply in case the first humiliation wasn’t enough. Boss was looking for AWS Glue in there and you didn’t use it.
raw_anon_1111 1y ago
And while you are being sarcastic, this is the Right Way to use queues. Upload file to S3 -> trigger an SNS message for fanout if you need it -> SNS -> SQS trigger -> SQS to ETL jobs. The ETL job can then be hosted using Lambda (easiest) or ECS/Docker/Fargate (still easy and scales on demand) or even a set of EC2 instances that scale based on the items in a queue (don’t do this unless you have a legacy app that can’t be containerized). If your client only supports SFTP, there is the SFTP Transfer Service on AWS that will allow them to send the file via SFTP and it is automatically copied to an S3 bucket. Alternatively, there are products that treat S3 as a mountable directory and they can just use whatever copy commands on their end to copy the file to a “folder”
- lumost 1y ago
  If I have a user facing upload button, why can't I simply have a webserver that receives the data and pushes it into s3 via multi-part upload. Something that can be written in a framework of your choice in 10 minutes with 0 setup? For uploads under 50 MB you could also skip the multipart upload and take a naive approach without taking a significant hit.
  - raw_anon_1111 1y ago
    You can - you generate the pre-signed S3 URL and they upload it to the place your URL tells it to. https://fullstackdojo.medium.com/s3-upload-with-presigned-url-react-and-nodejs-b77f348d54cc https://fullstackdojo.medium.com/s3-upload-with-presigned-ur... And before you cry “lock in”, S3 API compatible services are a dime a dozen outside of AWS including GCP and even Backblaze B2.

stuartjohnson12 1y ago

> Uploads to an S3 bucket can trigger a lambda… don’t complicate things. I read this and was getting ready to angrily start beating my keyboard. The best satire is hard to detect.

Dylan16807 1y ago
I don't really get the joke. S3 triggering a lambda doesn't sound meaningfully more complicated than using a lambda by itself. What am I missing?
- reactordev 1y ago
  Solving a serverless limitation with more serverless so you can continue doing serverless when you can’t FormUpload a simple 101mb zip file as an application/octet-stream. Doubling down on it for a triple beat.
  - Dylan16807 1y ago
    I wouldn't really call it "more" severless to rearrange the order a bit. Which makes it "solving a serverless limitation so you can continue doing severless". And that's just a deliberately awkward way of saying "solving a serverless limitation" because if you can solve it easily why would you not continue? Spite? So I still don't see how it's notably worse than the idea of using serverless at all.
    - reactordev 1y ago
      The controversy here is the fact that the API Gateway limits the upload resulting in having to engineer a workaround workflow using s3 and triggers (even if this is the serverless way) when all you want to do is upload a file. A POST call with an octet-octet stream. Let http handle resume. But you can’t and you end up going around the side door, when all you really want is client_body_max_size The sarcasm of correctness yet playing down its complexity is entirely my own. We used to be able to do things easily.
- akdev1l 1y ago
  It gets really complex in this workflow to even achieve something like “file coprocessor successfully” on the client side with this approach how will your client know if you backend lambda crashed or whatever? All it knows is the upload to s3 succeeded Basically you’re turning a synchronous process into asynchronous
fragmede 1y ago
unfortunately they ruined it at the end with that /s
- reactordev 1y ago
  Did I? I don’t think I did.

isoprophlex 1y ago

Every day we stray further from the light

themafia 1y ago

> Easy peasy. /s It actually is though. I don't need to build a custom upload client, I don't need to manage restart behavior, I get automatic restarts if any of the background workers fail, I have a dead letter queue built in to catch unusual failures, I can tie it all together with a common API that's a first class component of the system. Working in the cloud forces you to address the hard problems first. If you actually take the time to do this everything else becomes _absurdly_ easy. I want to write programs. I don't want to manage failures and fix bad data in the DB directly. I personally love the cloud and this separation of concerns.

sunrunner 1y ago
> Working in the cloud forces you to address the hard problems first. It also forces you to address all the non-existent problems first, the ones you just wish you had like all the larger companies that genuinely have to deal with thousands of file upload per second. And don't forget all the new infrastructure you added to do the job of just receiving the file in your app server and putting it into the place it was going to go anyway but via separate components that all always seem to end up with individual repositories, separate deployment pipelines, and that can't be effectively tested in isolation without going into their target environment. And all the additional monitoring you need on each of the individual components that were added, particularly on those helpful background workers to make sure they're actually getting triggered (you won't know they're failing if they never got called in the first place due to misconfiguration). And you're now likely locked into your upload system being directly coupled to your cloud vendor. Oh wait, you used Minio to provide a backend-agnostic intermediate layer? Great, that's another layer that needs managing. Is a content delivery network better suited to handling concurrent file uploads from millions of concurrent users than your app server? I'd honestly hope so, that's what it's designed for. Was it necessary? I'd like to see the numbers first. At the end of the day, every system design decision is a trade off and almost always involves some kind of additional complexity for some benefit. It might be worth the cost, but a lot of these system designs don't need this many moving parts to achieve the same results and this only serves to add complexity without solving a direct problem. If you're actually that company, good for you and genuinely congratulations on the business success. The problem is that companies that don't currently and may never need that are being sold system designs that, while technically more than capable, are over-designed for the problem they're solving.

MrDarcy 1y ago

If you don’t do it this way you fail the system design interview.

reactordev 1y ago
Nope, you didn’t use terracottax so you failed anyway. 6 months before you can reapply in case the first humiliation wasn’t enough. Boss was looking for AWS Glue in there and you didn’t use it.

raw_anon_1111 1y ago

And while you are being sarcastic, this is the Right Way to use queues. Upload file to S3 -> trigger an SNS message for fanout if you need it -> SNS -> SQS trigger -> SQS to ETL jobs. The ETL job can then be hosted using Lambda (easiest) or ECS/Docker/Fargate (still easy and scales on demand) or even a set of EC2 instances that scale based on the items in a queue (don’t do this unless you have a legacy app that can’t be containerized). If your client only supports SFTP, there is the SFTP Transfer Service on AWS that will allow them to send the file via SFTP and it is automatically copied to an S3 bucket. Alternatively, there are products that treat S3 as a mountable directory and they can just use whatever copy commands on their end to copy the file to a “folder”

lumost 1y ago
If I have a user facing upload button, why can't I simply have a webserver that receives the data and pushes it into s3 via multi-part upload. Something that can be written in a framework of your choice in 10 minutes with 0 setup? For uploads under 50 MB you could also skip the multipart upload and take a naive approach without taking a significant hit.
- raw_anon_1111 1y ago
  You can - you generate the pre-signed S3 URL and they upload it to the place your URL tells it to. https://fullstackdojo.medium.com/s3-upload-with-presigned-url-react-and-nodejs-b77f348d54cc https://fullstackdojo.medium.com/s3-upload-with-presigned-ur... And before you cry “lock in”, S3 API compatible services are a dime a dozen outside of AWS including GCP and even Backblaze B2.