7 ms·

The way to work around this issue is to provide a presigned S3 url Have the users upload to s3 directly and then they can either POST you what they uploaded or

The way to work around this issue is to provide a presigned S3 url Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something) I agree this is annoying and maybe I’ve been in AWS ecosystem for too long. However having an API that accepts an unbounded amount of data is a good recipe for DoS attacks, I suppose the 100MB is outdated as internet has gotten faster but eventually we do need some limit

voodooEntity 1y ago
Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also. In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more. And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs. But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place. Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.
- conductr 1y ago
  I find with these types of customers it’s always easier to just ask them to save files locally and grant me privileges to read the data. Sometimes they’ll be on Google, Dropbox, Microsoft, etc and I also run a SFTP for this in case they want to move them over to my service. Then I either batch/schedule the processing or give them an endpoint to just to trigger it (/data/import?filename=demo.csv) It’s actually so common that I just have the “data exchange” conversation and let them decide which fits their needs best. Most of it is available for self service configuration.
  - darkwater 1y ago
    Yep, I concur. You need to meet them on their (legacy) terrain, get access to their data and then you can do any fancy thing you want to do.
- nigga1234 1y ago
  [flagged]
reactordev 1y ago
Uploads to an S3 bucket can trigger a lambda… don’t complicate things. The upload trigger can tell the system about the upload and the client can continue on their day. Uploader on the client uses presigned url. S3 triggers lambda. Lambda function takes file path and tells background workers about it either via queue, mq, rest, gRPC, or doing the lift in workflow etl functions. Easy peasy. /s
- stuartjohnson12 1y ago
  > Uploads to an S3 bucket can trigger a lambda… don’t complicate things. I read this and was getting ready to angrily start beating my keyboard. The best satire is hard to detect.
  - Dylan16807 1y ago
    I don't really get the joke. S3 triggering a lambda doesn't sound meaningfully more complicated than using a lambda by itself. What am I missing?
    - reactordev 1y ago
      Solving a serverless limitation with more serverless so you can continue doing serverless when you can’t FormUpload a simple 101mb zip file as an application/octet-stream. Doubling down on it for a triple beat.
      - Dylan16807 1y ago
        I wouldn't really call it "more" severless to rearrange the order a bit. Which makes it "solving a serverless limitation so you can continue doing severless". And that's just a deliberately awkward way of saying "solving a serverless limitation" because if you can solve it easily why would you not continue? Spite? So I still don't see how it's notably worse than the idea of using serverless at all.
        1y ago
pluto_modadic 1y ago
this kinda proves the point that you have to know a silly workaround
deleted 1y ago
[deleted]

voodooEntity 1y ago

Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also. In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more. And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs. But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place. Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.

conductr 1y ago
I find with these types of customers it’s always easier to just ask them to save files locally and grant me privileges to read the data. Sometimes they’ll be on Google, Dropbox, Microsoft, etc and I also run a SFTP for this in case they want to move them over to my service. Then I either batch/schedule the processing or give them an endpoint to just to trigger it (/data/import?filename=demo.csv) It’s actually so common that I just have the “data exchange” conversation and let them decide which fits their needs best. Most of it is available for self service configuration.
- darkwater 1y ago
  Yep, I concur. You need to meet them on their (legacy) terrain, get access to their data and then you can do any fancy thing you want to do.
nigga1234 1y ago
[flagged]

reactordev 1y ago

Uploads to an S3 bucket can trigger a lambda… don’t complicate things. The upload trigger can tell the system about the upload and the client can continue on their day. Uploader on the client uses presigned url. S3 triggers lambda. Lambda function takes file path and tells background workers about it either via queue, mq, rest, gRPC, or doing the lift in workflow etl functions. Easy peasy. /s

stuartjohnson12 1y ago
> Uploads to an S3 bucket can trigger a lambda… don’t complicate things. I read this and was getting ready to angrily start beating my keyboard. The best satire is hard to detect.
- Dylan16807 1y ago
  I don't really get the joke. S3 triggering a lambda doesn't sound meaningfully more complicated than using a lambda by itself. What am I missing?
  - reactordev 1y ago
    Solving a serverless limitation with more serverless so you can continue doing serverless when you can’t FormUpload a simple 101mb zip file as an application/octet-stream. Doubling down on it for a triple beat.
    - Dylan16807 1y ago
      I wouldn't really call it "more" severless to rearrange the order a bit. Which makes it "solving a serverless limitation so you can continue doing severless". And that's just a deliberately awkward way of saying "solving a serverless limitation" because if you can solve it easily why would you not continue? Spite? So I still don't see how it's notably worse than the idea of using serverless at all.
      - 1y ago

pluto_modadic 1y ago

this kinda proves the point that you have to know a silly workaround

deleted 1y ago

[deleted]