4 ms·
Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also. In this specific case im getting oldschool f
by voodooEntity 1y ago
Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also.
In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more.
And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs.
But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place.
Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.
- conductr 1y agoI find with these types of customers it’s always easier to just ask them to save files locally and grant me privileges to read the data. Sometimes they’ll be on Google, Dropbox, Microsoft, etc and I also run a SFTP for this in case they want to move them over to my service. Then I either batch/schedule the processing or give them an endpoint to just to trigger it (/data/import?filename=demo.csv) It’s actually so common that I just have the “data exchange” conversation and let them decide which fits their needs best. Most of it is available for self service configuration.
- darkwater 1y agoYep, I concur. You need to meet them on their (legacy) terrain, get access to their data and then you can do any fancy thing you want to do.
- nigga1234 1y ago[flagged]