2 ms·

You specifically say "for large files". What's your bandwidth and latency like for small files (e.g. 20-20480 bytes), and how does it compare to raw S3's bandwi

You specifically say "for large files". What's your bandwidth and latency like for small files (e.g. 20-20480 bytes), and how does it compare to raw S3's bandwidth and latency for small files?

fenos 3y ago
You can think of the Storage product as an upload server that sits in front of S3. Generally, you would want to place an upload server to accept uploads from your customers, that is because you want to do some sort of file validation, access control or anything else once the file is uploaded. The nice thing is that we run Storage within the same AWS network, so the upload latency is as small as it can be. In terms of serving files, we provide a CDN out-of-the-box for any files that you upload to Storage, minimising latencies geographically
- Rapzid 3y ago
  > Generally, you would want to place an upload server to accept uploads from your customers A common pattern on AWS is to not handle the upload on your own servers. Checks are made ahead of time, conditions baked into the signed URL, and processing is handled after the fact via bucket events.
  - fenos 3y ago
    That is also a common pattern I agree, both ways are fine if the upload server is optimised accordingly :)
egorr 3y ago
hey, supabase engineer here; we didn’t check that out with files that small, but thanks for the idea, i will try it out the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb significantly, but tends to have similar or slightly slower speeds compared to s3 regular upload via supabase or simplest supabase storage upload for files with size about and less than 50mb. s3-multipart is indeed the fastest way to upload file to supabase with speeds up to 100mb/s(115 even) for files>500mb. But for files about 5mb or less you are not going to need to change anything in your upload logic just for performance cause you won’t notice any difference probably everything mentioned here is for upload only
ovaistariq 3y ago
S3 is not known for performing well with such small files. Is this primarily how your dataset on S3 looks like?

You specifically say "for large files". What's your bandwidth and latency like for small files (e.g. 20-20480 bytes), and how does it compare to raw S3's bandwidth and latency for small files?

fenos 3y ago

You can think of the Storage product as an upload server that sits in front of S3. Generally, you would want to place an upload server to accept uploads from your customers, that is because you want to do some sort of file validation, access control or anything else once the file is uploaded. The nice thing is that we run Storage within the same AWS network, so the upload latency is as small as it can be. In terms of serving files, we provide a CDN out-of-the-box for any files that you upload to Storage, minimising latencies geographically

Rapzid 3y ago
> Generally, you would want to place an upload server to accept uploads from your customers A common pattern on AWS is to not handle the upload on your own servers. Checks are made ahead of time, conditions baked into the signed URL, and processing is handled after the fact via bucket events.
- fenos 3y ago
  That is also a common pattern I agree, both ways are fine if the upload server is optimised accordingly :)

egorr 3y ago

hey, supabase engineer here; we didn’t check that out with files that small, but thanks for the idea, i will try it out the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb significantly, but tends to have similar or slightly slower speeds compared to s3 regular upload via supabase or simplest supabase storage upload for files with size about and less than 50mb. s3-multipart is indeed the fastest way to upload file to supabase with speeds up to 100mb/s(115 even) for files>500mb. But for files about 5mb or less you are not going to need to change anything in your upload logic just for performance cause you won’t notice any difference probably everything mentioned here is for upload only

ovaistariq 3y ago

S3 is not known for performing well with such small files. Is this primarily how your dataset on S3 looks like?