4 ms·

Supabase Storage v2: Image Resizing and Smart CDN

109 points by samwillis 4y ago

kiwicopple 4y ago
hey hn, supabase ceo here hopefully it's obvious from the title what we're shipping today. "Supabase Storage" is for larger assets - it's a wrapper around s3, where the buckets/folders are mapped into your postgres database so that you can write access rules using Row Level Security. This release adds image resizing. It looks like this: supabase.storage.from('bucket').getPublicUrl('image.jpg', { transform: { width: 500, height: 600, }, }) In tandem added some webhooks/events to the storage server to enable some smarter cache-busting. We're using Cloudflare for caching on our hosted platform, but the addition of webhooks should provide the primitives to use any CDN/service. The Storage API is 100% open source and can be hosted as a standalone server (with Postgres+PostgREST): https://github.com/supabase/storage-api https://github.com/supabase/storage-api
- danpalmer 4y ago
  > it's a wrapper around s3 > We're using Cloudflare for caching It sounds like Supabase is fairly explicitly targeting being the glue between a bunch of lower level infrastructure, that may be in different cloud services, and providing one platform with a better UX. Is this fair to say? If so this is an interesting strategy to be so open about this. Other platforms-as-a-service don't seem to like to acknowledge the services they use under the hood. I can see this being both a positive (brand recognition, trust), and a negative ("we could just build this ourselves and have more choice"). Is this an intentional strategy? Can you talk any more about how you're thinking about it?
  - kiwicopple 4y ago
    > explicitly targeting being the glue between a bunch of lower level infrastructure It might be more precise to say that we provide the primitives that a developer needs to host Supabase on any cloud provider. Becoming the "glue" appears to be a byproduct of this approach For example: we use s3 right now, but we've built the storage server so that it can be extended with other storage providers (in fact, it has a "local disk" provider for developing locally). We use Cloudflare right now for the CDN, but as mentioned in the blog post the Webhooks we built into this release are so that we could migrate to any CDN service without locking ourselves into any CDN-specific cache invalidation. Portability is something we think about a lot (although harder to solve this problem at a CDN-level). > Other platforms-as-a-service don't seem to like to acknowledge the services they use under the hood Is this an intentional strategy? Can you talk any more about how you're thinking about it? There's no strategy behind it. We are trying to build a great developer platform, and at our current stage these platforms provide a better service for some features than we could ourselves. There's no shame in that from our POV, and these services deserve the credit for delivering something useful. I doubt we'll ever get to a "build our own datacenter" stage - we'll probably just continue to focus on the application level (with a Database-centric view of the world)
    - no_wizard 4y ago
      I’m perhaps overly bullish on your platform so I say never say never! Transparently, I hope you guys are hiring developers again soon, I had look into becoming a technical engineer on the sales side but we couldn't make the numbers work. I really can't wait to see your guys next round of innovations!
  - chrisweekly 4y ago
    Great point. Maybe 2y ago I asked Vercel salespeople about their infr and learned it was virtually all AWS. (Technically "multi-cloud" bc they used GCP in some minor way.)
- braindead_in 4y ago
  Great work. Congrats! I am looking to build a file management app using Supabase + NextJS (e.g. Dropbox clone). Can you please point me the right way. Thanks!
  - kiwicopple 4y ago
    I'm not sure if I could describe the full implementation of a Dropbox clone in a HN comment, but our Storage Docs[0] should give you an idea of most of the details that you need. If you prefer videos, I see there are a few tutorials on YouTube when you search "Supabase Storage" [0] Storage Docs: https://supabase.com/docs/guides/storage https://supabase.com/docs/guides/storage
- techn00 4y ago
  When will the storage dashboard be open source?
  - kiwicopple 4y ago
    Just caught up with the FE team on this one - it's pending in this PR: https://github.com/supabase/supabase/pull/10810 https://github.com/supabase/supabase/pull/10810 One of the engineers is testing this now and it should get merged within the hour. The build for amd64 takes about 6mins, and about 4 hours for arm64. I'll update here when it's merged (edit: it was actually merged earlier today, I originally linked to the wrong PR) For more context, the Storage UI was merged a couple of weeks ago but we simultaneously rolled out a new login screen for our Dashboard (for SSO) which then got rolled out to the Docker image. We've added a feature flag. Sorry for the delays - the lead up to launch week has everyone a bit stretched.
    - eclipxe 4y ago
      What a great comment. I applaud the transparency and depth. Nice.
- no_wizard 4y ago
  Any plans to switch to R2? It’s cheaper than S3 and would run in network with Cloudflare, allowing Supabase to take advantage of the CDN more optimally over time
  - kiwicopple 4y ago
    It's one of the storage providers we'll investigate, now that R2 is GA. Our Storage server is hosted on AWS, so it wouldn't immediately save money (at least for egress) but we'll also investigate moving the servers to another cloud provider.
    - skrtskrt 4y ago
      Could this eventually be pluggable with bring-your-own s3-compatible provider? Would love this in front of my DigitalOcean object storage
      - kiwicopple 4y ago
        yes definitely. It should be easy to add s3-compatible providers (which are numerous, since s3 seems to be the defacto file storage API now). We haven't added "BYO" functionality to our hosted Platform because, frankly, we already spend a lot of time debugging various quirks and network issues - even for infra which we provide. Adding an extra level of opaqueness at our current team size would be very stressful from a support-perspective. That said, we're designing/developing so that it's possible and we'll continue to assess as the team grows.
- worldmerge 4y ago
  Hi, I know this is off topic but I would appreciate if there was a C++ library for the https://supabase.com/realtime https://supabase.com/realtime product. I would like to use it for sensors but no C++ library means I can't use it on my Arduino or ESP32s.
inian 4y ago
Supabase Engineer here There were a lot of primitives we built to ship image resizing - Storage events exposed via webhooks, rate limiting, a queue on top of Postgres, a smart CDN cache, object metadata endpoints, etc. These are already available if you are self hosting Supabase Storage, so that you can integrate your own CDN, listen to storage events, etc. Over the next few months, we will be working on exposing these on the Supabase platform too.
- LukeLambert 4y ago
  In nearly two decades developing for the web, I've never found a good reason to enlarge an image server-side, so thank you for anticipating that footgun. So many of your competitors get it wrong. However, it's unclear from the docs [0] if one of the most common use cases is supported: requesting an image at a specific aspect ratio while not exceeding a maximum size. For instance, I want a 1600×900px hero image at the top of my blog posts, but my source image is only 1200px wide. I request: transform: { width: 1600, height: 900, resize: 'cover' } Will the returned image be 675px tall, maintaining the 16×9 aspect ratio I want? A few more notes: The blog post states "default mode maintains aspect ratio and the resize mode is fill", but the docs say that cover is the default mode. The docs say, "If only one [width or height] parameter is specified, the image will be resized and cropped, maintaining the aspect ratio." I believe that with only one parameter, you could either resize or crop, but not both. I hope it's resize. For cropping modes, is it always from the center? Are there plans to support other "gravity" modes? [0] https://supabase.com/docs/guides/storage/image-transformations https://supabase.com/docs/guides/storage/image-transformatio...
  - kiwicopple 4y ago
    > The blog post states "default mode maintains aspect ratio and the resize mode is fill", but the docs say that cover is the default mode. The default resize mode is cover and the blog post had a typo. Fixed now, thanks! > resize or crop We crop by default since the default mode is cover. > gravity we will be adding crop with gravity soon.. currently the crop is always from the center > requesting an image at a specific aspect ratio while not exceeding a maximum size. I'll get back to you on this one - Inian had to get some sleep, it's very late (early) for him
laurieg 4y ago
I love the product and this update. I wondered if there are any plans for handling audio (or even video) in a similar way?
- inian 4y ago
  Thank you! We will be working on video after we implement more image transformations. What is your usecase for audio and what transformations would you like to see?
  - laurieg 4y ago
    My use case is in education, with users uploading short audio clips. Often these are in a variety of formats. It's nice to be able to re-encode them all to the same format and save as much bandwidth as possible as well as make sure they are supported in the weirdest variety of devices.
SpaghettiX 4y ago
Are images really this expensive to resize?: "it is $5 per additional 1000 images transformed.". Perhaps then we should ship a WebAssembly module for image resizing to the client and only validate image sizes server-side.

109 points by samwillis 4y ago

hey hn, supabase ceo here hopefully it's obvious from the title what we're shipping today. "Supabase Storage" is for larger assets - it's a wrapper around s3, where the buckets/folders are mapped into your postgres database so that you can write access rules using Row Level Security. This release adds image resizing. It looks like this: supabase.storage.from('bucket').getPublicUrl('image.jpg', { transform: { width: 500, height: 600, }, }) In tandem added some webhooks/events to the storage server to enable some smarter cache-busting. We're using Cloudflare for caching on our hosted platform, but the addition of webhooks should provide the primitives to use any CDN/service. The Storage API is 100% open source and can be hosted as a standalone server (with Postgres+PostgREST): https://github.com/supabase/storage-api https://github.com/supabase/storage-api

danpalmer 4y ago
> it's a wrapper around s3 > We're using Cloudflare for caching It sounds like Supabase is fairly explicitly targeting being the glue between a bunch of lower level infrastructure, that may be in different cloud services, and providing one platform with a better UX. Is this fair to say? If so this is an interesting strategy to be so open about this. Other platforms-as-a-service don't seem to like to acknowledge the services they use under the hood. I can see this being both a positive (brand recognition, trust), and a negative ("we could just build this ourselves and have more choice"). Is this an intentional strategy? Can you talk any more about how you're thinking about it?
- kiwicopple 4y ago
  > explicitly targeting being the glue between a bunch of lower level infrastructure It might be more precise to say that we provide the primitives that a developer needs to host Supabase on any cloud provider. Becoming the "glue" appears to be a byproduct of this approach For example: we use s3 right now, but we've built the storage server so that it can be extended with other storage providers (in fact, it has a "local disk" provider for developing locally). We use Cloudflare right now for the CDN, but as mentioned in the blog post the Webhooks we built into this release are so that we could migrate to any CDN service without locking ourselves into any CDN-specific cache invalidation. Portability is something we think about a lot (although harder to solve this problem at a CDN-level). > Other platforms-as-a-service don't seem to like to acknowledge the services they use under the hood Is this an intentional strategy? Can you talk any more about how you're thinking about it? There's no strategy behind it. We are trying to build a great developer platform, and at our current stage these platforms provide a better service for some features than we could ourselves. There's no shame in that from our POV, and these services deserve the credit for delivering something useful. I doubt we'll ever get to a "build our own datacenter" stage - we'll probably just continue to focus on the application level (with a Database-centric view of the world)
  - no_wizard 4y ago
    I’m perhaps overly bullish on your platform so I say never say never! Transparently, I hope you guys are hiring developers again soon, I had look into becoming a technical engineer on the sales side but we couldn't make the numbers work. I really can't wait to see your guys next round of innovations!
- chrisweekly 4y ago
  Great point. Maybe 2y ago I asked Vercel salespeople about their infr and learned it was virtually all AWS. (Technically "multi-cloud" bc they used GCP in some minor way.)
braindead_in 4y ago
Great work. Congrats! I am looking to build a file management app using Supabase + NextJS (e.g. Dropbox clone). Can you please point me the right way. Thanks!
- kiwicopple 4y ago
  I'm not sure if I could describe the full implementation of a Dropbox clone in a HN comment, but our Storage Docs[0] should give you an idea of most of the details that you need. If you prefer videos, I see there are a few tutorials on YouTube when you search "Supabase Storage" [0] Storage Docs: https://supabase.com/docs/guides/storage https://supabase.com/docs/guides/storage
techn00 4y ago
When will the storage dashboard be open source?
- kiwicopple 4y ago
  Just caught up with the FE team on this one - it's pending in this PR: https://github.com/supabase/supabase/pull/10810 https://github.com/supabase/supabase/pull/10810 One of the engineers is testing this now and it should get merged within the hour. The build for amd64 takes about 6mins, and about 4 hours for arm64. I'll update here when it's merged (edit: it was actually merged earlier today, I originally linked to the wrong PR) For more context, the Storage UI was merged a couple of weeks ago but we simultaneously rolled out a new login screen for our Dashboard (for SSO) which then got rolled out to the Docker image. We've added a feature flag. Sorry for the delays - the lead up to launch week has everyone a bit stretched.
  - eclipxe 4y ago
    What a great comment. I applaud the transparency and depth. Nice.
no_wizard 4y ago
Any plans to switch to R2? It’s cheaper than S3 and would run in network with Cloudflare, allowing Supabase to take advantage of the CDN more optimally over time
- kiwicopple 4y ago
  It's one of the storage providers we'll investigate, now that R2 is GA. Our Storage server is hosted on AWS, so it wouldn't immediately save money (at least for egress) but we'll also investigate moving the servers to another cloud provider.
  - skrtskrt 4y ago
    Could this eventually be pluggable with bring-your-own s3-compatible provider? Would love this in front of my DigitalOcean object storage
    - kiwicopple 4y ago
      yes definitely. It should be easy to add s3-compatible providers (which are numerous, since s3 seems to be the defacto file storage API now). We haven't added "BYO" functionality to our hosted Platform because, frankly, we already spend a lot of time debugging various quirks and network issues - even for infra which we provide. Adding an extra level of opaqueness at our current team size would be very stressful from a support-perspective. That said, we're designing/developing so that it's possible and we'll continue to assess as the team grows.
worldmerge 4y ago
Hi, I know this is off topic but I would appreciate if there was a C++ library for the https://supabase.com/realtime https://supabase.com/realtime product. I would like to use it for sensors but no C++ library means I can't use it on my Arduino or ESP32s.

inian 4y ago

Supabase Engineer here There were a lot of primitives we built to ship image resizing - Storage events exposed via webhooks, rate limiting, a queue on top of Postgres, a smart CDN cache, object metadata endpoints, etc. These are already available if you are self hosting Supabase Storage, so that you can integrate your own CDN, listen to storage events, etc. Over the next few months, we will be working on exposing these on the Supabase platform too.

LukeLambert 4y ago
In nearly two decades developing for the web, I've never found a good reason to enlarge an image server-side, so thank you for anticipating that footgun. So many of your competitors get it wrong. However, it's unclear from the docs [0] if one of the most common use cases is supported: requesting an image at a specific aspect ratio while not exceeding a maximum size. For instance, I want a 1600×900px hero image at the top of my blog posts, but my source image is only 1200px wide. I request: transform: { width: 1600, height: 900, resize: 'cover' } Will the returned image be 675px tall, maintaining the 16×9 aspect ratio I want? A few more notes: The blog post states "default mode maintains aspect ratio and the resize mode is fill", but the docs say that cover is the default mode. The docs say, "If only one [width or height] parameter is specified, the image will be resized and cropped, maintaining the aspect ratio." I believe that with only one parameter, you could either resize or crop, but not both. I hope it's resize. For cropping modes, is it always from the center? Are there plans to support other "gravity" modes? [0] https://supabase.com/docs/guides/storage/image-transformations https://supabase.com/docs/guides/storage/image-transformatio...
- kiwicopple 4y ago
  > The blog post states "default mode maintains aspect ratio and the resize mode is fill", but the docs say that cover is the default mode. The default resize mode is cover and the blog post had a typo. Fixed now, thanks! > resize or crop We crop by default since the default mode is cover. > gravity we will be adding crop with gravity soon.. currently the crop is always from the center > requesting an image at a specific aspect ratio while not exceeding a maximum size. I'll get back to you on this one - Inian had to get some sleep, it's very late (early) for him

laurieg 4y ago

I love the product and this update. I wondered if there are any plans for handling audio (or even video) in a similar way?

inian 4y ago
Thank you! We will be working on video after we implement more image transformations. What is your usecase for audio and what transformations would you like to see?
- laurieg 4y ago
  My use case is in education, with users uploading short audio clips. Often these are in a variety of formats. It's nice to be able to re-encode them all to the same format and save as much bandwidth as possible as well as make sure they are supported in the weirdest variety of devices.

SpaghettiX 4y ago

Are images really this expensive to resize?: "it is $5 per additional 1000 images transformed.". Perhaps then we should ship a WebAssembly module for image resizing to the client and only validate image sizes server-side.