6 ms·

Improving Dropbox Performance: Retrieving Thumbnails

94 points by lowe 13y ago

cmicali 13y ago
tldr: use SPDY, but until more universal support a similar custom hack is proposed Neat hack though.. frustrating how much perf is lost due to HTTP sometimes
WhitneyLand 13y ago
I think they could still do better without using SPDY. Going to try and make time to test a different approach.
zoren 13y ago
Base64 and then gzip just to make it easier on javascript? Seems hard on dropbox servers. Why not use a multipart binary response?
- acdha 13y ago
  gzip is basically free on modern server hardware – if it makes life easier on mobile devices, this is a pretty easy tradeoff.
cstuder 13y ago
What about a sprites-like system? Since you pretty much know which pictures the user will request, just glue the next 100 thumbnails together into one big picture and then take it apart again on the device.
- elwell 13y ago
  https://news.ycombinator.com/item?id=7133035 https://news.ycombinator.com/item?id=7133035
herf 13y ago
Mostly you shouldn't have to hide backend latency by reordering, and if you don't reorder, you can just use JPEG image strips/sprites, which don't need double compression. JPEGs also stream, but in an order determined by the client, not the server. In my experience, Dropbox is quite a bit slower (by 10-20x) than services optimized for serving images - using 1500ms to deliver a 25k file is very slow on the web, but it is very common when requesting files via the Dropbox web API. (I can only speculate about the reasons, but Amazon's s3 has big latencies too.)
- ziga 13y ago
  We thought of spriting the thumbnails on the server, but this introduces a few problems: - All thumbs would have to be available on the server before the sprite can be encoded, which defeats progressive rendering. (Progressively encoding JPEG might be a possibility, but this isn't something we explored.) - The additional cost of encoding/decoding a large image on the server/client. -- Ziga (author)
  - herf 13y ago
    Yes, you can stream JPEG pretty easily as long as you know the output size. Also to combine JPEGs into a sprite, you can avoid all DCT/iDCT/color operations and use only the Huffman portion of the codec (decode, concat, encode). Since Huffman throughput is a considerable multiple of gzip (~10x?), I think you'd be quite a ways ahead, for memory and time.
  - WhitneyLand 13y ago
    1) Not necessarily, you may not even need progressive rendering if sprite batches bring it down to a manageable number of concurrent requests, right? 2) How is it any more work to encode? It's the same total number of pixels and the memory footprint isn't that large for server or client. I don't mean to dis your efforts, looks like you've done some nice work that performs well. I just think cloud/media management is an interesting problem with a lot of different ways to look at it.
zaptheimpaler 13y ago
Basically, they chunk thumbnails - client asks for the next 10 thumbnails in one request, server sends back 10 thumbnails in the response.
drcross 13y ago
On the functionality end of things I think dropbox should be stop new feature development until heavy UX is done with large unbiased test groups. The application is simple and it works pretty well but recently I've noticed different barriers to use, notifications that I'd prefer not to see just getting in the way. It's a result of Zawinski's law, it's happening to gmail where they think adding more bells and whistles is needed but it's not, in many case user experience declines, for example playing a youtube clip now blacks out the rest of your gmail page while you watch it. Application developers need to learn that less is more.
- dropboxer 13y ago
  hi there, I work at dropbox and am curious to learn more. what things did you run into that were unexpected and retracted from the UX? hopefully we can make it better!
tinganho 13y ago
I just see one problem with this solution. You can't create Retina images out of base64 strings on any web browsers today.
- madeofpalk 13y ago
  > You can't create Retina images out of base64 strings Why not? There's no such thing as a 'retina image', they're just a larger resolution. There's nothing stopping you from requesting a @2x set of thumbnails
- simonw 13y ago
  Are you sure you can't just serve up a 400x400px base64 image and put it in an image element with a height and width of 200?
erikpukinskis 13y ago
I don't really use Dropbox, but I was randomly browsing through mine looking for a photo the other day and it was a "holy shit" moment. It's really fast to scroll through a huge number of photos. I had been scrolling through Flickr a few minutes before and it was a massive difference. Flickr was very much a scroll-and-wait experience.
- rhizome 13y ago
  For me, Flickr is basically unusable. Slideshows are impossibly slow, information is positioned outside the browser viewport, and simply bringing up an image-page is reminder of dialup. I can only blame inertial lockin for their usage numbers.
b0b0b0b 13y ago
Were you already serving thumbnails over https? Or were they served over http? edit: I was also wondering whether you can skip downloading some thumbnails based on the velocity of the scroll.
- ziga 13y ago
  We were already serving over https (we've implemented HSTS some time ago). And good observation about scrolling -- we do in fact queue, prioritize, and skip thumbnails based on scrolling.

94 points by lowe 13y ago

tldr: use SPDY, but until more universal support a similar custom hack is proposed Neat hack though.. frustrating how much perf is lost due to HTTP sometimes

WhitneyLand 13y ago

I think they could still do better without using SPDY. Going to try and make time to test a different approach.

zoren 13y ago

Base64 and then gzip just to make it easier on javascript? Seems hard on dropbox servers. Why not use a multipart binary response?

acdha 13y ago
gzip is basically free on modern server hardware – if it makes life easier on mobile devices, this is a pretty easy tradeoff.

cstuder 13y ago

What about a sprites-like system? Since you pretty much know which pictures the user will request, just glue the next 100 thumbnails together into one big picture and then take it apart again on the device.

elwell 13y ago
https://news.ycombinator.com/item?id=7133035 https://news.ycombinator.com/item?id=7133035

herf 13y ago

Mostly you shouldn't have to hide backend latency by reordering, and if you don't reorder, you can just use JPEG image strips/sprites, which don't need double compression. JPEGs also stream, but in an order determined by the client, not the server. In my experience, Dropbox is quite a bit slower (by 10-20x) than services optimized for serving images - using 1500ms to deliver a 25k file is very slow on the web, but it is very common when requesting files via the Dropbox web API. (I can only speculate about the reasons, but Amazon's s3 has big latencies too.)

ziga 13y ago
We thought of spriting the thumbnails on the server, but this introduces a few problems: - All thumbs would have to be available on the server before the sprite can be encoded, which defeats progressive rendering. (Progressively encoding JPEG might be a possibility, but this isn't something we explored.) - The additional cost of encoding/decoding a large image on the server/client. -- Ziga (author)
- herf 13y ago
  Yes, you can stream JPEG pretty easily as long as you know the output size. Also to combine JPEGs into a sprite, you can avoid all DCT/iDCT/color operations and use only the Huffman portion of the codec (decode, concat, encode). Since Huffman throughput is a considerable multiple of gzip (~10x?), I think you'd be quite a ways ahead, for memory and time.
- WhitneyLand 13y ago
  1) Not necessarily, you may not even need progressive rendering if sprite batches bring it down to a manageable number of concurrent requests, right? 2) How is it any more work to encode? It's the same total number of pixels and the memory footprint isn't that large for server or client. I don't mean to dis your efforts, looks like you've done some nice work that performs well. I just think cloud/media management is an interesting problem with a lot of different ways to look at it.

zaptheimpaler 13y ago

Basically, they chunk thumbnails - client asks for the next 10 thumbnails in one request, server sends back 10 thumbnails in the response.

drcross 13y ago

On the functionality end of things I think dropbox should be stop new feature development until heavy UX is done with large unbiased test groups. The application is simple and it works pretty well but recently I've noticed different barriers to use, notifications that I'd prefer not to see just getting in the way. It's a result of Zawinski's law, it's happening to gmail where they think adding more bells and whistles is needed but it's not, in many case user experience declines, for example playing a youtube clip now blacks out the rest of your gmail page while you watch it. Application developers need to learn that less is more.

dropboxer 13y ago
hi there, I work at dropbox and am curious to learn more. what things did you run into that were unexpected and retracted from the UX? hopefully we can make it better!

tinganho 13y ago

I just see one problem with this solution. You can't create Retina images out of base64 strings on any web browsers today.

madeofpalk 13y ago
> You can't create Retina images out of base64 strings Why not? There's no such thing as a 'retina image', they're just a larger resolution. There's nothing stopping you from requesting a @2x set of thumbnails
simonw 13y ago
Are you sure you can't just serve up a 400x400px base64 image and put it in an image element with a height and width of 200?

erikpukinskis 13y ago

I don't really use Dropbox, but I was randomly browsing through mine looking for a photo the other day and it was a "holy shit" moment. It's really fast to scroll through a huge number of photos. I had been scrolling through Flickr a few minutes before and it was a massive difference. Flickr was very much a scroll-and-wait experience.

rhizome 13y ago
For me, Flickr is basically unusable. Slideshows are impossibly slow, information is positioned outside the browser viewport, and simply bringing up an image-page is reminder of dialup. I can only blame inertial lockin for their usage numbers.

b0b0b0b 13y ago

Were you already serving thumbnails over https? Or were they served over http? edit: I was also wondering whether you can skip downloading some thumbnails based on the velocity of the scroll.

ziga 13y ago
We were already serving over https (we've implemented HSTS some time ago). And good observation about scrolling -- we do in fact queue, prioritize, and skip thumbnails based on scrolling.