3 ms·

Ask HN: The best way store thousands of Images on a server

2 points by barredo 18y ago

Hello. I've got a server of my own handling the multimedia files for a website. It has around 200Gb of images (all JPG's) stored in nested folders. I have Apache 2 on Linux and nothing else. I don't know if it is the best solution, but I think is not. What system (http server, cache, or anything) would you recommend me?

leej 18y ago
dont forget nginx
catch23 18y ago
Amazon S3
- barredo 18y ago
  I have my own server to do that. With 1TB of transfer a day, I cannot afford Amazon S3 (I have my own 200mbps low latency connection) Thanks, but I was looking for technology to install on my own server. Not moving to Amazon S3
pclark 18y ago
http://aws.amazon.com/cloudfront/ http://aws.amazon.com/cloudfront/
- barredo 18y ago
  30 days a month * 1024 Gb of transfer a day * 0,17$/Gb = more than 5,000$ a month. Am I right? My own 200mbps connection costs me the equivalent of 300$ a month.
  - frisco 18y ago
    A /terabyte/ of transfer a day?!?
    - barredo 18y ago
      Yes. It has a lot of traffic. Btw, it's not porn! :-)
mqatrombone 18y ago
A smaller web server like lighttpd or cherokee may be a better choice if all you are serving is images.
kierank 18y ago
MogileFS might be what you're looking for: http://www.danga.com/mogilefs/ http://www.danga.com/mogilefs/ with an nginx or lighttpd frontend. The documentation is a little sparse though the mailing list and irc are useful for help. Also it would somewhat depend on your traffic profile. If you've got thousands of random reads (e.g. thumbnails with dozens per page, ssds might be better; specifically the Intel X-25). Otherwise MogileFS with an array of fast drives would probably do the job. Mogile can also do load balancing based on drive I/O because each image is replicated across different files/machines. EDIT: Make sure you've got browser caching set up correctly (though admittedly that's a tiny bit harder with Mogile) If you can use a expiry time that's far in the future. Also don't send a Last-Modified nor an ETag Header. Only send Expires.
blurry 18y ago
Check out this Stanford presentation Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos http://www.flowgram.com/p/2qi3k8eicrfgkv/ http://www.flowgram.com/p/2qi3k8eicrfgkv/
soult 18y ago
I recommend gatling: http://www.fefe.de/gatling/ http://www.fefe.de/gatling/ It's the fastest httpd I know of. You might also want to try thttpd: http://www.acme.com/software/thttpd/ http://www.acme.com/software/thttpd/ I heard all the porn sites use it to host their images. With only the http server for the images running it should work quite well to rely on the filesystem cache.
iuguy 18y ago
How about Publicfile? - http://cr.yp.to/publicfile.html http://cr.yp.to/publicfile.html Super duper secure and very, very quick, especially for smaller files. I'd try and run some tests against it, thttpd and your existing apache setup.

2 points by barredo 18y ago

leej 18y ago

dont forget nginx

catch23 18y ago

Amazon S3

barredo 18y ago
I have my own server to do that. With 1TB of transfer a day, I cannot afford Amazon S3 (I have my own 200mbps low latency connection) Thanks, but I was looking for technology to install on my own server. Not moving to Amazon S3

pclark 18y ago

http://aws.amazon.com/cloudfront/ http://aws.amazon.com/cloudfront/

barredo 18y ago
30 days a month * 1024 Gb of transfer a day * 0,17$/Gb = more than 5,000$ a month. Am I right? My own 200mbps connection costs me the equivalent of 300$ a month.
- frisco 18y ago
  A /terabyte/ of transfer a day?!?
  - barredo 18y ago
    Yes. It has a lot of traffic. Btw, it's not porn! :-)

mqatrombone 18y ago

A smaller web server like lighttpd or cherokee may be a better choice if all you are serving is images.

kierank 18y ago

MogileFS might be what you're looking for: http://www.danga.com/mogilefs/ http://www.danga.com/mogilefs/ with an nginx or lighttpd frontend. The documentation is a little sparse though the mailing list and irc are useful for help. Also it would somewhat depend on your traffic profile. If you've got thousands of random reads (e.g. thumbnails with dozens per page, ssds might be better; specifically the Intel X-25). Otherwise MogileFS with an array of fast drives would probably do the job. Mogile can also do load balancing based on drive I/O because each image is replicated across different files/machines. EDIT: Make sure you've got browser caching set up correctly (though admittedly that's a tiny bit harder with Mogile) If you can use a expiry time that's far in the future. Also don't send a Last-Modified nor an ETag Header. Only send Expires.

blurry 18y ago

Check out this Stanford presentation Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos http://www.flowgram.com/p/2qi3k8eicrfgkv/ http://www.flowgram.com/p/2qi3k8eicrfgkv/

soult 18y ago

I recommend gatling: http://www.fefe.de/gatling/ http://www.fefe.de/gatling/ It's the fastest httpd I know of. You might also want to try thttpd: http://www.acme.com/software/thttpd/ http://www.acme.com/software/thttpd/ I heard all the porn sites use it to host their images. With only the http server for the images running it should work quite well to rely on the filesystem cache.

iuguy 18y ago

How about Publicfile? - http://cr.yp.to/publicfile.html http://cr.yp.to/publicfile.html Super duper secure and very, very quick, especially for smaller files. I'd try and run some tests against it, thttpd and your existing apache setup.