3 ms·

Do you find Dropbox works for you with a giant library? I have ~1.5Tb of photos at this point, which seems too big for any of their accounts. How do you have it

by Roedou 12y ago

Do you find Dropbox works for you with a giant library? I have ~1.5Tb of photos at this point, which seems too big for any of their accounts. How do you have it set up?

kingnight 12y ago
Ha, is "giant" relative or what... I am currently at ~98% of my 215GB Dropbox account and am stressing about what to do. The next tier doesn't seem cost effective for me and I don't have much room for clearing house. One thought I have is converting a sizable portion of old Canon CR2 raw files to adobe DNG since I've noticed that reduces filesize by roughly 20% for the files that I have. This isn't really a long term solution though and I'm actually kind of unsure of what to do. I keep hoping that I can hold out long enough until Dropbox's gb/price get's halved again. Then I'd likely be set for some time. I really can't think of a cloud service for ~1.5TB of photos. Have you found anything that would suit your purposes? I also use crashplan for offsite archiving of my photo library, but I really, really, like having the entirety of my photo catalogue accessible everywhere. Excessive, sure, but definitely nice. With loom, I am hoping they integrate the jpg rendering of raw files into their photos views on web/mobile.
- ChuckMcM 12y ago
  What is your access pattern on that 215GB? Is there any of it that you don't touch for at least a month? There has been a lot of work in the past on hierarchical storage managers for Enterprise types but perhaps there is a market for one for regular users. Something that moves your data constantly into S3 or Glacier and then if you haven't used it on DropBox for a month it deletes it there, only to bring it back if you look for it.
  - kingnight 12y ago
    There is definitely a huge drop off for access to files after a certain age. I'd say after a month there is really very little need for me to access them directly. After that, it is sporadic for when I would want a specific file. A benefit of Dropbox/Carousel (+ Lightroom locally), is I can fly through all 215GB via thumbnail representations without having to call the file explicitly. If there were some mechanism to offload storage to s3 or glacier, while retaining the nice UI afforded by the Dropbox/Lightroom clients, I would be all over that.
- rhizome 12y ago
  You can just do a straight S3 mount with FUSE, which will get you 1.5TB for what looks like under $50/mo https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse There's also Syncany for a more client-oriented approach: https://github.com/binwiederhier/syncany https://github.com/binwiederhier/syncany
- georgemcbay 12y ago
  This may or may not be relevant to you but for a long time I was auto-converting my CR2 files to DNG during Lightroom import (without embedding the original CR2 in the DNG since I was doing it for space savings as you are thinking of doing) and now I kind of regret it because the "Digital Lens Optimizer" that ships with recent versions of Canon's DPP is actually really good at pulling out image detail that Lightroom or ACR doesn't give me in the demosaicing process on photos taken with Canon lenses (since only those have 'DLO' profiles). And DPP doesn't support DNG loading. Granted, Lightroom/ACR still do a very good job of rendering RAW files, so the difference isn't that great without pixel-peeping, but it might be something you want to think about when it comes to switching formats. I've since gone back to importing the plain CR2s as my on-disk RAW format -- in practice CR2s are more universally useable than DNG files since not everything can read DNG whereas since you can always go from CR2->DNG, everything that can read DNG is covered by CR2s.
  - kingnight 12y ago
    Very relevant. Thanks for sharing that. I definitely had bought into the adobe dng idea of it being the superior format for compatibility, but didn't consider there might be situations where something could read CR2 better than dng.
- hrktb 12y ago
  My library is not as big as 1.5T but it won't fit in any standard cloud offering as well. One of the advice I received was to batch export to JPG to have a small and easily sharable copy, and keep all the originals in Glacier or other cost effective backup that doesn't need to be accessed too often. I haven't done it yet but it seems to be a reasonable approach.

by Roedou 12y ago

Do you find Dropbox works for you with a giant library? I have ~1.5Tb of photos at this point, which seems too big for any of their accounts. How do you have it set up?

kingnight 12y ago

Ha, is "giant" relative or what... I am currently at ~98% of my 215GB Dropbox account and am stressing about what to do. The next tier doesn't seem cost effective for me and I don't have much room for clearing house. One thought I have is converting a sizable portion of old Canon CR2 raw files to adobe DNG since I've noticed that reduces filesize by roughly 20% for the files that I have. This isn't really a long term solution though and I'm actually kind of unsure of what to do. I keep hoping that I can hold out long enough until Dropbox's gb/price get's halved again. Then I'd likely be set for some time. I really can't think of a cloud service for ~1.5TB of photos. Have you found anything that would suit your purposes? I also use crashplan for offsite archiving of my photo library, but I really, really, like having the entirety of my photo catalogue accessible everywhere. Excessive, sure, but definitely nice. With loom, I am hoping they integrate the jpg rendering of raw files into their photos views on web/mobile.

ChuckMcM 12y ago
What is your access pattern on that 215GB? Is there any of it that you don't touch for at least a month? There has been a lot of work in the past on hierarchical storage managers for Enterprise types but perhaps there is a market for one for regular users. Something that moves your data constantly into S3 or Glacier and then if you haven't used it on DropBox for a month it deletes it there, only to bring it back if you look for it.
- kingnight 12y ago
  There is definitely a huge drop off for access to files after a certain age. I'd say after a month there is really very little need for me to access them directly. After that, it is sporadic for when I would want a specific file. A benefit of Dropbox/Carousel (+ Lightroom locally), is I can fly through all 215GB via thumbnail representations without having to call the file explicitly. If there were some mechanism to offload storage to s3 or glacier, while retaining the nice UI afforded by the Dropbox/Lightroom clients, I would be all over that.
rhizome 12y ago
You can just do a straight S3 mount with FUSE, which will get you 1.5TB for what looks like under $50/mo https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse There's also Syncany for a more client-oriented approach: https://github.com/binwiederhier/syncany https://github.com/binwiederhier/syncany
georgemcbay 12y ago
This may or may not be relevant to you but for a long time I was auto-converting my CR2 files to DNG during Lightroom import (without embedding the original CR2 in the DNG since I was doing it for space savings as you are thinking of doing) and now I kind of regret it because the "Digital Lens Optimizer" that ships with recent versions of Canon's DPP is actually really good at pulling out image detail that Lightroom or ACR doesn't give me in the demosaicing process on photos taken with Canon lenses (since only those have 'DLO' profiles). And DPP doesn't support DNG loading. Granted, Lightroom/ACR still do a very good job of rendering RAW files, so the difference isn't that great without pixel-peeping, but it might be something you want to think about when it comes to switching formats. I've since gone back to importing the plain CR2s as my on-disk RAW format -- in practice CR2s are more universally useable than DNG files since not everything can read DNG whereas since you can always go from CR2->DNG, everything that can read DNG is covered by CR2s.
- kingnight 12y ago
  Very relevant. Thanks for sharing that. I definitely had bought into the adobe dng idea of it being the superior format for compatibility, but didn't consider there might be situations where something could read CR2 better than dng.
hrktb 12y ago
My library is not as big as 1.5T but it won't fit in any standard cloud offering as well. One of the advice I received was to batch export to JPG to have a small and easily sharable copy, and keep all the originals in Glacier or other cost effective backup that doesn't need to be accessed too often. I haven't done it yet but it seems to be a reasonable approach.