13 ms·

Storage for Photographers, Part 2

205 points by PStamatiou 11y ago

magic5227 11y ago
For those who don't need quite as much storage (12tb!) I recommend trying Apple's new iPhoto library. It has been pretty close to a dream solution for me. Now I can take photos on a trip, upload them via my iPad. Make edits to RAW files, curate, and it syncs every time I can connect to wifi, backing up my originals and syncing across all changes to any Apple device (and web).
- digikata 11y ago
  Do you backup the library to something outside of iPhoto? I've been wondering if there's a nice automatable way to backup that library onto a home Linux computer.
  - magic5227 11y ago
    Yes I also keep my own backups. I use Crashplan for the iphoto library, as well as a local SSD backup. Fortunately iPhoto libraries are just folders so by backing it up you can always take out the folder of originals and put them in Google or anything else.
clintonb 11y ago
> Ever since my first iPhone in 2007, I've been keeping every mobile phone photo and video I've taken. Have you ever considered deleting some of the photos or videos? I shoot headshots along with my personal projects (RAW files are 20MB+), and I don't generate anywhere close to 1TB of data per year. This is primarily because I only save what I need or think I will need later.
magic5227 11y ago
As a side note I think this brings up another interesting problem than just dealing with the storage, which is, how to enjoy so many photos in a lifetime. Personally, I decided to severely restrict how many images I keep after a trip, so I'm more likely to actually view them years from now.
- wlesieutre 11y ago
  I'm tending to keep all of my originals in case I ever want to flip through the full set, but I've been tagging a smaller portion into either a "good photography" or "good family/etc memory" categories. Then I can easily filter down to a smaller set of photos. Then again, I haven't been taking a huge amount of photos lately. If I ramp it back up, maybe I'll need to reconsider.
- acomjean 11y ago
  I'm kinda up in the air about how to display the images I've sorted and edited the set down to something reasonable. I was a photo editor at my college paper and each event gets one photo you get practice cutting down the photo sets. I send some to flickr, some google+ and so on. I need a more consistent strategy. I uploaded to google plus a couple months back and it asked to create a "story". I really like the presentation of photos. Google makes it hard to link and share but the results for a group of photos are good. The example: https://plus.google.com/115477748015409107850/stories/4b3ddf5f-f9d5-3db0-90a6-f2b96722be6714cd14331ca?authkey https://plus.google.com/115477748015409107850/stories/4b3ddf... Same link, differnt url. (sigh... google) https://plus.google.com/+AramComjean/posts/ig1cSZAvt9Q https://plus.google.com/+AramComjean/posts/ig1cSZAvt9Q
- calinet6 11y ago
  Shrug. I've got about 1.5 TB from the last ten or twelve years, and I do often go back through the organized albums and events. Often it's to find something specific, often just looking for something good, often just for nostalgia. I'm quite glad I have them, and rating and filtering is just as good as tossing them out in my experience.
- matwood 11y ago
  My workflow after a photo session is: * Quickly scan pics and delete anything that is flat out bad. I try to get these on the spot, but some slip through. * Go through again and rate. * Go through again and adjust ratings further. * Post process the 5 stars, and put them up to view. * Everything else is saved through an offline backup service. I find that the 5 star pics end up being great covers into an event. Often the 5 star pics are good enough, but sometimes they draw me in the look at the rest. The only downside is you have to be ruthless with your ratings.
aaronbrethorst 11y ago
How about taking fewer photos? Do you really need two nearly identical photos of a hand reaching for food in a Japanese restaurant, or three photos of a bowl of miso soup? On my last trip, I shot four rolls of 120 film (645 format, so 64 photos total), and maybe 500 digital photos, of which I deleted 90%. I still save about 5,000 RAW files per year, or about 100GB/year, but this is a relatively manageable amount. I'm approaching the saturation point on my iMac's hard drive, but should be able to last for another several years by shipping the oldest RAWs off to Glacier and keeping high-res JPGs around locally for reference. edit: also, next time you're up here in Seattle, you should visit the Jose Rizal Bridge at sunset, and the observatory on the 73rd floor of Columbia Tower.
- PStamatiou 11y ago
  Thanks for the Seattle tip! I'm up there somewhat often as part of my team is there. I have done the Columbia Tower but not that bridge. As for taking fewer photos, I like to capture as much as possible and then I distill that down later into my photo sets/stories. I want a big funnel to choose from and not have to do that on the fly. This is my hobby and passion and I love it. :) As for the nearly identical photos aspect - I always take a few shots & similar compositions of the same thing to make sure that at least one shot is super in focus.
  - aaronbrethorst 11y ago
    I always take a few shots & similar compositions of the same thing to make sure that at least one shot is super in focus. I get it, I do the same thing when I shoot digital. I'm just recommending that those photos never make it past the Previous Import section in Lightroom. If you're aiming for tack-sharp focus, it should be quite straightforward to do a very quick culling then. Also, please say hi to Chris Burrows for me; I haven't seen him since we worked on Visual Studio together about a million years ago :)
    - PStamatiou 11y ago
      Whoa you know Chris! He's on my team and we have standups every day. Amazing developer with a great design and product sense!
Obi_Juan_Kenobi 11y ago
> Where on earth do normal people store that much data? They don't. Normal people realize that keeping that many photos is a negative value proposition; it's a burden rather than an asset. Rather than trying to hold on to everything, they choose what is actually worth revisiting. This could mean they're a little more selective with the shutter, or they only keep the photos they like. By all means, keep everything if you want to, but I find it troublesome that storage is considered the problem, rather than making no attempt to cull images. Clearly a great deal of effort has gone into these images to make his various travelogues, so the effort to see which ones make the cut has been done already. That said, it doesn't seem like a NAS was even needed here. Any reasonable PC MoBo is going to have 6 SATA ports, so you can quite easily make a 4 drive RAID 5. Personally, I just RAID 1 a couple 3TB drives and call it good.
- chinhodado 11y ago
  I share the same sentiment. I have friends who would, on a trip, shoot everything they see, and shoot each scene a dozen times (literally) with little variation in angle, exposure, etc. They do this on maximum resolution, highest quality settings (Fine JPEG, even RAW). After a trip, they dump everything into their storage and hardly ever look at what they've taken again. And they would constantly complain about not having enough storage.
  - philliphaydon 11y ago
    I'll go to the Zoo or somewhere and take 1000s of photos, then spend hours in Lightroom deleting maybe 80% of that till I have ~100 I keep and ~20 I edit. I don't understand why people keep the same shot 23423352 times.
    - chinhodado 11y ago
      I think it's because of the mindset "I can revisit what I have taken and make my selection later. I don't want to lose any memory. Storage is cheap." Over time, it becomes a burden to maintain all that.
    - 11y ago
distantsounds 11y ago
If you're willing to pay for Amazon Prime, you get unlimited cloud storage for all your photos. RAW included. The only method to upload/download is via the clunky web UI, but if reliable storage is what you're after, it's a pretty hard bargain.
- detaro 11y ago
  Assuming you have the bandwidth to do that comfortably. And you still need backups somewhere.
- jlintz 11y ago
  problem is they don't support syncing folders. You'd have to keep track of what you've uploaded already. Also I haven't been able to get it to successfully upload all my photos, seems to always stall out after a few hundred photos or eventually hit some sort of rate limit that goes to a crawl
- alkonaut 11y ago
  A "real" cloud backup (That is, not just cloud storage) like CrashPlan is just a few dollars per month. I pay less than $3 per computer per month for backup of several TB with history for each file. I just couldn't imagine it would be worth buying the hardware or configuring my own backup to make a non-cloud equivalent.
  - NigsMcGee 11y ago
    Crashplan recently removed it's multi-year plan and increased prices. Gone are the days of 4-year plans that average out to a few dollars per month. Crashplan's prices have nearly doubled in the past three years alone. Disappointing.
    - alkonaut 11y ago
      How I reach as low as $3 per computer per month is through the family plan which is something like $12-13 but allows up to 10 computers. Not only does it back up my photos, it also solves the "I clicked OK in some dialog and now the computer behaves funny" of relatives.
Veratyr 11y ago
Wow, that was an amazing article. So comprehensive! A few things I'd like to add though: - Mylio (http://mylio.com http://mylio.com) is very helpful for syncing your collections across things. Not for everyone but it's worth a look to see if it works. Best thing for me is that it's peer to peer so I don't have to upload my collection to a cloud service to access it on all my devices. It does offer cloud but it's end to end encrypted (allegedly). Best thing is that it lets you configure whether you sync previews, thumbnails or originals to each device and even which photos to sync. Really handy if you want a new shoot on your phone to play with on the train or something. - You can use Google Drive to get your photos into Google Photos. This lets you keep a bit exact backup in Drive while using your quota for Photos as well. Further, Google Apps for Work Unlimited, through a glitch or deliberately I don't know, offers unlimited Drive storage to accounts even with a single user in their organisation. I pay $10/month for unlimited Google Drive storage. It's advertised as being 1TB for single users so I'm not sure if this is a bug but thought it was worth mentioning. - Google Photos will (quite helpfully) use the JPEG previews you embed in a DNG photo so if you tend to touch up things in Lightroom, embedding them will ensure that Google Photos displays things in the same way. Really though, this was an amazingly comprehensive article. Thanks for posting!
dperfect 11y ago
> The bad news is that I have over 1TB of photos and the next pricing tier after 1TB ($9.99/mo) is 10TB and that costs a whopping $99.99 per month. So I use Google Photos with the free compressed setting. I don't actually mind since I have my own file backup solution and I use it more for that added layer of intelligence, convenience and utility. I've tried doing as suggested here, but here's the problem: if it's too expensive to store your RAW images on the cloud (which it is for most people), then your cloud photo library is really just a proxy of your library. That's not to say it isn't useful, but unless it's synchronized both ways with the original files, you're just asking for disorganization - from my experience anyway. You have to be careful and basically only touch the originals to let changes propagate one-way to the cloud proxy. If it isn't hosting your RAW files, then it doesn't fill the role of a backup, and if your edits or tagging on the cloud aren't applied back to your originals, then any time/effort you put into organizing and editing your library on the cloud is somewhat wasted. To be honest, I do use Apple Photo Streams for something similar - but I just treat it as a convenience for low-quality output/viewing of recent photos, mostly from my phone. Even then, the Photo Stream part is still a mess. My main library resides on a NAS (as in the article), is backed up to 2 low-cost cloud backup providers (still much cheaper than hosting a single RAW copy on Google Photos), and I use Lightroom for all actual editing and tagging since it applies to the authoritative library. My point is - I too look forward to the day where at least one copy of my full library can be hosted on something like Google Photos in the cloud, but we're still a ways off from that being practical. Google Photos can be a convenience in some ways, but at current prices it really doesn't fit what I'm looking for.
- ValentineC 11y ago
  > My main library resides on a NAS (as in the article), is backed up to 2 low-cost cloud backup providers (still much cheaper than hosting a single RAW copy on Google Photos), and I use Lightroom for all actual editing and tagging since it applies to the authoritative library. I'm doing something similar, but just backing up to CrashPlan (which has a flat fee for backing up). What cloud providers are you using?
  - dperfect 11y ago
knurdle 11y ago
Great article, I've had a similar setup for years now. An 8 year old readynas that's still running, it's really slow but it works. I do a sync with that and a local desktop with a big external drive. And then I back up that local desktop to crashplan. I have a comment about the drives, I generally like purchasing different manufacturer drives for my NAS when I'm buying them in bulk. I always worry about multiple drives from the same batch failing around the same time. It's happened to me before so now I'll buy similar capacity drives but from different models or mfrs.
- devNoise 11y ago
  I had a ReadyNAS NV before they were bought by Netgear and though it was good. Unfortunately, I had more drives fail in the ReadyNAS than the computers I was making backups from. As time went on it became hard to find drives that were on the compatibility list.
jly 11y ago
Great article...lots of comments here about how you shouldn't keep that much data. I agree with that for hobbyists, but this is pretty relevant for professionals where the option to cull doesn't extend that far and there are business reasons for retaining terabytes of photos for an extended period. Additionally, next-gen cameras with 40-50MP RAWs are right around the corner. I use a similar combination of home-built network storage (no RAID - just manual multiple backups) and glacier for offsite redundancy. Dealing with images as a business, I typically only work on a couple shoots at a time so syncing across devices is not a big concern, but long-term archiving and redundancy is.
- munificent 11y ago
  > Additionally, next-gen cameras with 40-50MP RAWs are right around the corner. The future is here! Canon's 50.6 MP 5DS is shipping now.
  - PStamatiou 11y ago
    I just preordered the Sony A7r II: "only" 42MP but DxoMark ranks the sensor higher than the Canon.
    - varikin 11y ago
      These are not even counting scanning negatives (can be massive) or digital backs for medium format cameras. Hasseblad had a 60+MP back 10 years ago just because the sensor was larger.
    - mikeyouse 11y ago
      That looks like a monster of a camera, can't wait to see what you do with it.
      - PStamatiou 11y ago
        same here :) I'm sure I will write about it in a future post or photoset.
imaginenore 11y ago
As a photographer, he is doing it wrong. You should never ever ever store every shot you take. Your job as a photographer is to delete 95-99% of the shots. Select the best, delete the rest. The exception to that is the commercial client photos, but that's not what he is talking about. If you're going to save 275GB of photos per trip, it's guaranteed you will never look through them again. Look at his photos - I don't see a single shot I would have saved (well maybe the fisheye one from Hawaii). He has 3-4 nearly identical copies of each shot. Tons of boring stuff.
- PStamatiou 11y ago
  I keep everything, cull down to the best and post those online. I still like to keep the original set. Maybe down the line I'll be more rigorous here but sometimes I will make little movies out of every individual still, so that's where it comes in handy: https://www.youtube.com/watch?v=8fR4MjImSU0 https://www.youtube.com/watch?v=8fR4MjImSU0
- mpdehaan2 11y ago
  Yep, makes for an online gallery that isn't any fun to look at. Perhaps common sense, but I recommend remembering to print photos (say 4x6") you really care about (works especially if you lose your backup) and also hang some of the stuff you really care about it (because you're unlikely to look at them electronically). Take advantage of your walls, if you have a event decent camera, seeing your stuff 10x14, 16x20 or even 20x30 is also exceptionally worth it. I'm kind of fond of mpix standouts which to me look a lot better than seeing things behind glass - http://www.mpixpro.com/Catalog.aspx/standouts http://www.mpixpro.com/Catalog.aspx/standouts
  - jscheel 11y ago
    I agree 100%. Printing your photos really helps elicit a more visceral response. That said, what is mpix? I looked at it, but it wanted me to go through an approval process to create and account to see the prices.
    - mpdehaan2 11y ago
ScottBurson 11y ago
I considered a NAS, but I had a Linux server already, so I just upgraded it with 6 4TB drives and installed ZFS [0]. I'm using raidz2, which is doubly redundant like RAID-6. For long-term backup I just got a BD-R drive with M-Disc support [1]; haven't tried it yet. This still makes more sense to me than trying to store it all in the cloud. [0] http://zfsonlinux.org/ http://zfsonlinux.org/ [1] http://www.mdisc.com/ http://www.mdisc.com/
- lutorm 11y ago
  This is my solution, too. And then I push periodic zfs snapshots to an off-site server. It's not fast given my upload speed with TWC, but it's doable.
  - StavrosK 11y ago
    Try attic, you'll thank me.
NateDad 11y ago
Everyone needs to check out https://smugmug.com https://smugmug.com - for $40 a year you get unlimited photo and video uploads (yes, RAW files are ok). It's like flikr with no ads, only your own stuff on the pages, super customizable (with a power account - $60 a year - you can point your own URL at your account and no one even needs to know the files are on smugmug. It's a site designed for professional photographers that also happens to have kickass features for casual photographers as well. There's even lightroom plugins to upload directly to smugmug. One of the best features, in my opinion, is the customer support. They respond within a couple hours. They're super nice and knowledgeable. I've had an account at smugmug for nearly a decade, and would never go back to free services. Google photos is nice, but the size limitation is definitely a problem for prolific photographers. (I don't work for smugmug nor do I know anyone who works there, I just love their service.)
- georgemcbay 11y ago
  As a pretty serious hobbyist photo-taking-guy I'm a big fan of smugmug, my "sharing photos with friends and family" photo website (http://gmcbay.com http://gmcbay.com) is hosted with them, but I don't think of them or use them as a good long-term file "storage"/backup solution. All of the photo-specific type of storage solutions (Google, Amazon, Smugmug, Adobe's CC thing, etc) are kind of flawed for many reasons (some outlined in the OP blogpost) when used as what I think of as "digital negative storage", IMO. I just store all of my (original, RAW-format) photos/videos on giant harddrives inside my desktop computer and have that backed up automatically and continuously via backblaze. Local storage is ridiculously cheap, and so are offsite automatic backups. For people who just deal with jpg files from a phone camera or whatever then YMMV and these photo-specific services may be just the thing, but I've yet to try one that doesn't feel like it is getting in your way if your normal workflow is big RAW files with processing via Lightroom/Photoshop, etc.
  - fuzzywalrus 11y ago
    Indeed, I still use an aging 2008 Mac Pro and the benefits of internal storage are pretty profound. I have two Time Machine HDDs (An Internal + an External), which counts for two of my six HDDs. I'm debating going towards a service like blackblaze as I have a decade of photos on my computer.
beachstartup 11y ago
i'm so glad i wasn't born with the digital hoarder gene. apparently my absolute-zero proclivity to hang onto photographs and other media is extremely rare. i'm just not into it. i'll snap some stuff on my phone to share with my close friends, and then let it get deleted or whatever. i don't upload it to socialmedia. i just don't give a shit, to be honest. we're surrounded by this stuff day in and day out on every screen we have, i make no effort to keep any of it longer than a few weeks. in fact i have a problem getting rid of old photos, somehow they seem to follow me around on device to device through no fault of my own! they're almost like viruses.
fencepost 11y ago
The thing that kind of jumped out at me was the RAID5 configuration - I can't imagine doing that with 3TB drives unless he really does have everything on there also backed up to another location.
calinet6 11y ago
Just a +1 for the Synology NAS products. I'm frequently amazed every time I use it how clean the UI is and how they basically implemented a full window manager for the underlying Linux system in a web app. It's exactly what a NAS should be: powerful and flexible, yet easy to set up and use, but rock solid even if it just sits there. Lots of great things to say about using it in nearly exactly this setup for about a year now.
- CamperBob2 11y ago
  Everything you say is 100% true, but be sure to keep the OS up to date if you expose any IP addresses (security cameras, etc) to the world at large. NAS boxes are just about the highest of high-value targets.
  - calinet6 11y ago
    Synology DSM has great auto-update systems :) But, absolutely correct.
AbeEstrada 11y ago
My storage solution is Blue-ray discs + External HDD + AWS Glacier. Once I use the RAW files, I rarely need them, that's why I store them in Blue-ray discs as a backup.
jscheel 11y ago
Culling is an important part of the photographic process. However, far be it from me to tell another photographer what they should and shouldn't keep. We all have our quirks. Good luck convincing me to delete even a blurry photo of my daughter :) Storage is still a huge problem for photographers. There has got to be a better/easier solution than even this process. It's still outside the realm of a lot of photographers' skillsets.
- slantyyz 11y ago
  >> Culling is an important part of the photographic process. It is indeed important, but it is hard. I was a pack rat until about last year. I think I must have culled about 50% of my library's images, and then I converted whatever remaining RAW files I had to JPEG, which shrunk it to about a third of its original size. I realized there was no point in keeping the RAWs after I did my post -- I'm not that serious a photographer, and I was never going to revisit the images for editing again. Of course, every person has their own requirements -- I'm only speaking for myself. I also tried to be my own harshest critic, and each image had to meet only -one- of two criteria: 1- Am I be willing to print it and display it? 2- Do I have a strong emotional connection to the photo? (this is how a lot of less-than-perfect images survived the cull) These days, I spend more time culling after a day of shooting than I do in post on my keepers. More often than not, I will delete up to 80% of my shots in a day, and a lot of the deleted shots are perfectly fine. >> There has got to be a better/easier solution than even this process. My own process is relatively simple - I point Crashplan to three targets: the cloud, a mapped drive (NAS) and a local drive that's designated for backup only. I also periodically burn my library to BluRay discs (thankfully my library is now small enough to fit on 2x50GB discs). I also intend to use Google Photos, but just haven't found the time to do it yet.
alkonaut 11y ago
Storage is cheap. Backup is easy (if you trust the cloud). Keep as many TB local storage as you need (a non pro should probably be fine with a few TB for stills of you cull the imports of near-duplicates and OOF shots etc). Spinning disks cost next to nothing and are good enough. Then, if you want, use one or more means of local protection, such as mirrored local disks or sync to a NAS, preferably at a remote location to protect against theft and fire but local is ok if you must. If you trust your backup service and you have a very good internet connection you could skip this step and just use a few TB of local storage. Last and most importantly: have a proper backup. syncing to a copy isn't backup. A backup from which you can restore any file from history, after you corrupt it or accidentally delete it (you will do this, and it will happen many more times than you suffer from a disk malfunction or burglary). Even your own carefully crafted backup solution will fail. So plan for that too (by using a 3rd party service too). There are several very cheap providers of unlimited backup of this kind, for example CrashPlan. Regardless of whether I used proper backup to a remote storage, I'd still make sure to also backup to a cloud service, or even backup both the PC and the NAS to the same service (at no extra cost if it's an unlimited service such as the CrashPlan 10 computer family plan).
JimmaDaRustla 11y ago
Nice article! Wondering why a custom built solution was such a pain to maintain to the author - he mentions updates, but once you have a working system, no real reason to update unless there is a feature or security issue that needs to be addressed.
deleted 11y ago
[deleted]
rsync 11y ago
"Well let's put aside the disk failure issue. Modern 4-drive NAS systems can tolerate a lost drive and alert you promptly to replace it. You'd have to have pretty bad luck to lose more than one drive at the exact same time." Very interesting and well thought out posting, but the above quotation represents a very, very naive understanding of how these arrays work with large, multi-TB hard drives. In fact, the reality is exactly backwards to what he has written here: with multiple terabytes of data on the array, a single drive failure results in a long, intensive rebuild process that can serve to hasten the failure of the remaining drives. I am not anti-NAS - I use them myself for critical data - but with 3 and 4TB hard drive, I would only use raidz3 (or equivalent) at this point (and preferably with 12 or fewer drives in the array).
- ValentineC 11y ago
  > In fact, the reality is exactly backwards to what he has written here: with multiple terabytes of data on the array, a single drive failure results in a long, intensive rebuild process that can serve to hasten the failure of the remaining drives. Are you talking about Unrecoverable Read Errors? [1] [1] https://news.ycombinator.com/item?id=8306499 https://news.ycombinator.com/item?id=8306499
  - rsync 11y ago
    I'm talking about any problem that a hard drive could have. One drive dies, and the response to that drive dying is to go beat the shit out of the remaining drives, and given their large size, that beating goes on for days (possibly). Yes, an unrecoverable read can be one failure mode, although not necessarily the scariest - you can often tell your raid hardware to ignore UREs and just rebuild anyway, hoping that the missing bits aren't crucial (which in the case of photo storage isn't that bad of a bet...).
- slantyyz 11y ago
  Is a RAID really necessary though? I know the RAID units today are pretty good, but I've had some failures in the past with older systems, and I just stay away from them, because I have less than 1TB of data that I can't afford to lose, and the mirroring doesn't really add a lot of value for my needs. To keep things simple, I just make redundant copies by pointing Crashplan to multiple targets - the cloud, a single-drive NAS, and an internal hard drive in my desktop dedicated for backups. I also burn periodic copies of my most critical data to Blu-Rays. Super critical data gets burned to M-Disc Blu-Rays.
deleted 11y ago
[deleted]
uptown 11y ago
For any photographers looking to self-host their collections, I strongly recommend taking a look at Koken. Paul mentions it in his article. Unfortunately, they're looking to sell the company - so I'm not sure how long their product will be around, but it's a great solution for self-hosted photos. http://koken.me/ http://koken.me/
Amanjeev 11y ago
Has anyone used http://www.drobo.com http://www.drobo.com? What are the views when compared to Synology?
- detaro 11y ago
  They generally are not as well regarded. Nice and polished if they work, but less reliable than alternatives.
- ValentineC 11y ago
  Scott Kelby, a fairly well-known photographer, isn't a huge fan of Drobo, most likely because they're using proprietary hardware RAID [1]. Synology, on the other hand, uses Linux mdadm. You can stick your RAIDed disks into a normal PC and access your data. [1] http://scottkelby.com/2012/my-life-after-drobo/ http://scottkelby.com/2012/my-life-after-drobo/
- mozumder 11y ago
  I've had one for about 7 or 8 years. It's worked perfectly fine. The original Drobos are slow, but I only use it for archival backup.
trwhite 11y ago
I don't think I agree with any of this. There's no need to be storing anywhere near that amount as a hobbyist. 1) Learning to be more selective in the pictures you take forces you to be a better photographer. Shoot film for a month and you'll learn this. 2) Learning to edit (in the sense of choosing your best) photographs also forces you to be a better photographer. In realising what makes a better picture, you'll learn what looks good and what doesn't.
- PStamatiou 11y ago
  If you read my post you'll see that I ruthlessly edit and cull before posting any of my photos. It's not about that, it's just about keeping the original set. I want to keep everything on hand as I sometimes go back and make videos like this: https://www.youtube.com/watch?v=8fR4MjImSU0 https://www.youtube.com/watch?v=8fR4MjImSU0 Or now that I imported them to Google Photos it creates a bunch of GIFs and other fun things for me. If I didn't have the 5 shots of that same thing, etc ,etc, I wouldn't have any of this.
  - barrkel 11y ago
    Replying here so that you'll hopefully see it: WD Red drives do not have head parking disabled. In fact this is a matter of some annoyance in the NAS world, because the default settings (8 second timeout) lead to astronomical SMART Load_Cycle_Count numbers when used in a typical NAS. Google around and you'll see. Run smartctl on your drives and check the value of Load_Cycle_Count. That's the number of times the drive has come out of head parking. Most of my WD Reds were over 100K before I noticed - the duty limit is 600K. idle3-tools on Linux comes with idle3 tool, which can read and alter the head park timeout.
    - PStamatiou 11y ago
      I remember reading about this issue when I was setting the NAS up and I kept monitoring the load cycle counts but saw nothing out of the norm. Wondering if this was updated in recent drive firmware? I will take a look at that tool though, thanks!
coldtea 11y ago
>I captured 275GB of photos and videos on my last trip. Just one trip! Maybe throw the junk away and use some judgement as to what to shoot and what to keep?
akeck 11y ago
I love the detail in this article! Recently, my uncle passed away, and I started thinking about the legacy of my photos (27k). The system the author describes works for him and he understands it in detail, but I wonder if anyone else in his family does. If (when?) he passes away, his heirs will not have enough free time in rest of their lives to evaluate his stored media. Likewise, does he have enough free time left in his life to view each of his media objects at least once more? One reason why our generation will "go dark" historically, may be that our heirs, facing terabytes of our data to review, on top of their own data, will simply walk away from the task and delete our data.
- deleted 11y ago
  [deleted]
ksml 11y ago
Slightly off topic, but since no one else had mentioned it, just wanted to say the rest of his blog is incredibly well done (at least on mobile.) Clicking through to his "Greece" link (http://paulstamatiou.com/photos/greece/two-weeks-in-greece/ http://paulstamatiou.com/photos/greece/two-weeks-in-greece/) was a great decision. Great photography, great layout/design, and good UX overall (I love how it tells a story, and keeps track of what you've seen)
fxxxit 11y ago
I have a little i3 NUC with 2TB HDD. I quickly go through my photos when I upload them and then they go directly into my OneDrive folder which will sync while I sleep. 10TB (soon to be unlimited) storage because I have Office 365 means I never worry about running out of space.
jupiter909 11y ago
Reading these comments it makes me think that what is being show here in many of these responses is one of core issues with the software world at large. The issue is that of "It's not the way I do it therefore no-one should do it like that." If a person wants to hold on to and horde masses of data then that is their prerogative. To give suggestions on how one would do it from their own view is acceptable but to outright dismiss another persons wants and needs is very myopic. One could even view the building of the data storage systems as a hobby in itself and the act of doing so and documenting it will be of use to others, even in other industries. I know of some professional photographers who have really poor data setups as they are very not that tech savvy so linking them to an article like this is very helpful.
lips 11y ago
This makes my brain hurt just reading it. I honestly don't see any significant functional upside to all this complexity. My recipe: External 2-bay removable SATA, an order of cardboard boxes, esd foam, and esd bags, a desk drawer, a closet at work, your SO, family member Block level clones are fast and easy when you do them frequently on dedicated drives. I've successfully recovered from serious simultaneous failures (Deskstar disaster of 2008), and haven't changed the setup, managed settings, or paid any bills in years.
ishikawa 11y ago
I think it would really be great if services like Google Photos, Amazon Cloud Drive, Smugmug and others with unlimited paid plan for pictures and videos had self service stations on major tourist spots where people could easily transfer the data from SD cards and leave the place certain that all the content would be on their cloud account in less than 30 minutes. They could even receive a notification. This way people wouldn't need to worry about data storage while far from home. Too many backpackers have to carry a computer only because they have to unload their memory cards. And also Hotel connection are mostly very slow to upload hundreds of Gbs of pictures and videos.

205 points by PStamatiou 11y ago

For those who don't need quite as much storage (12tb!) I recommend trying Apple's new iPhoto library. It has been pretty close to a dream solution for me. Now I can take photos on a trip, upload them via my iPad. Make edits to RAW files, curate, and it syncs every time I can connect to wifi, backing up my originals and syncing across all changes to any Apple device (and web).

digikata 11y ago
Do you backup the library to something outside of iPhoto? I've been wondering if there's a nice automatable way to backup that library onto a home Linux computer.
- magic5227 11y ago
  Yes I also keep my own backups. I use Crashplan for the iphoto library, as well as a local SSD backup. Fortunately iPhoto libraries are just folders so by backing it up you can always take out the folder of originals and put them in Google or anything else.

clintonb 11y ago

> Ever since my first iPhone in 2007, I've been keeping every mobile phone photo and video I've taken. Have you ever considered deleting some of the photos or videos? I shoot headshots along with my personal projects (RAW files are 20MB+), and I don't generate anywhere close to 1TB of data per year. This is primarily because I only save what I need or think I will need later.

magic5227 11y ago

As a side note I think this brings up another interesting problem than just dealing with the storage, which is, how to enjoy so many photos in a lifetime. Personally, I decided to severely restrict how many images I keep after a trip, so I'm more likely to actually view them years from now.

wlesieutre 11y ago
I'm tending to keep all of my originals in case I ever want to flip through the full set, but I've been tagging a smaller portion into either a "good photography" or "good family/etc memory" categories. Then I can easily filter down to a smaller set of photos. Then again, I haven't been taking a huge amount of photos lately. If I ramp it back up, maybe I'll need to reconsider.
acomjean 11y ago
I'm kinda up in the air about how to display the images I've sorted and edited the set down to something reasonable. I was a photo editor at my college paper and each event gets one photo you get practice cutting down the photo sets. I send some to flickr, some google+ and so on. I need a more consistent strategy. I uploaded to google plus a couple months back and it asked to create a "story". I really like the presentation of photos. Google makes it hard to link and share but the results for a group of photos are good. The example: https://plus.google.com/115477748015409107850/stories/4b3ddf5f-f9d5-3db0-90a6-f2b96722be6714cd14331ca?authkey https://plus.google.com/115477748015409107850/stories/4b3ddf... Same link, differnt url. (sigh... google) https://plus.google.com/+AramComjean/posts/ig1cSZAvt9Q https://plus.google.com/+AramComjean/posts/ig1cSZAvt9Q
calinet6 11y ago
Shrug. I've got about 1.5 TB from the last ten or twelve years, and I do often go back through the organized albums and events. Often it's to find something specific, often just looking for something good, often just for nostalgia. I'm quite glad I have them, and rating and filtering is just as good as tossing them out in my experience.
matwood 11y ago
My workflow after a photo session is: * Quickly scan pics and delete anything that is flat out bad. I try to get these on the spot, but some slip through. * Go through again and rate. * Go through again and adjust ratings further. * Post process the 5 stars, and put them up to view. * Everything else is saved through an offline backup service. I find that the 5 star pics end up being great covers into an event. Often the 5 star pics are good enough, but sometimes they draw me in the look at the rest. The only downside is you have to be ruthless with your ratings.

aaronbrethorst 11y ago

How about taking fewer photos? Do you really need two nearly identical photos of a hand reaching for food in a Japanese restaurant, or three photos of a bowl of miso soup? On my last trip, I shot four rolls of 120 film (645 format, so 64 photos total), and maybe 500 digital photos, of which I deleted 90%. I still save about 5,000 RAW files per year, or about 100GB/year, but this is a relatively manageable amount. I'm approaching the saturation point on my iMac's hard drive, but should be able to last for another several years by shipping the oldest RAWs off to Glacier and keeping high-res JPGs around locally for reference. edit: also, next time you're up here in Seattle, you should visit the Jose Rizal Bridge at sunset, and the observatory on the 73rd floor of Columbia Tower.

PStamatiou 11y ago
Thanks for the Seattle tip! I'm up there somewhat often as part of my team is there. I have done the Columbia Tower but not that bridge. As for taking fewer photos, I like to capture as much as possible and then I distill that down later into my photo sets/stories. I want a big funnel to choose from and not have to do that on the fly. This is my hobby and passion and I love it. :) As for the nearly identical photos aspect - I always take a few shots & similar compositions of the same thing to make sure that at least one shot is super in focus.
- aaronbrethorst 11y ago
  I always take a few shots & similar compositions of the same thing to make sure that at least one shot is super in focus. I get it, I do the same thing when I shoot digital. I'm just recommending that those photos never make it past the Previous Import section in Lightroom. If you're aiming for tack-sharp focus, it should be quite straightforward to do a very quick culling then. Also, please say hi to Chris Burrows for me; I haven't seen him since we worked on Visual Studio together about a million years ago :)
  - PStamatiou 11y ago
    Whoa you know Chris! He's on my team and we have standups every day. Amazing developer with a great design and product sense!

Obi_Juan_Kenobi 11y ago

> Where on earth do normal people store that much data? They don't. Normal people realize that keeping that many photos is a negative value proposition; it's a burden rather than an asset. Rather than trying to hold on to everything, they choose what is actually worth revisiting. This could mean they're a little more selective with the shutter, or they only keep the photos they like. By all means, keep everything if you want to, but I find it troublesome that storage is considered the problem, rather than making no attempt to cull images. Clearly a great deal of effort has gone into these images to make his various travelogues, so the effort to see which ones make the cut has been done already. That said, it doesn't seem like a NAS was even needed here. Any reasonable PC MoBo is going to have 6 SATA ports, so you can quite easily make a 4 drive RAID 5. Personally, I just RAID 1 a couple 3TB drives and call it good.

chinhodado 11y ago
I share the same sentiment. I have friends who would, on a trip, shoot everything they see, and shoot each scene a dozen times (literally) with little variation in angle, exposure, etc. They do this on maximum resolution, highest quality settings (Fine JPEG, even RAW). After a trip, they dump everything into their storage and hardly ever look at what they've taken again. And they would constantly complain about not having enough storage.
- philliphaydon 11y ago
  I'll go to the Zoo or somewhere and take 1000s of photos, then spend hours in Lightroom deleting maybe 80% of that till I have ~100 I keep and ~20 I edit. I don't understand why people keep the same shot 23423352 times.
  - chinhodado 11y ago
    I think it's because of the mindset "I can revisit what I have taken and make my selection later. I don't want to lose any memory. Storage is cheap." Over time, it becomes a burden to maintain all that.
  - 11y ago

distantsounds 11y ago

If you're willing to pay for Amazon Prime, you get unlimited cloud storage for all your photos. RAW included. The only method to upload/download is via the clunky web UI, but if reliable storage is what you're after, it's a pretty hard bargain.

detaro 11y ago
Assuming you have the bandwidth to do that comfortably. And you still need backups somewhere.
jlintz 11y ago
problem is they don't support syncing folders. You'd have to keep track of what you've uploaded already. Also I haven't been able to get it to successfully upload all my photos, seems to always stall out after a few hundred photos or eventually hit some sort of rate limit that goes to a crawl
alkonaut 11y ago
A "real" cloud backup (That is, not just cloud storage) like CrashPlan is just a few dollars per month. I pay less than $3 per computer per month for backup of several TB with history for each file. I just couldn't imagine it would be worth buying the hardware or configuring my own backup to make a non-cloud equivalent.
- NigsMcGee 11y ago
  Crashplan recently removed it's multi-year plan and increased prices. Gone are the days of 4-year plans that average out to a few dollars per month. Crashplan's prices have nearly doubled in the past three years alone. Disappointing.
  - alkonaut 11y ago
    How I reach as low as $3 per computer per month is through the family plan which is something like $12-13 but allows up to 10 computers. Not only does it back up my photos, it also solves the "I clicked OK in some dialog and now the computer behaves funny" of relatives.

Veratyr 11y ago

Wow, that was an amazing article. So comprehensive! A few things I'd like to add though: - Mylio (http://mylio.com http://mylio.com) is very helpful for syncing your collections across things. Not for everyone but it's worth a look to see if it works. Best thing for me is that it's peer to peer so I don't have to upload my collection to a cloud service to access it on all my devices. It does offer cloud but it's end to end encrypted (allegedly). Best thing is that it lets you configure whether you sync previews, thumbnails or originals to each device and even which photos to sync. Really handy if you want a new shoot on your phone to play with on the train or something. - You can use Google Drive to get your photos into Google Photos. This lets you keep a bit exact backup in Drive while using your quota for Photos as well. Further, Google Apps for Work Unlimited, through a glitch or deliberately I don't know, offers unlimited Drive storage to accounts even with a single user in their organisation. I pay $10/month for unlimited Google Drive storage. It's advertised as being 1TB for single users so I'm not sure if this is a bug but thought it was worth mentioning. - Google Photos will (quite helpfully) use the JPEG previews you embed in a DNG photo so if you tend to touch up things in Lightroom, embedding them will ensure that Google Photos displays things in the same way. Really though, this was an amazingly comprehensive article. Thanks for posting!

dperfect 11y ago

> The bad news is that I have over 1TB of photos and the next pricing tier after 1TB ($9.99/mo) is 10TB and that costs a whopping $99.99 per month. So I use Google Photos with the free compressed setting. I don't actually mind since I have my own file backup solution and I use it more for that added layer of intelligence, convenience and utility. I've tried doing as suggested here, but here's the problem: if it's too expensive to store your RAW images on the cloud (which it is for most people), then your cloud photo library is really just a proxy of your library. That's not to say it isn't useful, but unless it's synchronized both ways with the original files, you're just asking for disorganization - from my experience anyway. You have to be careful and basically only touch the originals to let changes propagate one-way to the cloud proxy. If it isn't hosting your RAW files, then it doesn't fill the role of a backup, and if your edits or tagging on the cloud aren't applied back to your originals, then any time/effort you put into organizing and editing your library on the cloud is somewhat wasted. To be honest, I do use Apple Photo Streams for something similar - but I just treat it as a convenience for low-quality output/viewing of recent photos, mostly from my phone. Even then, the Photo Stream part is still a mess. My main library resides on a NAS (as in the article), is backed up to 2 low-cost cloud backup providers (still much cheaper than hosting a single RAW copy on Google Photos), and I use Lightroom for all actual editing and tagging since it applies to the authoritative library. My point is - I too look forward to the day where at least one copy of my full library can be hosted on something like Google Photos in the cloud, but we're still a ways off from that being practical. Google Photos can be a convenience in some ways, but at current prices it really doesn't fit what I'm looking for.

ValentineC 11y ago
> My main library resides on a NAS (as in the article), is backed up to 2 low-cost cloud backup providers (still much cheaper than hosting a single RAW copy on Google Photos), and I use Lightroom for all actual editing and tagging since it applies to the authoritative library. I'm doing something similar, but just backing up to CrashPlan (which has a flat fee for backing up). What cloud providers are you using?
- dperfect 11y ago

knurdle 11y ago

Great article, I've had a similar setup for years now. An 8 year old readynas that's still running, it's really slow but it works. I do a sync with that and a local desktop with a big external drive. And then I back up that local desktop to crashplan. I have a comment about the drives, I generally like purchasing different manufacturer drives for my NAS when I'm buying them in bulk. I always worry about multiple drives from the same batch failing around the same time. It's happened to me before so now I'll buy similar capacity drives but from different models or mfrs.

devNoise 11y ago
I had a ReadyNAS NV before they were bought by Netgear and though it was good. Unfortunately, I had more drives fail in the ReadyNAS than the computers I was making backups from. As time went on it became hard to find drives that were on the compatibility list.

jly 11y ago

Great article...lots of comments here about how you shouldn't keep that much data. I agree with that for hobbyists, but this is pretty relevant for professionals where the option to cull doesn't extend that far and there are business reasons for retaining terabytes of photos for an extended period. Additionally, next-gen cameras with 40-50MP RAWs are right around the corner. I use a similar combination of home-built network storage (no RAID - just manual multiple backups) and glacier for offsite redundancy. Dealing with images as a business, I typically only work on a couple shoots at a time so syncing across devices is not a big concern, but long-term archiving and redundancy is.

munificent 11y ago
> Additionally, next-gen cameras with 40-50MP RAWs are right around the corner. The future is here! Canon's 50.6 MP 5DS is shipping now.
- PStamatiou 11y ago
  I just preordered the Sony A7r II: "only" 42MP but DxoMark ranks the sensor higher than the Canon.
  - varikin 11y ago
    These are not even counting scanning negatives (can be massive) or digital backs for medium format cameras. Hasseblad had a 60+MP back 10 years ago just because the sensor was larger.
  - mikeyouse 11y ago
    That looks like a monster of a camera, can't wait to see what you do with it.
    - PStamatiou 11y ago
      same here :) I'm sure I will write about it in a future post or photoset.

imaginenore 11y ago

As a photographer, he is doing it wrong. You should never ever ever store every shot you take. Your job as a photographer is to delete 95-99% of the shots. Select the best, delete the rest. The exception to that is the commercial client photos, but that's not what he is talking about. If you're going to save 275GB of photos per trip, it's guaranteed you will never look through them again. Look at his photos - I don't see a single shot I would have saved (well maybe the fisheye one from Hawaii). He has 3-4 nearly identical copies of each shot. Tons of boring stuff.

PStamatiou 11y ago
I keep everything, cull down to the best and post those online. I still like to keep the original set. Maybe down the line I'll be more rigorous here but sometimes I will make little movies out of every individual still, so that's where it comes in handy: https://www.youtube.com/watch?v=8fR4MjImSU0 https://www.youtube.com/watch?v=8fR4MjImSU0
mpdehaan2 11y ago
Yep, makes for an online gallery that isn't any fun to look at. Perhaps common sense, but I recommend remembering to print photos (say 4x6") you really care about (works especially if you lose your backup) and also hang some of the stuff you really care about it (because you're unlikely to look at them electronically). Take advantage of your walls, if you have a event decent camera, seeing your stuff 10x14, 16x20 or even 20x30 is also exceptionally worth it. I'm kind of fond of mpix standouts which to me look a lot better than seeing things behind glass - http://www.mpixpro.com/Catalog.aspx/standouts http://www.mpixpro.com/Catalog.aspx/standouts
- jscheel 11y ago
  I agree 100%. Printing your photos really helps elicit a more visceral response. That said, what is mpix? I looked at it, but it wanted me to go through an approval process to create and account to see the prices.
  - mpdehaan2 11y ago

ScottBurson 11y ago

I considered a NAS, but I had a Linux server already, so I just upgraded it with 6 4TB drives and installed ZFS [0]. I'm using raidz2, which is doubly redundant like RAID-6. For long-term backup I just got a BD-R drive with M-Disc support [1]; haven't tried it yet. This still makes more sense to me than trying to store it all in the cloud. [0] http://zfsonlinux.org/ http://zfsonlinux.org/ [1] http://www.mdisc.com/ http://www.mdisc.com/

lutorm 11y ago
This is my solution, too. And then I push periodic zfs snapshots to an off-site server. It's not fast given my upload speed with TWC, but it's doable.
- StavrosK 11y ago
  Try attic, you'll thank me.

NateDad 11y ago

Everyone needs to check out https://smugmug.com https://smugmug.com - for $40 a year you get unlimited photo and video uploads (yes, RAW files are ok). It's like flikr with no ads, only your own stuff on the pages, super customizable (with a power account - $60 a year - you can point your own URL at your account and no one even needs to know the files are on smugmug. It's a site designed for professional photographers that also happens to have kickass features for casual photographers as well. There's even lightroom plugins to upload directly to smugmug. One of the best features, in my opinion, is the customer support. They respond within a couple hours. They're super nice and knowledgeable. I've had an account at smugmug for nearly a decade, and would never go back to free services. Google photos is nice, but the size limitation is definitely a problem for prolific photographers. (I don't work for smugmug nor do I know anyone who works there, I just love their service.)

georgemcbay 11y ago
As a pretty serious hobbyist photo-taking-guy I'm a big fan of smugmug, my "sharing photos with friends and family" photo website (http://gmcbay.com http://gmcbay.com) is hosted with them, but I don't think of them or use them as a good long-term file "storage"/backup solution. All of the photo-specific type of storage solutions (Google, Amazon, Smugmug, Adobe's CC thing, etc) are kind of flawed for many reasons (some outlined in the OP blogpost) when used as what I think of as "digital negative storage", IMO. I just store all of my (original, RAW-format) photos/videos on giant harddrives inside my desktop computer and have that backed up automatically and continuously via backblaze. Local storage is ridiculously cheap, and so are offsite automatic backups. For people who just deal with jpg files from a phone camera or whatever then YMMV and these photo-specific services may be just the thing, but I've yet to try one that doesn't feel like it is getting in your way if your normal workflow is big RAW files with processing via Lightroom/Photoshop, etc.
- fuzzywalrus 11y ago
  Indeed, I still use an aging 2008 Mac Pro and the benefits of internal storage are pretty profound. I have two Time Machine HDDs (An Internal + an External), which counts for two of my six HDDs. I'm debating going towards a service like blackblaze as I have a decade of photos on my computer.

beachstartup 11y ago

i'm so glad i wasn't born with the digital hoarder gene. apparently my absolute-zero proclivity to hang onto photographs and other media is extremely rare. i'm just not into it. i'll snap some stuff on my phone to share with my close friends, and then let it get deleted or whatever. i don't upload it to socialmedia. i just don't give a shit, to be honest. we're surrounded by this stuff day in and day out on every screen we have, i make no effort to keep any of it longer than a few weeks. in fact i have a problem getting rid of old photos, somehow they seem to follow me around on device to device through no fault of my own! they're almost like viruses.

fencepost 11y ago

The thing that kind of jumped out at me was the RAID5 configuration - I can't imagine doing that with 3TB drives unless he really does have everything on there also backed up to another location.

calinet6 11y ago

Just a +1 for the Synology NAS products. I'm frequently amazed every time I use it how clean the UI is and how they basically implemented a full window manager for the underlying Linux system in a web app. It's exactly what a NAS should be: powerful and flexible, yet easy to set up and use, but rock solid even if it just sits there. Lots of great things to say about using it in nearly exactly this setup for about a year now.

CamperBob2 11y ago
Everything you say is 100% true, but be sure to keep the OS up to date if you expose any IP addresses (security cameras, etc) to the world at large. NAS boxes are just about the highest of high-value targets.
- calinet6 11y ago
  Synology DSM has great auto-update systems :) But, absolutely correct.

AbeEstrada 11y ago

My storage solution is Blue-ray discs + External HDD + AWS Glacier. Once I use the RAW files, I rarely need them, that's why I store them in Blue-ray discs as a backup.

jscheel 11y ago

Culling is an important part of the photographic process. However, far be it from me to tell another photographer what they should and shouldn't keep. We all have our quirks. Good luck convincing me to delete even a blurry photo of my daughter :) Storage is still a huge problem for photographers. There has got to be a better/easier solution than even this process. It's still outside the realm of a lot of photographers' skillsets.

slantyyz 11y ago
>> Culling is an important part of the photographic process. It is indeed important, but it is hard. I was a pack rat until about last year. I think I must have culled about 50% of my library's images, and then I converted whatever remaining RAW files I had to JPEG, which shrunk it to about a third of its original size. I realized there was no point in keeping the RAWs after I did my post -- I'm not that serious a photographer, and I was never going to revisit the images for editing again. Of course, every person has their own requirements -- I'm only speaking for myself. I also tried to be my own harshest critic, and each image had to meet only -one- of two criteria: 1- Am I be willing to print it and display it? 2- Do I have a strong emotional connection to the photo? (this is how a lot of less-than-perfect images survived the cull) These days, I spend more time culling after a day of shooting than I do in post on my keepers. More often than not, I will delete up to 80% of my shots in a day, and a lot of the deleted shots are perfectly fine. >> There has got to be a better/easier solution than even this process. My own process is relatively simple - I point Crashplan to three targets: the cloud, a mapped drive (NAS) and a local drive that's designated for backup only. I also periodically burn my library to BluRay discs (thankfully my library is now small enough to fit on 2x50GB discs). I also intend to use Google Photos, but just haven't found the time to do it yet.

alkonaut 11y ago

Storage is cheap. Backup is easy (if you trust the cloud). Keep as many TB local storage as you need (a non pro should probably be fine with a few TB for stills of you cull the imports of near-duplicates and OOF shots etc). Spinning disks cost next to nothing and are good enough. Then, if you want, use one or more means of local protection, such as mirrored local disks or sync to a NAS, preferably at a remote location to protect against theft and fire but local is ok if you must. If you trust your backup service and you have a very good internet connection you could skip this step and just use a few TB of local storage. Last and most importantly: have a proper backup. syncing to a copy isn't backup. A backup from which you can restore any file from history, after you corrupt it or accidentally delete it (you will do this, and it will happen many more times than you suffer from a disk malfunction or burglary). Even your own carefully crafted backup solution will fail. So plan for that too (by using a 3rd party service too). There are several very cheap providers of unlimited backup of this kind, for example CrashPlan. Regardless of whether I used proper backup to a remote storage, I'd still make sure to also backup to a cloud service, or even backup both the PC and the NAS to the same service (at no extra cost if it's an unlimited service such as the CrashPlan 10 computer family plan).

JimmaDaRustla 11y ago

Nice article! Wondering why a custom built solution was such a pain to maintain to the author - he mentions updates, but once you have a working system, no real reason to update unless there is a feature or security issue that needs to be addressed.

deleted 11y ago

[deleted]

rsync 11y ago

"Well let's put aside the disk failure issue. Modern 4-drive NAS systems can tolerate a lost drive and alert you promptly to replace it. You'd have to have pretty bad luck to lose more than one drive at the exact same time." Very interesting and well thought out posting, but the above quotation represents a very, very naive understanding of how these arrays work with large, multi-TB hard drives. In fact, the reality is exactly backwards to what he has written here: with multiple terabytes of data on the array, a single drive failure results in a long, intensive rebuild process that can serve to hasten the failure of the remaining drives. I am not anti-NAS - I use them myself for critical data - but with 3 and 4TB hard drive, I would only use raidz3 (or equivalent) at this point (and preferably with 12 or fewer drives in the array).

ValentineC 11y ago
> In fact, the reality is exactly backwards to what he has written here: with multiple terabytes of data on the array, a single drive failure results in a long, intensive rebuild process that can serve to hasten the failure of the remaining drives. Are you talking about Unrecoverable Read Errors? [1] [1] https://news.ycombinator.com/item?id=8306499 https://news.ycombinator.com/item?id=8306499
- rsync 11y ago
  I'm talking about any problem that a hard drive could have. One drive dies, and the response to that drive dying is to go beat the shit out of the remaining drives, and given their large size, that beating goes on for days (possibly). Yes, an unrecoverable read can be one failure mode, although not necessarily the scariest - you can often tell your raid hardware to ignore UREs and just rebuild anyway, hoping that the missing bits aren't crucial (which in the case of photo storage isn't that bad of a bet...).
slantyyz 11y ago
Is a RAID really necessary though? I know the RAID units today are pretty good, but I've had some failures in the past with older systems, and I just stay away from them, because I have less than 1TB of data that I can't afford to lose, and the mirroring doesn't really add a lot of value for my needs. To keep things simple, I just make redundant copies by pointing Crashplan to multiple targets - the cloud, a single-drive NAS, and an internal hard drive in my desktop dedicated for backups. I also burn periodic copies of my most critical data to Blu-Rays. Super critical data gets burned to M-Disc Blu-Rays.

deleted 11y ago

[deleted]

uptown 11y ago

For any photographers looking to self-host their collections, I strongly recommend taking a look at Koken. Paul mentions it in his article. Unfortunately, they're looking to sell the company - so I'm not sure how long their product will be around, but it's a great solution for self-hosted photos. http://koken.me/ http://koken.me/

Amanjeev 11y ago

Has anyone used http://www.drobo.com http://www.drobo.com? What are the views when compared to Synology?

detaro 11y ago
They generally are not as well regarded. Nice and polished if they work, but less reliable than alternatives.
ValentineC 11y ago
Scott Kelby, a fairly well-known photographer, isn't a huge fan of Drobo, most likely because they're using proprietary hardware RAID [1]. Synology, on the other hand, uses Linux mdadm. You can stick your RAIDed disks into a normal PC and access your data. [1] http://scottkelby.com/2012/my-life-after-drobo/ http://scottkelby.com/2012/my-life-after-drobo/
mozumder 11y ago
I've had one for about 7 or 8 years. It's worked perfectly fine. The original Drobos are slow, but I only use it for archival backup.

trwhite 11y ago

I don't think I agree with any of this. There's no need to be storing anywhere near that amount as a hobbyist. 1) Learning to be more selective in the pictures you take forces you to be a better photographer. Shoot film for a month and you'll learn this. 2) Learning to edit (in the sense of choosing your best) photographs also forces you to be a better photographer. In realising what makes a better picture, you'll learn what looks good and what doesn't.

PStamatiou 11y ago
If you read my post you'll see that I ruthlessly edit and cull before posting any of my photos. It's not about that, it's just about keeping the original set. I want to keep everything on hand as I sometimes go back and make videos like this: https://www.youtube.com/watch?v=8fR4MjImSU0 https://www.youtube.com/watch?v=8fR4MjImSU0 Or now that I imported them to Google Photos it creates a bunch of GIFs and other fun things for me. If I didn't have the 5 shots of that same thing, etc ,etc, I wouldn't have any of this.
- barrkel 11y ago
  Replying here so that you'll hopefully see it: WD Red drives do not have head parking disabled. In fact this is a matter of some annoyance in the NAS world, because the default settings (8 second timeout) lead to astronomical SMART Load_Cycle_Count numbers when used in a typical NAS. Google around and you'll see. Run smartctl on your drives and check the value of Load_Cycle_Count. That's the number of times the drive has come out of head parking. Most of my WD Reds were over 100K before I noticed - the duty limit is 600K. idle3-tools on Linux comes with idle3 tool, which can read and alter the head park timeout.
  - PStamatiou 11y ago
    I remember reading about this issue when I was setting the NAS up and I kept monitoring the load cycle counts but saw nothing out of the norm. Wondering if this was updated in recent drive firmware? I will take a look at that tool though, thanks!

coldtea 11y ago

>I captured 275GB of photos and videos on my last trip. Just one trip! Maybe throw the junk away and use some judgement as to what to shoot and what to keep?

akeck 11y ago

I love the detail in this article! Recently, my uncle passed away, and I started thinking about the legacy of my photos (27k). The system the author describes works for him and he understands it in detail, but I wonder if anyone else in his family does. If (when?) he passes away, his heirs will not have enough free time in rest of their lives to evaluate his stored media. Likewise, does he have enough free time left in his life to view each of his media objects at least once more? One reason why our generation will "go dark" historically, may be that our heirs, facing terabytes of our data to review, on top of their own data, will simply walk away from the task and delete our data.

deleted 11y ago
[deleted]

ksml 11y ago

Slightly off topic, but since no one else had mentioned it, just wanted to say the rest of his blog is incredibly well done (at least on mobile.) Clicking through to his "Greece" link (http://paulstamatiou.com/photos/greece/two-weeks-in-greece/ http://paulstamatiou.com/photos/greece/two-weeks-in-greece/) was a great decision. Great photography, great layout/design, and good UX overall (I love how it tells a story, and keeps track of what you've seen)

fxxxit 11y ago

I have a little i3 NUC with 2TB HDD. I quickly go through my photos when I upload them and then they go directly into my OneDrive folder which will sync while I sleep. 10TB (soon to be unlimited) storage because I have Office 365 means I never worry about running out of space.

jupiter909 11y ago

Reading these comments it makes me think that what is being show here in many of these responses is one of core issues with the software world at large. The issue is that of "It's not the way I do it therefore no-one should do it like that." If a person wants to hold on to and horde masses of data then that is their prerogative. To give suggestions on how one would do it from their own view is acceptable but to outright dismiss another persons wants and needs is very myopic. One could even view the building of the data storage systems as a hobby in itself and the act of doing so and documenting it will be of use to others, even in other industries. I know of some professional photographers who have really poor data setups as they are very not that tech savvy so linking them to an article like this is very helpful.

lips 11y ago

This makes my brain hurt just reading it. I honestly don't see any significant functional upside to all this complexity. My recipe: External 2-bay removable SATA, an order of cardboard boxes, esd foam, and esd bags, a desk drawer, a closet at work, your SO, family member Block level clones are fast and easy when you do them frequently on dedicated drives. I've successfully recovered from serious simultaneous failures (Deskstar disaster of 2008), and haven't changed the setup, managed settings, or paid any bills in years.

ishikawa 11y ago

I think it would really be great if services like Google Photos, Amazon Cloud Drive, Smugmug and others with unlimited paid plan for pictures and videos had self service stations on major tourist spots where people could easily transfer the data from SD cards and leave the place certain that all the content would be on their cloud account in less than 30 minutes. They could even receive a notification. This way people wouldn't need to worry about data storage while far from home. Too many backpackers have to carry a computer only because they have to unload their memory cards. And also Hotel connection are mostly very slow to upload hundreds of Gbs of pictures and videos.