16 ms·

I nearly lost the Lightroom catalog with of all my photos

249 points by pchm 5y ago

formerly_proven 5y ago
Personally I wouldn't use that NAS for storing data until figuring out what went wrong, since every single file was corrupted. This is way too high an error rate and unlikely to be caused by a hard drive or network issue. It's probably an issue caused by NFS/SMB/... client/server bugs. Edit: Quick googling finds many people using Macs report files are corrupted when written to a Synology NAS over SMB.
cube00 5y ago
This is confusing, it sounds like the RAW files themselves are corrupt so why bring any discussion of the Lightroom catalog into it? Having said that people need to take care backing up their catalog because without that single catalog file you will lose decades of photo editing work in Lightroom.
- tiernano 5y ago
  Given how photos are "imported" into a Lightroom Catalog, some might think thats where they live... On a Mac, IIRC (its been a while since i used LR on a Mac) if you look at your catalog, its a single large "file" with all the meta data, edits, originals, etc, in one place...
  - iriche 5y ago
    This isn't really correct, the original image is always stored in the original place or the place you select to copy it too. That's why I also think the title is a bit miss leading, cause it was never the Lightroom catalog that was the issue, it was the RAW-images them self.
  - Saris 5y ago
    The catalog only contains the edits and changes to an image, it doesn't actually contain any images. Lightroom can move or copy images to a new folder for you, but they're still just normal image files you can open with any supported software.
- pembrook 5y ago
  Yes, Lightroom has nothing to do with it. Seems like that’s simply the app that the author is using to view the files. The title of this post should be changed to “file transfers from MacOS to Synology NAS causes file corruption” It’s a known issue with no fix at the moment.
  - otterley 5y ago
    Known to whom? And do you have a reference to this information from either Apple or Synology?
- lentil 5y ago
  You can also set Lightroom to write sidecar xmp files, which means every edited raw file will also have its own copy of the edits kept alongside it. Coupled with using a good directory structure, that significantly lessens the impact of losing a catalog file.
gorgoiler 5y ago
Lightroom is not a word I am comfortable reading. Not since my archive copy became uninstallable in macOS (32 bit installer) and Adobe required a monthly subscription just for me to have access to my catalog. I have been made a fool of.
- beermonster 5y ago
  Better off using digikam to as a DAM and being in charge of your own data.
  - toastal 5y ago
    Darktable as well
    - beermonster 5y ago
      Yes. They’re both fantastic. Though digikam is a closer Lightroom replacement.
teekert 5y ago
Time Machine backups should live on Apple Filesystems. The Ext4 or btrfs in Synology's is a timebomb for Apple's proprietary data blobs. It has something to do with the filesystem attributes. I asked my father to just give up on it. Just manage pictures through the file system. Photos (former iPhotos) databases also get corrupt on a Synology NAS. It's a matter of time. The strange thing is that it can go well for months on end, giving you a false sense of security. Do iCloud or local drive backups or stay away from Time Machine and Apple Databases is my advice (although my father had a lot of problems as well with an "incorrectly unplugged" external HFS+ drive). I can't find the sources right now but after my father's last drama (and there have been several) I did some intensive searching and this was my conclusion.
- Maakuth 5y ago
  Are these applications really adding so much value that it's worth it with their proprietary catalogs? My personal solution for photo storage is a normal directory tree with files. I have Piwigo set up to catalog those so that I can browse by shooting date and tags, but the file hierarchy stays untouched. I can pull files from there to edit with any application and there's a lot of proven tools to keep the files in good shape and backed up.
  - teekert 5y ago
    If you're in the Apple ecosystem with an iPhone, some iPads and a Macbook, you sort of roll into it. Is it worth it? Not sure, it's sure easy and just works. But you're on your own when you try to interface with your data in "non conventional" ways. I myself use NextCloud for everything, I recently moved from Android to iOS and it's nice to see most things working... except that NextCloud has issues making previews from .heic pictures (or I should say heic picture containers containing heif images coded in hevc :s), and so the drama starts again. It's always plug and pray outside the Apple ecosystem, always ymmv.
    - MereInterest 5y ago
      I like your quotes around "non conventional", as everything about storing files in a directory is the traditional and conventional way to store files.
sneak 5y ago
It looks like he’s using HFS or APFS. These filesystems still lack data checksums for data on disk, aside from any of the other issues he ran into. The only sane way to deal with unique/irreplaceable data (i.e. one’s own photos) where the authoritative copy is on a mac is to make several independent copies, with several kept offline, and assume that the macOS or Adobe’s terrible house of cards that is the creative suite is going to fuck everything up at some point. I say this as someone with >1000GiB in a Lightroom catalog, run on a mac. I have a copy on an always-connected external disk (made with rsync && rsync -c), a copy on a ZFS NAS (made with rsync && rsync -c user@host:), a Time Machine backup that is only connected weekly, and an offsite rsync to a USB drive, with a SHASUMS file sidecar. I also copy the RAWs into the LR masters photo structure when I import them, and they go first from the camera into a Syncthing folder (pre-import) that is immediately and automatically replicated onto four machines in three buildings (two of which are ZFS with autosnapshots).
abcd_f 5y ago
> A quick googling indicated a hard drive problem In the vast majority of cases spontaneous data corruption happens in _transit_ due to RAM glitches. All modern drives implement forward error correction on per sector basis. This allows the drive to automatically repair up to 10% of damage to any given sector... in which case correct data is returned to the requestor and the sector is either tagged for relocation or is relocated right away. In cases when the data can’t be cleanly recovered, the read request us failed. That is, chances of a read request returning mangled data from a disk is next to absolute zero. Meaning in turn if you do see data corruption, it happened before this data hit the disk - i.e. it happened in transit.
- lmilcin 5y ago
  In vast majority of cases corruption (that doesn't involve drive failure) happens due to bad or naive implementation of software that handles files. While memory does fail sometimes, if it was failing at the rate you describe PCs would not be suitable to any work at all. I have worked in ops for many years. A lot of software that copies files is perfectly happy to leave you with copy of different length than original.
  - abcd_f 5y ago
    Software bugs is an altogether different issue. As it is a far more exotic one. The context of the OPs post and my reply is the case when copying in bulk with a mature tool yields corruption in a small fraction of the data. In this case the cause is the in-transit corruption (rather than at-rest, which is a fairly common belief dubbed as a “bitrot” phenomenon).
    - lmilcin 5y ago
      Software bugs exotic. That is funny. File IO is one of more error prone activities with low understanding by general software dev community. It is actually rare to see a piece of code that is NOT broken. I had once observed a piece of software write a million files, all empty, because it could not handle situation where it could create file but not write it. It is quite normal for files to be truncated because written directly to destination file rather than through temp. Then script gets killed and next instance assumes file exists so it resumes from next. Most software can do the job if everything works, but almost none can correctly handle every error condition.
ygra 5y ago
I once got a bit of a scare in that many of my raw files has weird color patterns in them. Even the backups. Turns out, I had faulty RAM and Lightroom happened to always hit the faulty parts when reading a raw image, apparently.
hackshack 5y ago
This is incredibly frightening. I’ve seen data rot up close, and as a former data janitor, I can sympathize. My advice is to control everything down to the wall socket. I considered the Synology boxes for my home NAS, but my “spider sense” was tingling, so I built around an HP MicroServer Gen8 with an LSI PCIe card, and mirrored drives. It’s on a double-conversion UPS (Smart-UPS RT). It runs FreeNAS on VMware ESXi, and the drives present as raw volumes to ZFS. No data integrity issues after ~3 years. Note the trend: clean power, good server-y hardware, matched drives in a simple RAID, properly resourced ZFS, etc. But yeah, spinning disks are generally not a “backup.”
- Maakuth 5y ago
  Even in this setup there's a caveat: HP sells Microservers with non-ECC hardware too. I think after human error, in your setup the likeliest threat to data integrity is faulty memory modules propagating corruption to disks (in essence you'll have a dirty buffer in faulty RAM). ZFS helps you catch this, but it's not magic so be sure to have the ECC version.
- deleted 5y ago
  [deleted]
lmilcin 5y ago
Even worse because Lightroom normally does not modify the files. I am using separate storage for raw files and Lightroom database for this reason. Immutable files should be the easiest case to handle reliably.
tomaszs 5y ago
What I find work best for storing data is a regular USB drive. With no software like Lightroom, no NAS. Just a disc, you move data with a regular file manager. You don't get all the cool features, but process is pretty safe and you don't worry about hackers because it is offline most of the time.
- tikkabhuna 5y ago
  How do you protect yourself from losing data when the drive breaks?
  - kylehotchkiss 5y ago
    Arq.app can backup external disks to most of the big cloud storage providers really well. Unlike Backblaze, they don’t delete these backups if you don’t connect your drive for a specific amount of time. I’ve used Arq to backup about 2TB of external drives to AWS Glacier. It even let me set the encryption key for the data myself!
    - jnieminen 5y ago
      How is your experience with restoring? Have you had expensive data retrieval costs?
      - kylehotchkiss 5y ago
        I used to backup with Arq to S3 and the restores were very fast. With Glacier, Arq initiates the data retrieval and waits for file availability in the UI. Glacier can take a day or so, laptop needs to be open and connected to internet during the time for the transfer to complete. Given I'm primarily backing up external drives, I find this UX not an issue, but if you wanted backups with instant accessibility, backing up to S3 really isn't that pricy.
  - tomaszs 5y ago
    I have second backup disk that has snapshots of the backup disk. It is unlikely two disks from two manufacturers will break in the same time. So it gives me fairly good level of safety
jwr 5y ago
This sort of thing scares me. It's why I started running consistency checks on my important archives (like my photo library), which I keep backed up in multiple places. We tend to think that in a digital world bits are just bits and do not get corrupted — which is decidedly untrue. I wrote my own consistency checker, as I wasn't happy with what was out there. I wanted it to be simple, and maintainable in the long term (>10 years horizon). See https://github.com/jwr/ccheck https://github.com/jwr/ccheck if you need something like this. I now update my checksums regularly and check for corruption.
- teekert 5y ago
  Still, using rsync with checksumming enable can get your Photos library messed up if you sync (or just copy) to a non APF or HFS+ file system. Be very careful.
  - nix23 5y ago
    How is that possible?
    - teekert 5y ago
      I read that some of the key information is in attributes unique to Apple's file systems, I can't find the source but that was my conclusion after trying to figure out what went wrong in the Time-Machine/Synology/Rsync-via-webdav system I set up for my father some years ago. If you rsync ext4 files to exfat, you also have issues, but those are very clearly reported when you attempt to do so.
      - nix23 5y ago
        It makes sense that you loose some information from a posix filesystem to exfat, but from HFS+ to lets say XFS? Both can have extended attributes..hmm However hold on, i do some research. Now you hooked me ;) EDIT: Ok got it, both Filesystem NEED xattrs enabled, and you need to run rsync with the -X or --xattrs option, that should fix the problems.
trzeci 5y ago
I found myself, that my enthusiastic approach to photography does not utilize the CC plan on Adobe to pay ~10$/month. What I do is to import images from SD + do a base color correction. What I found unique and useful is face recognition that Lightroom offers. Nonetheless, I decided to move on to either open source software or software that I can buy. At the moment I'm testing On1 - but I wanted to check if somebody from HN has either own solution that wanted to share or something to recommend.
deleted 5y ago
[deleted]
logotype 5y ago
Important backups should be stored in a variety of ways. I store my backups on multiple external HDD’s (solid state and spinning disks) in redundancy. Using multiple filesystems (ext4, and exFAT). I do use ZFS for daily/active storage but not for backups. Besides that I also store the data in AWS S3 and a duplicate in Backblaze B2. These solutions are scalable. iCloud/Google Drive is good if you only have 2TB or less. Just store the raw files directly in a regular file tree structure and avoid any kind of catalog/index/db file. I do find ZFS very interesting but in practical use it has been trouble working with, getting weird errors when trying to remove a vdev.
livinginfear 5y ago
I don't use Apple's products. I've never used Time Machine. I have a Synology NAS though. My understanding based upon reading the article, an the comments here, is that Apple's proprietary Time Machine format is to blame here for this corruption. Can anyone here shed some light on the exact cause for this, and whether this is of any concern to a non-Apple user using a Synology NAS?
- vetinari 5y ago
  My first though was, that there must be something wrong with the cables, and occasionally a packet is damaged. And at the end of the article, there it goes, powerline. If you are using unreliable connection, use something that can verify the transfer, rsync, for example.
  - MonaroVXR 5y ago
    Not sure if that's the issue
    - vetinari 5y ago
      Either that, or bad RAM, and the copy buffer hit it. In the pictures, it seems too regular.
- pfranz 5y ago
  > My understanding based upon reading the article, an the comments here, is that Apple's proprietary Time Machine format is to blame here for this corruption. How did you get that? From reading the article it sounded like they just moved their RAW files to the NAS (no Time Machine involved). One thing they use for backup is Time Machine, the RAW files came from a USB drive that they had wiped, but they forgot to attach it during any backup so they were omitted from the Time Machine backup. The details were light in the post so I'm inferring most of this. I'm not sure how "Apple's proprietary Time Machine format" (it's a disk image and the actual data is just files) were to blame. Time Machine on a NAS (unrelated to Synology) can have problems. macOS generally tells you "verification" failed and recommends starting a new backup...which sucks, is time consuming, and you're at risk of data loss during the initial backup, but is way better than keeping around a corrupt backup. I've used a Time Machine network backup as one form of backup for many many years; both hand rolled and on a Synology. The times I get a corrupted backup I've: manually mounted the backup disk image while a backup started or I disconnected or closed my laptop while backing up. I'm not sure if the problem is on Apple's end or on the server's implementation of Time Machine or the SMB or AFP implementation. I've generally been able to "repair" these backups by running disk utility (fsck) and changing the value in a config file, but that can take a very long time over a network and I'm not sure I'd trust that backup anyway. I think the cause of the problem was close to what the author suspects. Something got corrupt in reading, transferring, or writing the data. Maybe it was non-ECC ram, bits flipped in transit (due to hardware or protocol), or a corrupt disk. The source data seemed ok since he could recover it, the disks appear ok since they wrote ok a second time--but maybe it wrote to a different area of the disk? I've had RAM issues that only showed up under certain circumstances. Maybe "Enable data checksum for advanced data integrity" was on the second time? I kind of think confirming data was written correctly and using a filesystem that protects against bitrot (or keeping hashes of the files yourself) are the most accessible ways to prevent this. Unfortunately, knowing the hashes no longer match means you just know the files are corrupt and won't help you unless you have another copy.
j1elo 5y ago
One thing that surprised me, if they are savvy enough to use the command line and write custom scripts, they would probably have no issue with using PhotoRec instead of buying a commercial recovery software. Not that this kind of money is relevant when it's your personal files that are at stake, just a small observation plus a recommendation for whoever reads this and didn't know about it: https://www.cgsecurity.org/wiki/PhotoRec https://www.cgsecurity.org/wiki/PhotoRec
weinzierl 5y ago
I do md5sum (shasum more recently) out of habit for anything large and at least a bit important to me. I started this about 15 years ago when I had a couple of eSATA drives and noticed corrupted files when copying. I've been checking file integrity since, and occasionally I encounter corrupted files - often enough to make me continue my checks. It was an eye-opener to me to realize that data corruption is much more common than most people think. On the flips-side because single (or few) bit flips often go unnoticed, people overestimate the impact of data corruption. The idea that if your image or video looks good then it must be intact is flawed. Even software binaries survive a bit of corruption pretty well. I think in the long run file system based integrity checks everywhere would be great. For now, in a world with a multitude of storage technologies and file systems, shasums will have to do.
rdevsrex 5y ago
Why not just back them up on S3, B2, or whatever else as well? 2 is 1 and 1 is none.
hughrr 5y ago
I keep my stuff in files only carefully categorised in directories. I use Beyond Compare to do a byte level comparison between what’s on my disk and what’s on the (one of the several disconnected) backup volumes I use. That identifies corruption issues at either side of the fence. I lost about a year of family photos thanks to this sort of issue once so I am very wary of problems.
- bawana 5y ago
  We use beyond compare to do the same for a vast set of municipal documents. We periodically update the collection with new documents. Beyond compare shows shows the new documents as blue(orphans) in the working directory. Interestingly, we occasionally see a red document in the backup indicating a differences the byte level. The documents appear the same in the word processor so we don’t know what’s going on. These are windows boxes and a symbology nas (smb) so no apple involved.
  - hughrr 5y ago
    Might be time stamp or metadata changes in the files if someone opens them. I’ve seen that with some MS office products.
sharmin123 5y ago
Why Is It Necessary To Hire A Professional Hacker?: https://www.hackerslist.co/why-is-it-necessary-to-hire-a-professional-hacker/ https://www.hackerslist.co/why-is-it-necessary-to-hire-a-pro...
bambax 5y ago
It's so easy to think you have backups when you don't. I have the following setup: my main machine is Windows and my photos are on a local NTFS drive, that is backed up (mirrored) on a Netgear NAS via rsync; I rotate the backup drives on a weekly basis. Every time the backup job runs it sends an email to tell me how it went. The email is sent via gmail. This used to work well but at some point (a year ago maybe?) gmail decided this was "unsecure" and stopped forwarding the emails. Instead it sent a notice that "somebody tried to send an email and we blocked it". I couldn't be bothered to fix it and accepted the gmail warning notice in lieu of the actual Netgear backup report. Then eventually I upgraded the email notification system... only to find out that the backups were failing systematically. Luckily nothing was lost as the main drive was fine; I was able to fix the problem and do the backups correctly. But of course it could easily have gone a different way: the main drive could have failed and with empty mirrors there would have been no solution, and I would have had only myself to blame. It is so easy to tell oneself that everything is a-okay when it really isn't.
- sincerely 5y ago
  This is like a story where someone says they strap a blindfold on before driving and then the moral is "it's so easy to drive on the wrong side of the road and cause accidents"
  - bambax 5y ago
    Yeah okay, but I should have added that during the (long) time when the emails were being sent correctly, the backups never failed. So I had a high confidence everything was ok.
    - beervirus 5y ago
      Do you know what happened that caused them to start failing?
      - bambax 5y ago
        I don't remember the exact specifics but it had to do with filename encodings. One file had special characters in its name, rsync on Windows (DeltaCopy) was reporting it existing, but it could not be transmitted and so the remote server kept asking for it, or something like that, and after some back and forth the whole process failed.
benjaminwootton 5y ago
Managing large volumes of digital photos is still an unsolved problem. I have tens of thousands and they are probably my most treasured possession. It’s quite concerning that they are languishing on Google Photos, with a few partial backups floating around that I’m not confident in. I have had a few attempts at cleaning it up but haven’t found the right software. I would also like to print my favourites, which probably also amounts to thousands of pictures, but working through the sheer volume is quite intimidating. Feels like the whole thing needs a better workflow around it.
- kilroy123 5y ago
  I feel the same exact way. I have 60,000+ photos to go through, edit, and remove. Where do I even begin? What app do I use to go through them all? I've searched and tried out many apps and I still don't have a good workflow. I truly don't understand how the pros do it.
  - nszceta 5y ago
    Lightroom. Import all photos from all sources and have lightroom MOVE THE ORIGINALS into a new root folder with the following structure YYYY/MM Keep this folder backed up! It's more important than the metadata catalog but now all the raw imagery is in one place. Then maintain the lightroom catalog with lots of metadata, including using the facial recognition system for offline person recognition, the flag system enabling a 2-pass deletion review, star system for ranking photos by desirability, and add tags to everything. Go through each filter mechanism and develop a strategy for using each filter. Use the image gallery filter system to the limit. If you go through images by date & time captured you can usually bulk tag images much more quickly than if you have to switch between events.
    - kilroy123 5y ago
      I tried and used Lightroom for a while. I didn't want to keep paying Adobe so I tried to switch to darktable[0]. https://www.darktable.org https://www.darktable.org Maybe I'll give lightroom another try.
curt15 5y ago
>It's likely I messed up by not checking the "Enable data checksum for advanced data integrity" option when I created the shared folder. Still, this option is unchecked by default and it's not really clear that leaving it that way could lead to data corruption What's the point of using btrfs if Synology disables one of its signature features?
knolan 5y ago
While I use Photoshop CameraRAW to edit RAW files I don’t use Lightroom or any file library management software. Everything goes into folders on my PC on a dedicated drive which is shared so our Macs can access them and also backed up to BackBlaze. Any JPEGs that I want to share with family are then copied to iCloud photo album or posted to Flickr etc. I was using a WD myCloud that I retired when we moved house a couple of years ago as a secondary backup but that’s not a runner anymore. I’ve 32TB RAID6 volume on my personal office server and I’ll probably use that as a separate offsite backup. We do use Time Machine with a Time Capsule that I’ve recently replaced the disk in. I don’t really trust it and also pay for iCloud Drive for most of my other data. Work also provides Google Drive which I use for data that is shared with others. It’s all a bit of a mess but I think I’ve got some protection in place.
- y04nn 5y ago
  The (not last) version of Lightroom that I use store and organize all the pictures in an accessible directory. The only issue I had when I started using Lightroom was with the import method, I had not selected the "Copy" method and realized that after I deleted the originals. Now I always double check. I would also advise to use multiple Lightroom catalogs, Lightroom makes it easy. Also I don't think Lightroom is the source of the issue here. Anyway that's why it's important to always have at least 2 backup solutions with one being offline.
chromatin 5y ago
This (well, worse corruption than this) is one of my worst data nightmares. Losing pictures of my children, relatives who are now deceased, etc. I know it may sound like overkill, but I switched from a Synology NAS to ZFS for the checksumming specifically to avoid this fate. Not a knock on Synology specifically; I amazing all are about the same in this regard. What I wish is that there were a consumer friendly low power NAS running ZFS … (?) so I don’t have to maintain a server. (And a weekly sweep to AWS glacier to be sure)
account42 5y ago
In addition to backups, error correction data is a must for things you care about. With PAR2 you can add that to any type of file: https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- gruez 5y ago
  Bitrot is a thing, but mitigating it using par files seem suboptimal to me. The biggest problem is that you have to manually create the files (what if you forgot?) or write a script to do that for you (what if the script silently failed?). You're better off using RAID or a filesystem that has it built in (eg. ZFS or REFS).
  - mnw21cam 5y ago
    It doesn't need to be manual. par2 is a command-line program, so it can be automated. Yes, ZFS is better for in-flight data, but it makes sense to create par2 files for your backup archive files, if you're then sending those files somewhere else (which of course you should).
notyourday 5y ago
I use lightroom and i have over 3TB of photos. This is my current workflow. There's no reason why it would not work on a Mac 1. Lightroom runs on Windows 10 system 2. I add new images onto it from one set of cards. The cards are pulled out and sat on a desk. A previous set of cards is added back to the camera and wiped. At this point there are images in the Lightroom and images on the the cards. 3. Every night, windows box rsyncs the Lightroom catalog and raw data onto a Linux box running in a RAID-1 4. Every night, after the rsync is completed Linux box triggers a backup of the rsync'ed catalog and files into S3 compatible remote bucket. This means that I have all images and catalog stored at least twice locally and once remotely. I have also validated that I can recover from a failure of Windows disk, Linux server and remote backup.
HerbsMan 5y ago
I have lost in the past one of my best music creations due to believing in iCloud. I had project, saved on my computer in iCloud directory - After reboot I didn't have the file locally nor in iCloud. Literally gone :)
devwastaken 5y ago
Apple wants you to buy a whole new MacBook for the extra storage, and only use their icloud for backup. Apple intentionally has bad implementations of network filesystems. They know it, but as per usual, they know their customers won't do anything about it.
mikehollinger 5y ago
Based on this, and on my own experiences, it looks like it's time to move on from Synology. But... to what? I've had a DS412+ since, well, 2012. I've dutifully replaced drives as they went bad, run disk scrubs, file system scrubs, and (in the last nine years) had to occasionally run time machine restores from it. It worked like a champ each time, including "restoring" from an old computer to a brand new one. However, in the last couple of years, we couldn't get my husband's machine to back up or restore successfully, and now out of our three macs, only one is able to successfully back up at this point, and (looking at this) I'm skeptical that it's actually working.
- pembrook 5y ago
  > it looks like it's time to move on from Synology I don’t think this is a Synology problem. No Windows users are reporting this issue. On the other hand, NAS devices have always been treated as second class citizens by Apple. If reliable network attached storage is important to you, maybe it’s time to move on from MacOS? As much as I hate Windows, using my NAS with it is a dream. It’s how the experience of network storage was supposed to feel.
- eddieroger 5y ago
  > The whole ordeal was largely of my own making That's not the conclusion the author came to - why is it the one you came to? I use a Synology with a Mac nearly every day. I don't use it for Time Machine, but that's just because I use different backup strategies, like storing the files directly on the Synology, or copying them over after local editing depending on the type of file. Also, like the author learned, I check my backups periodically to make sure the file is the same on the other side. > I'm skeptical that it's actually working You should test it then. If you've never tested you backups, you have none.
  - mikehollinger 5y ago
    > You should test it then. If you've never tested you backups, you have none. I have, and so far we're one out of three. Mac A seems to restore (at least the last time we tried it) Mac B failed to restore Mac C fails to back up
parasubvert 5y ago
Samba (on Linux, not just Mac) has always had corruption issues for large files related to locks and filesystem caching quirks. Windows and the protocol itself has a different approach to locks than *NIX. Many have gotten more stable by disabling caching (though this hurts performance on low bandwidth / high latency fileshares). Also from what I recall, Apple SMB is based on a forked version of Samba from 10 years ago. They did this when Samba went GPLv3. Not clear how well they’re keeping up with quality improvements. My solution: use WebDAV on your Synology for transferring files from a Mac or Linux host. It has its own quirks but is far more reliable. Unfortunately this won’t work for time machine. But would work with “rsync -aXv —partial”.
sieve 5y ago
A mirrored ZFS setup with one offline backup and one cloud backup will prevent nonsense like this.[1] If you know what you are doing, you can do a lot of stuff with ZFS even on a single disk.[2] [1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ [2] Forbidden Arts of ZFS | Episode 2 | Using ZFS on a single drive [https://www.youtube.com/watch?v=-wlbvt9tM-Q https://www.youtube.com/watch?v=-wlbvt9tM-Q]
- bluGill 5y ago
  The problem is I get home at night and I want to spend time with my kids, not admin my ZFS system. Which is why I'm running a really old version of FreeNAS, it has owncloud (not nextCloud) which fails to start since the last time I updated freeNAS, and I haven't figured out how to make it work again - mostly for lack of time/interest in doing it.
  - sieve 5y ago
    > not admin my ZFS system. Ubuntu/Manjaro comes with default ZFS support. There is no administration involved beyond the initial setup.
    - bluGill 5y ago
      There is still administration required. Sure ZFS itself is easy, but the rest of the ecosystem on top of that is hard. You need a lot more software than zfs to make something useful.
- smarx007 5y ago
  I am not sure. Quite possible bad data was written over SMB, not necessarily that the drives failed. As you see in the article, the damage to images is systemic and drives don't seem to be failing. More like Lightroom/macOS/macbook RAM/Samba wrote (accepted) bad data.
  - sieve 5y ago
tannhaeuser 5y ago
Switched to using SD cards for Time Machine-like backup (using BackInTime) plus monthly archival on big disks on Linux for many years now; requires some organization due to space restrictions, though. Distinctly remembering when iPhoto (5?) wanted me to convert/store my private photos "in the database" rather than as plain files on my old PowerBook which I refused to do however, knowing all too well this would've been a bad idea.
- cl3misch 5y ago
  Aren't SD cards pretty limited in terms of write durability?
  - jnieminen 5y ago
    There are high endurance SD cards, but eventually those will have also IO problems.
stewx 5y ago
I picked up a new DSLR and found out my copy of Lightroom 3 didn't support the new RAW format. Not willing to pay $125USD/year for the "latest and greatest" version of Lightroom, I found out I could use Adobe's free DNG conversion tool to just convert the fancy new format to DNG and continue using my bought-and-paid-for copy of Lightroom. Also, major plus: it supports lossy conversion, which churns out files ⅓ the size of the originals, with no perceptible loss in quality. I ended up converting my entire photo catalog, saving hundreds of GBs of disk space. The tool has a CLI as well. https://helpx.adobe.com/camera-raw/using/adobe-dng-converter.html https://helpx.adobe.com/camera-raw/using/adobe-dng-converter...
- mnw21cam 5y ago
  dcraw is a free tool that supports pretty-much every camera that has ever existed. You could try that too.
- sillysaurusx 5y ago
  From https://photographylife.com/dng-vs-raw https://photographylife.com/dng-vs-raw > DNG strips out most of the unrecognized meta data (such as Active D-Lighting, Picture Controls, Focus Point, etc) from RAW files, making it impossible to retrieve this data from DNG in the future. The last time I used a DSLR was when I was doing color research in 2012. At the time, raw format was the only thing that preserved all necessary information to make scientific observations. Most people don't need to care about such things. I just wanted to mention you're irretrievably throwing away metadata when you DNG-dong your pictures. It's probably a worthwhile trade in most cases though.
  - matja 5y ago
    > At the time, raw format was the only thing that preserved all necessary information to make scientific observations. Which "RAW" format? I think that is the problem - there are many and few are completely documented, which is where DNG comes in (https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf https://www.adobe.com/content/dam/acom/en/products/photoshop...). Great point about throwing away metadata, and probably worthwhile to safeguard that one's flavor of RAW files (CR2/NEF/etc) can be reliably read in the future when the software necessary to read them inevitably disappears from the cloud.
rcarmo 5y ago
I haven’t had any issues with my Synology NAS yet, but I did have to stop using my iMac due to recurring _local_ data corruption errors: https://taoofmac.com/space/blog/2021/08/08/1600 https://taoofmac.com/space/blog/2021/08/08/1600 I wonder if the OP also has a Fusion drive - merely reading off a corrupt SSD cache or bad RAM would explain some of the damage.
shell0x 5y ago
I also have no idea how to organize photos and backups well. As an engineer, I'm good doing it for enterprise stuff, but it's really challenging for personal stuff. I don't trust Google products anymore, I got burned too many times when they killed their products. They work, but I just don't want to use them anymore. I currently have a 100GB OneDrive subscription for $3, so that works for now, but I'm pretty close to the storage limit. I'm not a Microsoft fan, but considering it's integrated with Windows now, I assume it will stick around for long. The same is probaly true with Apple and iCloud. I've relocated internationally a dozen times, so a NAS doesn't work for me. There are also other options for just storage like S3 or Azure storage, but the price seems to be almost the same as OneDrive.
- unixhero 5y ago
  +1 for Onedrive. I pay usd6 I think for 1 TB with a valid Office365 license. That's a pretty sweet deal. Not a microsoft fan either, however this works well. Tight integration with Linux CLI tools, Android Onedrive app is excellent et cetera.
  - lostmsu 5y ago
    > Android OneDrive app is excellent If only they would actually sync files outside the app data dir like they do on desktop.
- tilolebo 5y ago
  Backblaze B2 is a really good option for remote backups. I use it to make daily incremental backups of my NAS, but you could replace the NAS by a USB drive.
- stewx 5y ago
  My online backup recommendation: Backblaze. No fuss, no muss. No limits.
- vaughan 5y ago
  > I've relocated internationally a dozen times, so a NAS doesn't work for me. Same here. I used to use 2x Seagate portable 5TB. Would mirror one to the other every week. Now I use a Western Digital MyBook 20TB in RAID 1, and backup this weekly to my portable externals using Carbon Copy Cloner for macOS (ideally I would have an identical second one for this). And run Backblaze for offsite backup. I avoided a NAS because I prefer the speed of USB3 versus network - super quick to view RAW and video, and for ease of relocation. I also have a 1TB portable SSD for my Apple Photos syncing iPhone content without filling up my MacBook disk. The WD MyBook's are a bit dodgy though, there are so many horror stories. They encrypt all data, so if your RAID config is corrupted...which is supposedly stored on the hard disk, then you are screwed. Also, when you tell it to sleep when inactive, whenever you open a file dialog, the laptop freezes for 5/10s while it wakes up. There is also a bug in the utility application that says "can't access drive" which makes you think your drive is broken but it's actually just a software bug. Not running the utility software fixes it. I also think it can cause some unable to sleep issues for macOS. Sometimes it is super loud too, and its quite noisy.
mwidell 5y ago
Don't complicate things. I did lots of research of different NAS solutions etc. but in the end I just got a huge external USB drive and then automatic backup with backblaze for $60 per year. Works great and super cheap, plus I have offsite backup in case of fire or theft.
- ghaff 5y ago
  And you can add one or two additional USB drives for redundancy. I suspect that a lot of people building NASs using btrfs configured as RAID whatever would be better with a few local USB drives plus Backblaze.
  - horsawlarway 5y ago
    I did the whole "Save it to a large usb drive" method for backing up personal files for a while. I finally upgraded to a synology nas and I feel a little silly for not doing it earlier. It's far more convenient given I bounce between 3 machines during the day (personal laptop, work laptop, desktop), my wife can easily access files on her machines as well. Added bonuses are that it comes back up on its own after a power outage, doesn't require my desktop to be on for me to hit it externally or for a successful backup, and I don't have to remember where I last placed the usb drive. Basically - USB drives totally work, but the NAS is better in pretty much every way outside of price (and possibly some configuration, if the person isn't very technical).
    - mackrevinack 5y ago
      you can still make good use of the usb drive with hyperbackup on synology! i have a backup job set to run nightly but keep the drive disconnected (so something like a powersurge can't take out all the disks at once) and then once a month plug in the drive before i go to sleep then plug it back out the next day. the backup job gracefully fails for the rest of the month when the drive is not plugged in, but basically it just means i don't have to log into the synology just to run the backup job. i have a second drive with an identical backup job and the drive lives in my sister's house 364 days of the year. there's still the risk of losing a months worth of files but its a pretty low effort and cheap backup system apart from that
G4E 5y ago
Before buying an expensive software for data recovery, don't forget about Photorec[0] and its dangerously powerful brother TestDisk[1]. It will be able to help you more often than not, and it's FOSS. [0] https://www.cgsecurity.org/wiki/PhotoRec https://www.cgsecurity.org/wiki/PhotoRec [1] https://www.cgsecurity.org/wiki/TestDisk https://www.cgsecurity.org/wiki/TestDisk
- pavel_lishin 5y ago
  I don't think $79 is particularly expensive to recover half a terabyte of valuable photos, but thank you for the recommendation!
  - sigg3 5y ago
    It's expensive if you ain't got it. I used testdisk as a student living on loans :)
    - iluvcommunism 5y ago
      I tried using test disk but I had an obscure situation. I was recovering from a failed REFS pool.
  - smoldesu 5y ago
    It's considerably more expensive than $0, and just because $79 doesn't seem like a big issue in the heat of passion doesn't mean that you won't kick yourself for it later.
- webmobdev 5y ago
  Let me also chime in with a free and open source decent alternative to LightRoom - https://www.darktable.org/ https://www.darktable.org/ ...
- squarefoot 5y ago
  +1 for TestDisk. That gem literally saved my butt years ago when a controller failure on my old NAS almost destroyed all my data by turning all disks (Linux soft RAID1) into unreadable doorstoppers. And it wasn't like just a damaged partition table or anything like that; each file had to be recovered and copied elsewhere.
drittich 5y ago
I use Lightroom on Windows and also Logic Pro X on Mac, and both save to my Synology. This post is terrifying me, because I don't know an easy way to validate that my Logic Pro audio projects are not corrupted on the NAS. With a photo you can just look at it. With an audio project I would need to listen to the whole thing, and when there are 32 tracks or more to be checked, that can be unmanageable. For Lightroom, I keep the library on my local SSD drive and the photos on the NAS. One thing to note is that you can import photos into a Lightroom library in-place (from wherever they currently reside), or you can have Lightroom copy or move them to a location of your choice, with a few options around how to organize the files. In this case, the Lightroom software itself could well be the cause of the corruption, as it is doing the copying. I copy the files to a destination on the NAS using a drive-mapped letter, but it works with UNC paths as well. On Windows, I have ever experienced any corruption in about 8 years of using the Synology/
ghaff 5y ago
Something to be aware of the Lightroom catalog itself (which is unrelated to the OP's problem) is that it's an SQLite database. (Or it was when I ran into my problem about a decade ago bit I believe it still is.) I had hardware-related flakiness with a Windows system at the time. I swapped it out for an iMac but sometime later I discovered that there was some residual corruption of my Lightroom catalog even though I had restored a backup that seemed to be OK. Fortunately I was able to use some SQLite tools to largely fix the catalog. https://bitmason.blogspot.com/2013/01/recovering-corrupt-lightroom-catalogs.html https://bitmason.blogspot.com/2013/01/recovering-corrupt-lig...
patorjk 5y ago
Nikon RAW files store a full resolution jpg version of the photo inside of the RAW file (this may be true of other RAW formats too). I had a similar problem to what's described in the article a few years ago, and for photos that were damaged which I didn't have backups I was able to get the jpg out of the RAW by running this command: exiv2 -e p4 image_name.NEF Apparently this jpg is what the camera shows on its backscreen when you're previewing the image/RAW file.
aeharding 5y ago
My bet is the problem is related to the USB transfer. I've read about similar problems migrating data to Synology via USB. Don't remember where I read that, though...
ubermonkey 5y ago
Gaaaah. This is mostly not about Lightroom, or photos, or any of those specifics. This is about (a) a failure in a migration followed by (b) *failure of the user to verify the migration before deleting the original copies*: >To make matters worse, I deleted the photos from the USB hard drive, without verifying the new catalog. That line is, not coincidentally, where I stopped reading.
- selfhoster11 5y ago
  It's not obvious to amateur users that these safety measures must be in place for data migration projects. Members of my family only learnt after the first data loss.
rrauenza 5y ago
Anyone using lightroom should also consider configuring it to always drop the XMP sidecar file with your raw files. If the DB is ever corrupted beyond repair, you'll at least have a record of your current edits per file. https://fstoppers.com/post-production/most-important-setting-lightroom-set-default-8366 https://fstoppers.com/post-production/most-important-setting...
iluvcommunism 5y ago
I had a similar experience. Lost close to 9 yrs of photos and my wedding photos. I eventually got them back via a similar method as author. It took me a year to go through them all. After that experience I fully went to the cloud. The cause was a REFS mirror deciding to “mirror” my corrupt drive to my working drive. Great job MS. I also stopped using Microsoft for my raid stuff.
- lostmsu 5y ago
  > REFS mirror deciding to “mirror” my corrupt drive to my working drive REFS has an option for integrity checking, which (unfortunately) might be off by default.
  - iluvcommunism 5y ago
    Yes I forget if it was on or not. You are correct. On the Windows OS (not server) you have to run a command to enable it. I will never forget the feeling I had watching the space on my good drive dropping precipitously as it was wiped away into “oblivion.”
jquaint 5y ago
Slightly off topic: I recently switched to Capture One for all of my photography needs. I would highly recommend it over Lightroom. Rather than subscribing to a service you can just buy it outright. It has all the features of Lightroom with some extras.
- 153957 5y ago
  Similarly I recently switched the other way. I recommend Lightroom over Capture One. The subscription is a slight downside, but Capture One is similarly expensive if you want to keep up with the latest version, I know you don't have to, but if you do it is expensive. Moreover Capture One is missing some features which Lightroom has: The ability to filter collections based on their name. A shortcut for 'increase/decrease rating'. Filter photos for 'x stars and higher'. Show all images in a folder, including those in subalbums/subfolders. Being able to mark photos with a 'Reject' flag/rating (and easily hiding, not removing, those).
- jslpc 5y ago
  I tried switching to Capture One as well, and enjoyed using it for the most part, but I couldn't fully switch for one primary reason. I'm still fairly new to photography and use a few preset packs that I've paid for as a base before making modifications to them to my liking. I downloaded a program that converts Lightroom presets to LUTs, and those worked decently, but they were visibly inferior due to the fact that the converter cannot handle camera/lens profiles. I ended up just going back to Lightroom Classic for now, but hopefully once I progress enough to make my own edits from scratch I'll be able to switch back to Capture One.
donut2d 5y ago
Last year all of my photos and presets had disappeared after updating the Lightroom app. That was 2+ years of edits that are just gone, lost, unrecoverable. I do photography as a hobby so I never saw a need for backing up photos and I never paid for the subscription (which would include cloud storage) because I didn’t use any of the tools that came along with the subscription.
Saris 5y ago
This sounds like it doesn't have anything to do with Lightroom, but is instead file corruption caused by some kind of bug. Lightroom catalogs also don't contain any images, I think a lot of people are confused and think it does.
ksec 5y ago
>Lessons Learned The whole ordeal was largely of my own making. I never anticipated that the simple act of copying files from one place to another could go so horribly wrong. It isn't user's own making or user's fault at all! ( A voice is literally screaming this inside my head ) Tech shouldn't be like this. The DS218 support BTRFS which prevent certain file corruption but this is not what's happening either. I have been saying ( or ranting ) about this for now more than a decade. [1] And every time the answer was there are no market for it. Or consumer or market are not willing to pay premium for high quality NAS. The Kobol NAS was my hope that a high quality, reasonably priced, and somewhat reasonably easy to setup and managed NAS would be. But they are closed down as well [2]. I really hope Kobol release all of the work as open source including hardware design. May be someone ( or me ) could crowdfund it once the chip shortage is over. [1] https://news.ycombinator.com/item?id=10886702 https://news.ycombinator.com/item?id=10886702 [2] https://news.ycombinator.com/item?id=28299573 https://news.ycombinator.com/item?id=28299573
mleonhard 5y ago
I am writing a Rust CLI program for managing my backups: https://github.com/mleonhard/deduposaur https://github.com/mleonhard/deduposaur I tend to end up with multiple copies of photos and other docs: SD card, laptop, backup drives, and USB thumb drives. Deduposaur helps me to move files into my latest backup drive. It flags duplicates, modified files, and files that I previously deleted. It also checks each file's SHA-256 digest to detect corruption. The program is usable but needs some usability improvements and more tests. The license is Apache 2.0. I am happy to receive problem reports, feature requests, and pull requests.

249 points by pchm 5y ago

Personally I wouldn't use that NAS for storing data until figuring out what went wrong, since every single file was corrupted. This is way too high an error rate and unlikely to be caused by a hard drive or network issue. It's probably an issue caused by NFS/SMB/... client/server bugs. Edit: Quick googling finds many people using Macs report files are corrupted when written to a Synology NAS over SMB.

cube00 5y ago

This is confusing, it sounds like the RAW files themselves are corrupt so why bring any discussion of the Lightroom catalog into it? Having said that people need to take care backing up their catalog because without that single catalog file you will lose decades of photo editing work in Lightroom.

tiernano 5y ago
Given how photos are "imported" into a Lightroom Catalog, some might think thats where they live... On a Mac, IIRC (its been a while since i used LR on a Mac) if you look at your catalog, its a single large "file" with all the meta data, edits, originals, etc, in one place...
- iriche 5y ago
  This isn't really correct, the original image is always stored in the original place or the place you select to copy it too. That's why I also think the title is a bit miss leading, cause it was never the Lightroom catalog that was the issue, it was the RAW-images them self.
- Saris 5y ago
  The catalog only contains the edits and changes to an image, it doesn't actually contain any images. Lightroom can move or copy images to a new folder for you, but they're still just normal image files you can open with any supported software.
pembrook 5y ago
Yes, Lightroom has nothing to do with it. Seems like that’s simply the app that the author is using to view the files. The title of this post should be changed to “file transfers from MacOS to Synology NAS causes file corruption” It’s a known issue with no fix at the moment.
- otterley 5y ago
  Known to whom? And do you have a reference to this information from either Apple or Synology?
lentil 5y ago
You can also set Lightroom to write sidecar xmp files, which means every edited raw file will also have its own copy of the edits kept alongside it. Coupled with using a good directory structure, that significantly lessens the impact of losing a catalog file.

gorgoiler 5y ago

Lightroom is not a word I am comfortable reading. Not since my archive copy became uninstallable in macOS (32 bit installer) and Adobe required a monthly subscription just for me to have access to my catalog. I have been made a fool of.

beermonster 5y ago
Better off using digikam to as a DAM and being in charge of your own data.
- toastal 5y ago
  Darktable as well
  - beermonster 5y ago
    Yes. They’re both fantastic. Though digikam is a closer Lightroom replacement.

teekert 5y ago

Time Machine backups should live on Apple Filesystems. The Ext4 or btrfs in Synology's is a timebomb for Apple's proprietary data blobs. It has something to do with the filesystem attributes. I asked my father to just give up on it. Just manage pictures through the file system. Photos (former iPhotos) databases also get corrupt on a Synology NAS. It's a matter of time. The strange thing is that it can go well for months on end, giving you a false sense of security. Do iCloud or local drive backups or stay away from Time Machine and Apple Databases is my advice (although my father had a lot of problems as well with an "incorrectly unplugged" external HFS+ drive). I can't find the sources right now but after my father's last drama (and there have been several) I did some intensive searching and this was my conclusion.

Maakuth 5y ago
Are these applications really adding so much value that it's worth it with their proprietary catalogs? My personal solution for photo storage is a normal directory tree with files. I have Piwigo set up to catalog those so that I can browse by shooting date and tags, but the file hierarchy stays untouched. I can pull files from there to edit with any application and there's a lot of proven tools to keep the files in good shape and backed up.
- teekert 5y ago
  If you're in the Apple ecosystem with an iPhone, some iPads and a Macbook, you sort of roll into it. Is it worth it? Not sure, it's sure easy and just works. But you're on your own when you try to interface with your data in "non conventional" ways. I myself use NextCloud for everything, I recently moved from Android to iOS and it's nice to see most things working... except that NextCloud has issues making previews from .heic pictures (or I should say heic picture containers containing heif images coded in hevc :s), and so the drama starts again. It's always plug and pray outside the Apple ecosystem, always ymmv.
  - MereInterest 5y ago
    I like your quotes around "non conventional", as everything about storing files in a directory is the traditional and conventional way to store files.

sneak 5y ago

It looks like he’s using HFS or APFS. These filesystems still lack data checksums for data on disk, aside from any of the other issues he ran into. The only sane way to deal with unique/irreplaceable data (i.e. one’s own photos) where the authoritative copy is on a mac is to make several independent copies, with several kept offline, and assume that the macOS or Adobe’s terrible house of cards that is the creative suite is going to fuck everything up at some point. I say this as someone with >1000GiB in a Lightroom catalog, run on a mac. I have a copy on an always-connected external disk (made with rsync && rsync -c), a copy on a ZFS NAS (made with rsync && rsync -c user@host:), a Time Machine backup that is only connected weekly, and an offsite rsync to a USB drive, with a SHASUMS file sidecar. I also copy the RAWs into the LR masters photo structure when I import them, and they go first from the camera into a Syncthing folder (pre-import) that is immediately and automatically replicated onto four machines in three buildings (two of which are ZFS with autosnapshots).

abcd_f 5y ago

> A quick googling indicated a hard drive problem In the vast majority of cases spontaneous data corruption happens in _transit_ due to RAM glitches. All modern drives implement forward error correction on per sector basis. This allows the drive to automatically repair up to 10% of damage to any given sector... in which case correct data is returned to the requestor and the sector is either tagged for relocation or is relocated right away. In cases when the data can’t be cleanly recovered, the read request us failed. That is, chances of a read request returning mangled data from a disk is next to absolute zero. Meaning in turn if you do see data corruption, it happened before this data hit the disk - i.e. it happened in transit.

lmilcin 5y ago
In vast majority of cases corruption (that doesn't involve drive failure) happens due to bad or naive implementation of software that handles files. While memory does fail sometimes, if it was failing at the rate you describe PCs would not be suitable to any work at all. I have worked in ops for many years. A lot of software that copies files is perfectly happy to leave you with copy of different length than original.
- abcd_f 5y ago
  Software bugs is an altogether different issue. As it is a far more exotic one. The context of the OPs post and my reply is the case when copying in bulk with a mature tool yields corruption in a small fraction of the data. In this case the cause is the in-transit corruption (rather than at-rest, which is a fairly common belief dubbed as a “bitrot” phenomenon).
  - lmilcin 5y ago
    Software bugs exotic. That is funny. File IO is one of more error prone activities with low understanding by general software dev community. It is actually rare to see a piece of code that is NOT broken. I had once observed a piece of software write a million files, all empty, because it could not handle situation where it could create file but not write it. It is quite normal for files to be truncated because written directly to destination file rather than through temp. Then script gets killed and next instance assumes file exists so it resumes from next. Most software can do the job if everything works, but almost none can correctly handle every error condition.

ygra 5y ago

I once got a bit of a scare in that many of my raw files has weird color patterns in them. Even the backups. Turns out, I had faulty RAM and Lightroom happened to always hit the faulty parts when reading a raw image, apparently.

hackshack 5y ago

This is incredibly frightening. I’ve seen data rot up close, and as a former data janitor, I can sympathize. My advice is to control everything down to the wall socket. I considered the Synology boxes for my home NAS, but my “spider sense” was tingling, so I built around an HP MicroServer Gen8 with an LSI PCIe card, and mirrored drives. It’s on a double-conversion UPS (Smart-UPS RT). It runs FreeNAS on VMware ESXi, and the drives present as raw volumes to ZFS. No data integrity issues after ~3 years. Note the trend: clean power, good server-y hardware, matched drives in a simple RAID, properly resourced ZFS, etc. But yeah, spinning disks are generally not a “backup.”

Maakuth 5y ago
Even in this setup there's a caveat: HP sells Microservers with non-ECC hardware too. I think after human error, in your setup the likeliest threat to data integrity is faulty memory modules propagating corruption to disks (in essence you'll have a dirty buffer in faulty RAM). ZFS helps you catch this, but it's not magic so be sure to have the ECC version.
deleted 5y ago
[deleted]

lmilcin 5y ago

Even worse because Lightroom normally does not modify the files. I am using separate storage for raw files and Lightroom database for this reason. Immutable files should be the easiest case to handle reliably.

tomaszs 5y ago

What I find work best for storing data is a regular USB drive. With no software like Lightroom, no NAS. Just a disc, you move data with a regular file manager. You don't get all the cool features, but process is pretty safe and you don't worry about hackers because it is offline most of the time.

tikkabhuna 5y ago
How do you protect yourself from losing data when the drive breaks?
- kylehotchkiss 5y ago
  Arq.app can backup external disks to most of the big cloud storage providers really well. Unlike Backblaze, they don’t delete these backups if you don’t connect your drive for a specific amount of time. I’ve used Arq to backup about 2TB of external drives to AWS Glacier. It even let me set the encryption key for the data myself!
  - jnieminen 5y ago
    How is your experience with restoring? Have you had expensive data retrieval costs?
    - kylehotchkiss 5y ago
      I used to backup with Arq to S3 and the restores were very fast. With Glacier, Arq initiates the data retrieval and waits for file availability in the UI. Glacier can take a day or so, laptop needs to be open and connected to internet during the time for the transfer to complete. Given I'm primarily backing up external drives, I find this UX not an issue, but if you wanted backups with instant accessibility, backing up to S3 really isn't that pricy.
- tomaszs 5y ago
  I have second backup disk that has snapshots of the backup disk. It is unlikely two disks from two manufacturers will break in the same time. So it gives me fairly good level of safety

jwr 5y ago

This sort of thing scares me. It's why I started running consistency checks on my important archives (like my photo library), which I keep backed up in multiple places. We tend to think that in a digital world bits are just bits and do not get corrupted — which is decidedly untrue. I wrote my own consistency checker, as I wasn't happy with what was out there. I wanted it to be simple, and maintainable in the long term (>10 years horizon). See https://github.com/jwr/ccheck https://github.com/jwr/ccheck if you need something like this. I now update my checksums regularly and check for corruption.

teekert 5y ago
Still, using rsync with checksumming enable can get your Photos library messed up if you sync (or just copy) to a non APF or HFS+ file system. Be very careful.
- nix23 5y ago
  How is that possible?
  - teekert 5y ago
    I read that some of the key information is in attributes unique to Apple's file systems, I can't find the source but that was my conclusion after trying to figure out what went wrong in the Time-Machine/Synology/Rsync-via-webdav system I set up for my father some years ago. If you rsync ext4 files to exfat, you also have issues, but those are very clearly reported when you attempt to do so.
    - nix23 5y ago
      It makes sense that you loose some information from a posix filesystem to exfat, but from HFS+ to lets say XFS? Both can have extended attributes..hmm However hold on, i do some research. Now you hooked me ;) EDIT: Ok got it, both Filesystem NEED xattrs enabled, and you need to run rsync with the -X or --xattrs option, that should fix the problems.

trzeci 5y ago

I found myself, that my enthusiastic approach to photography does not utilize the CC plan on Adobe to pay ~10$/month. What I do is to import images from SD + do a base color correction. What I found unique and useful is face recognition that Lightroom offers. Nonetheless, I decided to move on to either open source software or software that I can buy. At the moment I'm testing On1 - but I wanted to check if somebody from HN has either own solution that wanted to share or something to recommend.

deleted 5y ago

[deleted]

logotype 5y ago

Important backups should be stored in a variety of ways. I store my backups on multiple external HDD’s (solid state and spinning disks) in redundancy. Using multiple filesystems (ext4, and exFAT). I do use ZFS for daily/active storage but not for backups. Besides that I also store the data in AWS S3 and a duplicate in Backblaze B2. These solutions are scalable. iCloud/Google Drive is good if you only have 2TB or less. Just store the raw files directly in a regular file tree structure and avoid any kind of catalog/index/db file. I do find ZFS very interesting but in practical use it has been trouble working with, getting weird errors when trying to remove a vdev.

livinginfear 5y ago

I don't use Apple's products. I've never used Time Machine. I have a Synology NAS though. My understanding based upon reading the article, an the comments here, is that Apple's proprietary Time Machine format is to blame here for this corruption. Can anyone here shed some light on the exact cause for this, and whether this is of any concern to a non-Apple user using a Synology NAS?

vetinari 5y ago
My first though was, that there must be something wrong with the cables, and occasionally a packet is damaged. And at the end of the article, there it goes, powerline. If you are using unreliable connection, use something that can verify the transfer, rsync, for example.
- MonaroVXR 5y ago
  Not sure if that's the issue
  - vetinari 5y ago
    Either that, or bad RAM, and the copy buffer hit it. In the pictures, it seems too regular.
pfranz 5y ago
> My understanding based upon reading the article, an the comments here, is that Apple's proprietary Time Machine format is to blame here for this corruption. How did you get that? From reading the article it sounded like they just moved their RAW files to the NAS (no Time Machine involved). One thing they use for backup is Time Machine, the RAW files came from a USB drive that they had wiped, but they forgot to attach it during any backup so they were omitted from the Time Machine backup. The details were light in the post so I'm inferring most of this. I'm not sure how "Apple's proprietary Time Machine format" (it's a disk image and the actual data is just files) were to blame. Time Machine on a NAS (unrelated to Synology) can have problems. macOS generally tells you "verification" failed and recommends starting a new backup...which sucks, is time consuming, and you're at risk of data loss during the initial backup, but is way better than keeping around a corrupt backup. I've used a Time Machine network backup as one form of backup for many many years; both hand rolled and on a Synology. The times I get a corrupted backup I've: manually mounted the backup disk image while a backup started or I disconnected or closed my laptop while backing up. I'm not sure if the problem is on Apple's end or on the server's implementation of Time Machine or the SMB or AFP implementation. I've generally been able to "repair" these backups by running disk utility (fsck) and changing the value in a config file, but that can take a very long time over a network and I'm not sure I'd trust that backup anyway. I think the cause of the problem was close to what the author suspects. Something got corrupt in reading, transferring, or writing the data. Maybe it was non-ECC ram, bits flipped in transit (due to hardware or protocol), or a corrupt disk. The source data seemed ok since he could recover it, the disks appear ok since they wrote ok a second time--but maybe it wrote to a different area of the disk? I've had RAM issues that only showed up under certain circumstances. Maybe "Enable data checksum for advanced data integrity" was on the second time? I kind of think confirming data was written correctly and using a filesystem that protects against bitrot (or keeping hashes of the files yourself) are the most accessible ways to prevent this. Unfortunately, knowing the hashes no longer match means you just know the files are corrupt and won't help you unless you have another copy.

j1elo 5y ago

One thing that surprised me, if they are savvy enough to use the command line and write custom scripts, they would probably have no issue with using PhotoRec instead of buying a commercial recovery software. Not that this kind of money is relevant when it's your personal files that are at stake, just a small observation plus a recommendation for whoever reads this and didn't know about it: https://www.cgsecurity.org/wiki/PhotoRec https://www.cgsecurity.org/wiki/PhotoRec

weinzierl 5y ago

I do md5sum (shasum more recently) out of habit for anything large and at least a bit important to me. I started this about 15 years ago when I had a couple of eSATA drives and noticed corrupted files when copying. I've been checking file integrity since, and occasionally I encounter corrupted files - often enough to make me continue my checks. It was an eye-opener to me to realize that data corruption is much more common than most people think. On the flips-side because single (or few) bit flips often go unnoticed, people overestimate the impact of data corruption. The idea that if your image or video looks good then it must be intact is flawed. Even software binaries survive a bit of corruption pretty well. I think in the long run file system based integrity checks everywhere would be great. For now, in a world with a multitude of storage technologies and file systems, shasums will have to do.

rdevsrex 5y ago

Why not just back them up on S3, B2, or whatever else as well? 2 is 1 and 1 is none.

hughrr 5y ago

I keep my stuff in files only carefully categorised in directories. I use Beyond Compare to do a byte level comparison between what’s on my disk and what’s on the (one of the several disconnected) backup volumes I use. That identifies corruption issues at either side of the fence. I lost about a year of family photos thanks to this sort of issue once so I am very wary of problems.

bawana 5y ago
We use beyond compare to do the same for a vast set of municipal documents. We periodically update the collection with new documents. Beyond compare shows shows the new documents as blue(orphans) in the working directory. Interestingly, we occasionally see a red document in the backup indicating a differences the byte level. The documents appear the same in the word processor so we don’t know what’s going on. These are windows boxes and a symbology nas (smb) so no apple involved.
- hughrr 5y ago
  Might be time stamp or metadata changes in the files if someone opens them. I’ve seen that with some MS office products.

sharmin123 5y ago

Why Is It Necessary To Hire A Professional Hacker?: https://www.hackerslist.co/why-is-it-necessary-to-hire-a-professional-hacker/ https://www.hackerslist.co/why-is-it-necessary-to-hire-a-pro...

bambax 5y ago

It's so easy to think you have backups when you don't. I have the following setup: my main machine is Windows and my photos are on a local NTFS drive, that is backed up (mirrored) on a Netgear NAS via rsync; I rotate the backup drives on a weekly basis. Every time the backup job runs it sends an email to tell me how it went. The email is sent via gmail. This used to work well but at some point (a year ago maybe?) gmail decided this was "unsecure" and stopped forwarding the emails. Instead it sent a notice that "somebody tried to send an email and we blocked it". I couldn't be bothered to fix it and accepted the gmail warning notice in lieu of the actual Netgear backup report. Then eventually I upgraded the email notification system... only to find out that the backups were failing systematically. Luckily nothing was lost as the main drive was fine; I was able to fix the problem and do the backups correctly. But of course it could easily have gone a different way: the main drive could have failed and with empty mirrors there would have been no solution, and I would have had only myself to blame. It is so easy to tell oneself that everything is a-okay when it really isn't.

sincerely 5y ago
This is like a story where someone says they strap a blindfold on before driving and then the moral is "it's so easy to drive on the wrong side of the road and cause accidents"
- bambax 5y ago
  Yeah okay, but I should have added that during the (long) time when the emails were being sent correctly, the backups never failed. So I had a high confidence everything was ok.
  - beervirus 5y ago
    Do you know what happened that caused them to start failing?
    - bambax 5y ago
      I don't remember the exact specifics but it had to do with filename encodings. One file had special characters in its name, rsync on Windows (DeltaCopy) was reporting it existing, but it could not be transmitted and so the remote server kept asking for it, or something like that, and after some back and forth the whole process failed.

benjaminwootton 5y ago

Managing large volumes of digital photos is still an unsolved problem. I have tens of thousands and they are probably my most treasured possession. It’s quite concerning that they are languishing on Google Photos, with a few partial backups floating around that I’m not confident in. I have had a few attempts at cleaning it up but haven’t found the right software. I would also like to print my favourites, which probably also amounts to thousands of pictures, but working through the sheer volume is quite intimidating. Feels like the whole thing needs a better workflow around it.

kilroy123 5y ago
I feel the same exact way. I have 60,000+ photos to go through, edit, and remove. Where do I even begin? What app do I use to go through them all? I've searched and tried out many apps and I still don't have a good workflow. I truly don't understand how the pros do it.
- nszceta 5y ago
  Lightroom. Import all photos from all sources and have lightroom MOVE THE ORIGINALS into a new root folder with the following structure YYYY/MM Keep this folder backed up! It's more important than the metadata catalog but now all the raw imagery is in one place. Then maintain the lightroom catalog with lots of metadata, including using the facial recognition system for offline person recognition, the flag system enabling a 2-pass deletion review, star system for ranking photos by desirability, and add tags to everything. Go through each filter mechanism and develop a strategy for using each filter. Use the image gallery filter system to the limit. If you go through images by date & time captured you can usually bulk tag images much more quickly than if you have to switch between events.
  - kilroy123 5y ago
    I tried and used Lightroom for a while. I didn't want to keep paying Adobe so I tried to switch to darktable[0]. https://www.darktable.org https://www.darktable.org Maybe I'll give lightroom another try.

curt15 5y ago

>It's likely I messed up by not checking the "Enable data checksum for advanced data integrity" option when I created the shared folder. Still, this option is unchecked by default and it's not really clear that leaving it that way could lead to data corruption What's the point of using btrfs if Synology disables one of its signature features?

knolan 5y ago

While I use Photoshop CameraRAW to edit RAW files I don’t use Lightroom or any file library management software. Everything goes into folders on my PC on a dedicated drive which is shared so our Macs can access them and also backed up to BackBlaze. Any JPEGs that I want to share with family are then copied to iCloud photo album or posted to Flickr etc. I was using a WD myCloud that I retired when we moved house a couple of years ago as a secondary backup but that’s not a runner anymore. I’ve 32TB RAID6 volume on my personal office server and I’ll probably use that as a separate offsite backup. We do use Time Machine with a Time Capsule that I’ve recently replaced the disk in. I don’t really trust it and also pay for iCloud Drive for most of my other data. Work also provides Google Drive which I use for data that is shared with others. It’s all a bit of a mess but I think I’ve got some protection in place.

y04nn 5y ago
The (not last) version of Lightroom that I use store and organize all the pictures in an accessible directory. The only issue I had when I started using Lightroom was with the import method, I had not selected the "Copy" method and realized that after I deleted the originals. Now I always double check. I would also advise to use multiple Lightroom catalogs, Lightroom makes it easy. Also I don't think Lightroom is the source of the issue here. Anyway that's why it's important to always have at least 2 backup solutions with one being offline.

chromatin 5y ago

This (well, worse corruption than this) is one of my worst data nightmares. Losing pictures of my children, relatives who are now deceased, etc. I know it may sound like overkill, but I switched from a Synology NAS to ZFS for the checksumming specifically to avoid this fate. Not a knock on Synology specifically; I amazing all are about the same in this regard. What I wish is that there were a consumer friendly low power NAS running ZFS … (?) so I don’t have to maintain a server. (And a weekly sweep to AWS glacier to be sure)

account42 5y ago

In addition to backups, error correction data is a must for things you care about. With PAR2 you can add that to any type of file: https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive

gruez 5y ago
Bitrot is a thing, but mitigating it using par files seem suboptimal to me. The biggest problem is that you have to manually create the files (what if you forgot?) or write a script to do that for you (what if the script silently failed?). You're better off using RAID or a filesystem that has it built in (eg. ZFS or REFS).
- mnw21cam 5y ago
  It doesn't need to be manual. par2 is a command-line program, so it can be automated. Yes, ZFS is better for in-flight data, but it makes sense to create par2 files for your backup archive files, if you're then sending those files somewhere else (which of course you should).

notyourday 5y ago

I use lightroom and i have over 3TB of photos. This is my current workflow. There's no reason why it would not work on a Mac 1. Lightroom runs on Windows 10 system 2. I add new images onto it from one set of cards. The cards are pulled out and sat on a desk. A previous set of cards is added back to the camera and wiped. At this point there are images in the Lightroom and images on the the cards. 3. Every night, windows box rsyncs the Lightroom catalog and raw data onto a Linux box running in a RAID-1 4. Every night, after the rsync is completed Linux box triggers a backup of the rsync'ed catalog and files into S3 compatible remote bucket. This means that I have all images and catalog stored at least twice locally and once remotely. I have also validated that I can recover from a failure of Windows disk, Linux server and remote backup.

HerbsMan 5y ago

I have lost in the past one of my best music creations due to believing in iCloud. I had project, saved on my computer in iCloud directory - After reboot I didn't have the file locally nor in iCloud. Literally gone :)

devwastaken 5y ago

Apple wants you to buy a whole new MacBook for the extra storage, and only use their icloud for backup. Apple intentionally has bad implementations of network filesystems. They know it, but as per usual, they know their customers won't do anything about it.

mikehollinger 5y ago

Based on this, and on my own experiences, it looks like it's time to move on from Synology. But... to what? I've had a DS412+ since, well, 2012. I've dutifully replaced drives as they went bad, run disk scrubs, file system scrubs, and (in the last nine years) had to occasionally run time machine restores from it. It worked like a champ each time, including "restoring" from an old computer to a brand new one. However, in the last couple of years, we couldn't get my husband's machine to back up or restore successfully, and now out of our three macs, only one is able to successfully back up at this point, and (looking at this) I'm skeptical that it's actually working.

pembrook 5y ago
> it looks like it's time to move on from Synology I don’t think this is a Synology problem. No Windows users are reporting this issue. On the other hand, NAS devices have always been treated as second class citizens by Apple. If reliable network attached storage is important to you, maybe it’s time to move on from MacOS? As much as I hate Windows, using my NAS with it is a dream. It’s how the experience of network storage was supposed to feel.
eddieroger 5y ago
> The whole ordeal was largely of my own making That's not the conclusion the author came to - why is it the one you came to? I use a Synology with a Mac nearly every day. I don't use it for Time Machine, but that's just because I use different backup strategies, like storing the files directly on the Synology, or copying them over after local editing depending on the type of file. Also, like the author learned, I check my backups periodically to make sure the file is the same on the other side. > I'm skeptical that it's actually working You should test it then. If you've never tested you backups, you have none.
- mikehollinger 5y ago
  > You should test it then. If you've never tested you backups, you have none. I have, and so far we're one out of three. Mac A seems to restore (at least the last time we tried it) Mac B failed to restore Mac C fails to back up

parasubvert 5y ago

Samba (on Linux, not just Mac) has always had corruption issues for large files related to locks and filesystem caching quirks. Windows and the protocol itself has a different approach to locks than *NIX. Many have gotten more stable by disabling caching (though this hurts performance on low bandwidth / high latency fileshares). Also from what I recall, Apple SMB is based on a forked version of Samba from 10 years ago. They did this when Samba went GPLv3. Not clear how well they’re keeping up with quality improvements. My solution: use WebDAV on your Synology for transferring files from a Mac or Linux host. It has its own quirks but is far more reliable. Unfortunately this won’t work for time machine. But would work with “rsync -aXv —partial”.

sieve 5y ago

A mirrored ZFS setup with one offline backup and one cloud backup will prevent nonsense like this.[1] If you know what you are doing, you can do a lot of stuff with ZFS even on a single disk.[2] [1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ [2] Forbidden Arts of ZFS | Episode 2 | Using ZFS on a single drive [https://www.youtube.com/watch?v=-wlbvt9tM-Q https://www.youtube.com/watch?v=-wlbvt9tM-Q]

bluGill 5y ago
The problem is I get home at night and I want to spend time with my kids, not admin my ZFS system. Which is why I'm running a really old version of FreeNAS, it has owncloud (not nextCloud) which fails to start since the last time I updated freeNAS, and I haven't figured out how to make it work again - mostly for lack of time/interest in doing it.
- sieve 5y ago
  > not admin my ZFS system. Ubuntu/Manjaro comes with default ZFS support. There is no administration involved beyond the initial setup.
  - bluGill 5y ago
    There is still administration required. Sure ZFS itself is easy, but the rest of the ecosystem on top of that is hard. You need a lot more software than zfs to make something useful.
smarx007 5y ago
I am not sure. Quite possible bad data was written over SMB, not necessarily that the drives failed. As you see in the article, the damage to images is systemic and drives don't seem to be failing. More like Lightroom/macOS/macbook RAM/Samba wrote (accepted) bad data.
- sieve 5y ago

tannhaeuser 5y ago

Switched to using SD cards for Time Machine-like backup (using BackInTime) plus monthly archival on big disks on Linux for many years now; requires some organization due to space restrictions, though. Distinctly remembering when iPhoto (5?) wanted me to convert/store my private photos "in the database" rather than as plain files on my old PowerBook which I refused to do however, knowing all too well this would've been a bad idea.

cl3misch 5y ago
Aren't SD cards pretty limited in terms of write durability?
- jnieminen 5y ago
  There are high endurance SD cards, but eventually those will have also IO problems.

stewx 5y ago

I picked up a new DSLR and found out my copy of Lightroom 3 didn't support the new RAW format. Not willing to pay $125USD/year for the "latest and greatest" version of Lightroom, I found out I could use Adobe's free DNG conversion tool to just convert the fancy new format to DNG and continue using my bought-and-paid-for copy of Lightroom. Also, major plus: it supports lossy conversion, which churns out files ⅓ the size of the originals, with no perceptible loss in quality. I ended up converting my entire photo catalog, saving hundreds of GBs of disk space. The tool has a CLI as well. https://helpx.adobe.com/camera-raw/using/adobe-dng-converter.html https://helpx.adobe.com/camera-raw/using/adobe-dng-converter...

mnw21cam 5y ago
dcraw is a free tool that supports pretty-much every camera that has ever existed. You could try that too.
sillysaurusx 5y ago
From https://photographylife.com/dng-vs-raw https://photographylife.com/dng-vs-raw > DNG strips out most of the unrecognized meta data (such as Active D-Lighting, Picture Controls, Focus Point, etc) from RAW files, making it impossible to retrieve this data from DNG in the future. The last time I used a DSLR was when I was doing color research in 2012. At the time, raw format was the only thing that preserved all necessary information to make scientific observations. Most people don't need to care about such things. I just wanted to mention you're irretrievably throwing away metadata when you DNG-dong your pictures. It's probably a worthwhile trade in most cases though.
- matja 5y ago
  > At the time, raw format was the only thing that preserved all necessary information to make scientific observations. Which "RAW" format? I think that is the problem - there are many and few are completely documented, which is where DNG comes in (https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf https://www.adobe.com/content/dam/acom/en/products/photoshop...). Great point about throwing away metadata, and probably worthwhile to safeguard that one's flavor of RAW files (CR2/NEF/etc) can be reliably read in the future when the software necessary to read them inevitably disappears from the cloud.

rcarmo 5y ago

I haven’t had any issues with my Synology NAS yet, but I did have to stop using my iMac due to recurring _local_ data corruption errors: https://taoofmac.com/space/blog/2021/08/08/1600 https://taoofmac.com/space/blog/2021/08/08/1600 I wonder if the OP also has a Fusion drive - merely reading off a corrupt SSD cache or bad RAM would explain some of the damage.

shell0x 5y ago

I also have no idea how to organize photos and backups well. As an engineer, I'm good doing it for enterprise stuff, but it's really challenging for personal stuff. I don't trust Google products anymore, I got burned too many times when they killed their products. They work, but I just don't want to use them anymore. I currently have a 100GB OneDrive subscription for $3, so that works for now, but I'm pretty close to the storage limit. I'm not a Microsoft fan, but considering it's integrated with Windows now, I assume it will stick around for long. The same is probaly true with Apple and iCloud. I've relocated internationally a dozen times, so a NAS doesn't work for me. There are also other options for just storage like S3 or Azure storage, but the price seems to be almost the same as OneDrive.

unixhero 5y ago
+1 for Onedrive. I pay usd6 I think for 1 TB with a valid Office365 license. That's a pretty sweet deal. Not a microsoft fan either, however this works well. Tight integration with Linux CLI tools, Android Onedrive app is excellent et cetera.
- lostmsu 5y ago
  > Android OneDrive app is excellent If only they would actually sync files outside the app data dir like they do on desktop.
tilolebo 5y ago
Backblaze B2 is a really good option for remote backups. I use it to make daily incremental backups of my NAS, but you could replace the NAS by a USB drive.
stewx 5y ago
My online backup recommendation: Backblaze. No fuss, no muss. No limits.
vaughan 5y ago
> I've relocated internationally a dozen times, so a NAS doesn't work for me. Same here. I used to use 2x Seagate portable 5TB. Would mirror one to the other every week. Now I use a Western Digital MyBook 20TB in RAID 1, and backup this weekly to my portable externals using Carbon Copy Cloner for macOS (ideally I would have an identical second one for this). And run Backblaze for offsite backup. I avoided a NAS because I prefer the speed of USB3 versus network - super quick to view RAW and video, and for ease of relocation. I also have a 1TB portable SSD for my Apple Photos syncing iPhone content without filling up my MacBook disk. The WD MyBook's are a bit dodgy though, there are so many horror stories. They encrypt all data, so if your RAID config is corrupted...which is supposedly stored on the hard disk, then you are screwed. Also, when you tell it to sleep when inactive, whenever you open a file dialog, the laptop freezes for 5/10s while it wakes up. There is also a bug in the utility application that says "can't access drive" which makes you think your drive is broken but it's actually just a software bug. Not running the utility software fixes it. I also think it can cause some unable to sleep issues for macOS. Sometimes it is super loud too, and its quite noisy.

mwidell 5y ago

Don't complicate things. I did lots of research of different NAS solutions etc. but in the end I just got a huge external USB drive and then automatic backup with backblaze for $60 per year. Works great and super cheap, plus I have offsite backup in case of fire or theft.

ghaff 5y ago
And you can add one or two additional USB drives for redundancy. I suspect that a lot of people building NASs using btrfs configured as RAID whatever would be better with a few local USB drives plus Backblaze.
- horsawlarway 5y ago
  I did the whole "Save it to a large usb drive" method for backing up personal files for a while. I finally upgraded to a synology nas and I feel a little silly for not doing it earlier. It's far more convenient given I bounce between 3 machines during the day (personal laptop, work laptop, desktop), my wife can easily access files on her machines as well. Added bonuses are that it comes back up on its own after a power outage, doesn't require my desktop to be on for me to hit it externally or for a successful backup, and I don't have to remember where I last placed the usb drive. Basically - USB drives totally work, but the NAS is better in pretty much every way outside of price (and possibly some configuration, if the person isn't very technical).
  - mackrevinack 5y ago
    you can still make good use of the usb drive with hyperbackup on synology! i have a backup job set to run nightly but keep the drive disconnected (so something like a powersurge can't take out all the disks at once) and then once a month plug in the drive before i go to sleep then plug it back out the next day. the backup job gracefully fails for the rest of the month when the drive is not plugged in, but basically it just means i don't have to log into the synology just to run the backup job. i have a second drive with an identical backup job and the drive lives in my sister's house 364 days of the year. there's still the risk of losing a months worth of files but its a pretty low effort and cheap backup system apart from that

G4E 5y ago

Before buying an expensive software for data recovery, don't forget about Photorec[0] and its dangerously powerful brother TestDisk[1]. It will be able to help you more often than not, and it's FOSS. [0] https://www.cgsecurity.org/wiki/PhotoRec https://www.cgsecurity.org/wiki/PhotoRec [1] https://www.cgsecurity.org/wiki/TestDisk https://www.cgsecurity.org/wiki/TestDisk

pavel_lishin 5y ago
I don't think $79 is particularly expensive to recover half a terabyte of valuable photos, but thank you for the recommendation!
- sigg3 5y ago
  It's expensive if you ain't got it. I used testdisk as a student living on loans :)
  - iluvcommunism 5y ago
    I tried using test disk but I had an obscure situation. I was recovering from a failed REFS pool.
- smoldesu 5y ago
  It's considerably more expensive than $0, and just because $79 doesn't seem like a big issue in the heat of passion doesn't mean that you won't kick yourself for it later.
webmobdev 5y ago
Let me also chime in with a free and open source decent alternative to LightRoom - https://www.darktable.org/ https://www.darktable.org/ ...
squarefoot 5y ago
+1 for TestDisk. That gem literally saved my butt years ago when a controller failure on my old NAS almost destroyed all my data by turning all disks (Linux soft RAID1) into unreadable doorstoppers. And it wasn't like just a damaged partition table or anything like that; each file had to be recovered and copied elsewhere.

drittich 5y ago

I use Lightroom on Windows and also Logic Pro X on Mac, and both save to my Synology. This post is terrifying me, because I don't know an easy way to validate that my Logic Pro audio projects are not corrupted on the NAS. With a photo you can just look at it. With an audio project I would need to listen to the whole thing, and when there are 32 tracks or more to be checked, that can be unmanageable. For Lightroom, I keep the library on my local SSD drive and the photos on the NAS. One thing to note is that you can import photos into a Lightroom library in-place (from wherever they currently reside), or you can have Lightroom copy or move them to a location of your choice, with a few options around how to organize the files. In this case, the Lightroom software itself could well be the cause of the corruption, as it is doing the copying. I copy the files to a destination on the NAS using a drive-mapped letter, but it works with UNC paths as well. On Windows, I have ever experienced any corruption in about 8 years of using the Synology/

ghaff 5y ago

Something to be aware of the Lightroom catalog itself (which is unrelated to the OP's problem) is that it's an SQLite database. (Or it was when I ran into my problem about a decade ago bit I believe it still is.) I had hardware-related flakiness with a Windows system at the time. I swapped it out for an iMac but sometime later I discovered that there was some residual corruption of my Lightroom catalog even though I had restored a backup that seemed to be OK. Fortunately I was able to use some SQLite tools to largely fix the catalog. https://bitmason.blogspot.com/2013/01/recovering-corrupt-lightroom-catalogs.html https://bitmason.blogspot.com/2013/01/recovering-corrupt-lig...

patorjk 5y ago

Nikon RAW files store a full resolution jpg version of the photo inside of the RAW file (this may be true of other RAW formats too). I had a similar problem to what's described in the article a few years ago, and for photos that were damaged which I didn't have backups I was able to get the jpg out of the RAW by running this command: exiv2 -e p4 image_name.NEF Apparently this jpg is what the camera shows on its backscreen when you're previewing the image/RAW file.

aeharding 5y ago

My bet is the problem is related to the USB transfer. I've read about similar problems migrating data to Synology via USB. Don't remember where I read that, though...

ubermonkey 5y ago

Gaaaah. This is mostly not about Lightroom, or photos, or any of those specifics. This is about (a) a failure in a migration followed by (b) *failure of the user to verify the migration before deleting the original copies*: >To make matters worse, I deleted the photos from the USB hard drive, without verifying the new catalog. That line is, not coincidentally, where I stopped reading.

selfhoster11 5y ago
It's not obvious to amateur users that these safety measures must be in place for data migration projects. Members of my family only learnt after the first data loss.

rrauenza 5y ago

Anyone using lightroom should also consider configuring it to always drop the XMP sidecar file with your raw files. If the DB is ever corrupted beyond repair, you'll at least have a record of your current edits per file. https://fstoppers.com/post-production/most-important-setting-lightroom-set-default-8366 https://fstoppers.com/post-production/most-important-setting...

iluvcommunism 5y ago

I had a similar experience. Lost close to 9 yrs of photos and my wedding photos. I eventually got them back via a similar method as author. It took me a year to go through them all. After that experience I fully went to the cloud. The cause was a REFS mirror deciding to “mirror” my corrupt drive to my working drive. Great job MS. I also stopped using Microsoft for my raid stuff.

lostmsu 5y ago
> REFS mirror deciding to “mirror” my corrupt drive to my working drive REFS has an option for integrity checking, which (unfortunately) might be off by default.
- iluvcommunism 5y ago
  Yes I forget if it was on or not. You are correct. On the Windows OS (not server) you have to run a command to enable it. I will never forget the feeling I had watching the space on my good drive dropping precipitously as it was wiped away into “oblivion.”

jquaint 5y ago

Slightly off topic: I recently switched to Capture One for all of my photography needs. I would highly recommend it over Lightroom. Rather than subscribing to a service you can just buy it outright. It has all the features of Lightroom with some extras.

153957 5y ago
Similarly I recently switched the other way. I recommend Lightroom over Capture One. The subscription is a slight downside, but Capture One is similarly expensive if you want to keep up with the latest version, I know you don't have to, but if you do it is expensive. Moreover Capture One is missing some features which Lightroom has: The ability to filter collections based on their name. A shortcut for 'increase/decrease rating'. Filter photos for 'x stars and higher'. Show all images in a folder, including those in subalbums/subfolders. Being able to mark photos with a 'Reject' flag/rating (and easily hiding, not removing, those).
jslpc 5y ago
I tried switching to Capture One as well, and enjoyed using it for the most part, but I couldn't fully switch for one primary reason. I'm still fairly new to photography and use a few preset packs that I've paid for as a base before making modifications to them to my liking. I downloaded a program that converts Lightroom presets to LUTs, and those worked decently, but they were visibly inferior due to the fact that the converter cannot handle camera/lens profiles. I ended up just going back to Lightroom Classic for now, but hopefully once I progress enough to make my own edits from scratch I'll be able to switch back to Capture One.

donut2d 5y ago

Last year all of my photos and presets had disappeared after updating the Lightroom app. That was 2+ years of edits that are just gone, lost, unrecoverable. I do photography as a hobby so I never saw a need for backing up photos and I never paid for the subscription (which would include cloud storage) because I didn’t use any of the tools that came along with the subscription.

Saris 5y ago

This sounds like it doesn't have anything to do with Lightroom, but is instead file corruption caused by some kind of bug. Lightroom catalogs also don't contain any images, I think a lot of people are confused and think it does.

ksec 5y ago

>Lessons Learned The whole ordeal was largely of my own making. I never anticipated that the simple act of copying files from one place to another could go so horribly wrong. It isn't user's own making or user's fault at all! ( A voice is literally screaming this inside my head ) Tech shouldn't be like this. The DS218 support BTRFS which prevent certain file corruption but this is not what's happening either. I have been saying ( or ranting ) about this for now more than a decade. [1] And every time the answer was there are no market for it. Or consumer or market are not willing to pay premium for high quality NAS. The Kobol NAS was my hope that a high quality, reasonably priced, and somewhat reasonably easy to setup and managed NAS would be. But they are closed down as well [2]. I really hope Kobol release all of the work as open source including hardware design. May be someone ( or me ) could crowdfund it once the chip shortage is over. [1] https://news.ycombinator.com/item?id=10886702 https://news.ycombinator.com/item?id=10886702 [2] https://news.ycombinator.com/item?id=28299573 https://news.ycombinator.com/item?id=28299573

mleonhard 5y ago

I am writing a Rust CLI program for managing my backups: https://github.com/mleonhard/deduposaur https://github.com/mleonhard/deduposaur I tend to end up with multiple copies of photos and other docs: SD card, laptop, backup drives, and USB thumb drives. Deduposaur helps me to move files into my latest backup drive. It flags duplicates, modified files, and files that I previously deleted. It also checks each file's SHA-256 digest to detect corruption. The program is usable but needs some usability improvements and more tests. The license is Apache 2.0. I am happy to receive problem reports, feature requests, and pull requests.