5 ms·

RAID won't save you from bit rot, and after I added bit rot detection in PhotoStructure, I was surprised to see how many of my files on older spindles had succu

RAID won't save you from bit rot, and after I added bit rot detection in PhotoStructure, I was surprised to see how many of my files on older spindles had succumbed to minor corruption. Using a filesystem that supports data scrubbing doesn't require an "enterprise class" budget anymore. After fielding this question from several of my beta users, I wrote it up the why's and how's on my blog: https://photostructure.com/faq/how-do-i-safely-store-files/ https://photostructure.com/faq/how-do-i-safely-store-files/

skoskie 6y ago
You are Mr. Photostructure? Cool. Thanks so much for making an awesome piece of software.
- mceachen 6y ago
  Thanks! BTW, I just released v0.8 last night: make sure you check it out, it's quite an upgrade.
ksec 6y ago
I do wish there are Consumer NAS that has all these built in and does it automatically. Btrfs is not available on All Synology Model. And I am waiting for QNAP's ZFS based OS. It is god damn annoying.
- pronoiac 6y ago
  Huh? My Synology NAS runs btrfs.
  - ksec 6y ago
    Sorry. I missed the word on All Synology Model.
ars 6y ago
> RAID won't save you from bit rot Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian. You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.
- mceachen 6y ago
  I'm sorry, I should have clarified: as far as I know, mdadm on RAID1 with consumer-grade spinning rust will not be able to save you from bit rot (or at least I haven't seen evidence to that effect). I haven't tried on RAID6 (where there's at least a chance due to the data duplication such that the flipped bit will be "out-voted" by the parity spindles).
murgindrag 6y ago
Yeah. I wish my file system did periodic integrity checks. I should switch to a more modern file system at some point. I'm still on ext4. Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? I haven't run into much bit rot yet, at least where it mattered, but I'm guessing I'd see more of it if I went up to bigger drives. For now, my data lives on 4tb. PhotoStructure sounds neat. Business model may be tough. I wouldn't run a proprietary solution for data management, especially from a small startup. I just wouldn't trust it. I want to know what happens to my data, and to be able to get it out. I understand the claim you'll open source in case of business failure, but there are many other paths (e.g. bought by Oracle, milked for profits, moved to incompetent management, etc). I would totally would pay for an open solution, if there were a good business model for that, but I'm not sure what that is. I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). I'd also consider paying for a split model, where the data management used open formats and auditable, documented open source, while there were UI/UX or search functionality on top of that. That may be what you're doing; I'm not sure. I can see some open source stuff, but I'm not sure what's open and what's closed. Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, have an automated way to move to offline storage once a quarter as you recommended, and have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Ideally, I'd also have some way to securely share with people
- mceachen 6y ago
  > Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work. > I want to know what happens to my data, and to be able to get it out. You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library. > ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed. > I can see some open source stuff, but I'm not sure what's open and what's closed. The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source. > Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting. > Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, borg, tarsnap, and BackBlaze b2 are good tools for this. > have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/ https://photostructure.com/faq/what-do-you-mean-by-dedupe/ > Ideally, I'd also have some way to securely share with people This is coming in a future version of PhotoStructure.
pjc50 6y ago
Aha! I've had almost exactly that project in my ideas list for years. It sounds like the Right Solution for home photos! I have subscribed to your newsletter.
- mceachen 6y ago
  Excellent! Expect an invite email today or tomorrow. Ping me (there's a chat button on the website) if it doesn't go through, I'll set you up.

skoskie 6y ago

You are Mr. Photostructure? Cool. Thanks so much for making an awesome piece of software.

mceachen 6y ago
Thanks! BTW, I just released v0.8 last night: make sure you check it out, it's quite an upgrade.

ksec 6y ago

I do wish there are Consumer NAS that has all these built in and does it automatically. Btrfs is not available on All Synology Model. And I am waiting for QNAP's ZFS based OS. It is god damn annoying.

pronoiac 6y ago
Huh? My Synology NAS runs btrfs.
- ksec 6y ago
  Sorry. I missed the word on All Synology Model.

ars 6y ago

> RAID won't save you from bit rot Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian. You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.

mceachen 6y ago
I'm sorry, I should have clarified: as far as I know, mdadm on RAID1 with consumer-grade spinning rust will not be able to save you from bit rot (or at least I haven't seen evidence to that effect). I haven't tried on RAID6 (where there's at least a chance due to the data duplication such that the flipped bit will be "out-voted" by the parity spindles).

murgindrag 6y ago

Yeah. I wish my file system did periodic integrity checks. I should switch to a more modern file system at some point. I'm still on ext4. Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? I haven't run into much bit rot yet, at least where it mattered, but I'm guessing I'd see more of it if I went up to bigger drives. For now, my data lives on 4tb. PhotoStructure sounds neat. Business model may be tough. I wouldn't run a proprietary solution for data management, especially from a small startup. I just wouldn't trust it. I want to know what happens to my data, and to be able to get it out. I understand the claim you'll open source in case of business failure, but there are many other paths (e.g. bought by Oracle, milked for profits, moved to incompetent management, etc). I would totally would pay for an open solution, if there were a good business model for that, but I'm not sure what that is. I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). I'd also consider paying for a split model, where the data management used open formats and auditable, documented open source, while there were UI/UX or search functionality on top of that. That may be what you're doing; I'm not sure. I can see some open source stuff, but I'm not sure what's open and what's closed. Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, have an automated way to move to offline storage once a quarter as you recommended, and have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Ideally, I'd also have some way to securely share with people

mceachen 6y ago
> Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work. > I want to know what happens to my data, and to be able to get it out. You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library. > ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed. > I can see some open source stuff, but I'm not sure what's open and what's closed. The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source. > Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting. > Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, borg, tarsnap, and BackBlaze b2 are good tools for this. > have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/ https://photostructure.com/faq/what-do-you-mean-by-dedupe/ > Ideally, I'd also have some way to securely share with people This is coming in a future version of PhotoStructure.

pjc50 6y ago

Aha! I've had almost exactly that project in my ideas list for years. It sounds like the Right Solution for home photos! I have subscribed to your newsletter.

mceachen 6y ago
Excellent! Expect an invite email today or tomorrow. Ping me (there's a chat button on the website) if it doesn't go through, I'll set you up.