3 ms·

Yeah. I wish my file system did periodic integrity checks. I should switch to a more modern file system at some point. I'm still on ext4. Your blog post suggest

Yeah. I wish my file system did periodic integrity checks. I should switch to a more modern file system at some point. I'm still on ext4. Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? I haven't run into much bit rot yet, at least where it mattered, but I'm guessing I'd see more of it if I went up to bigger drives. For now, my data lives on 4tb. PhotoStructure sounds neat. Business model may be tough. I wouldn't run a proprietary solution for data management, especially from a small startup. I just wouldn't trust it. I want to know what happens to my data, and to be able to get it out. I understand the claim you'll open source in case of business failure, but there are many other paths (e.g. bought by Oracle, milked for profits, moved to incompetent management, etc). I would totally would pay for an open solution, if there were a good business model for that, but I'm not sure what that is. I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). I'd also consider paying for a split model, where the data management used open formats and auditable, documented open source, while there were UI/UX or search functionality on top of that. That may be what you're doing; I'm not sure. I can see some open source stuff, but I'm not sure what's open and what's closed. Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, have an automated way to move to offline storage once a quarter as you recommended, and have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Ideally, I'd also have some way to securely share with people

mceachen 6y ago
> Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work. > I want to know what happens to my data, and to be able to get it out. You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library. > ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed. > I can see some open source stuff, but I'm not sure what's open and what's closed. The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source. > Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting. > Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, borg, tarsnap, and BackBlaze b2 are good tools for this. > have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/ https://photostructure.com/faq/what-do-you-mean-by-dedupe/ > Ideally, I'd also have some way to securely share with people This is coming in a future version of PhotoStructure.
- murgindrag 6y ago
  > > Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? > All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work. Tech friend (yes you!): Which one do you recommend? I mostly care about avoiding headaches (so reliability, stability, and ease-of-use). > You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library. I got that, but my goal is to avoid headaches. If your business goes under, I don't want to lose a couple weekends to data recovery. I think open source APIs to get at my data would be the level which would give me confidence. If I have access to the same APIs as your UI/UX uses to get at the data, I'd be okay. > That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed. I wouldn't look at other failures as ways to discredit an idea. Most failures are due to poor execution. That said, hardware business models are very different from software, and if it's outside of your comfort zone, you'll probably fail to execute too. But my point was: 1) There's a definite business proposition in providing a complete solution to the problem you articulated with very close to the technology you're building. You seem to be on the right track. 2) I'm not sure you've stumbled on the right business model yet. I want a solution which minimizes headaches, both current and future potential. I'm glad to pay for that. > I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting. Well, in part. I think the other pieces are: 1) Reliability. Google can and does shut down accounts regularly. It loses data too. I'm a statistic. 2) Google downscales images and doesn't handle ones from my big cameras very well (it's great for cell phone picks). 3) Storage. I can buy many TB of disk space. Many TB in the cloud is still expensive. 4) Trust. If you build a tool which (a) lines up incentives right (I pay you) (b) has the right legal code (my data is mine, and unless you receive a subpoena, stays mine) (c) has the right technology so I only trust you when I absolutely have to (I can share files with end-to-end encryption, or openly, depending on what I'm doing) 5) Integration. I'd like RAW files and similar to live locally. All I want in the cloud are pics for showing people on my phone, and nice search tools. I want metadata synced, though. What you're building is a lot more useful than anything else out there, by design, I think. > borg, tarsnap, and BackBlaze b2 are good tools for this. Thank you. A few points, though: 1) These do backups. I want synced file management WITH backups, not just backups. If I edit a Word file on my Mac, copy it to my Windows box, edit it again, and look 6 months later, I have a mess. If I then get rid of both machines, and buy a Linux box and copy both files over unto a RAID, I have a mess with duplicates. 2) Tarsnap is $0.25/GB/mo, so $250/TB/month. a 4TB backup would be $12k per year. Just about the only thing I feel I have well-organized is my code, which lives in git. And a lot of my documents, I author in Markdown or LaTeX, and those live in git too.
fomine3 6y ago
XFS don't support checksum for data but only for metadata. dm-integrity could be used with XFS. https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-detect-it-rhel https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-de...
paulie_a 6y ago
If you want data integrity I would avoid butterfs at all costs. It deals with it, by not having any.

mceachen 6y ago

> Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work. > I want to know what happens to my data, and to be able to get it out. You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library. > ... I'd pay for integration (a piece of hardware arrives on my doorstep with no futzing, and with automatic security updates, etc. and your competitive edge is trust). That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed. > I can see some open source stuff, but I'm not sure what's open and what's closed. The metadata parsing code (exiftool-vendored) and process management (batch-cluster) is open-source. The metadata inference and cleaning, image duplication detection and comparison, and UI are closed-source. > Or better yet, where there's a cloud model. My data lives at home, but you have low-res versions of my photos which automatically sync to my phone, which can be shared, etc. securely. I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting. > Mostly, what I want is a place to reliably be able to dump my files (not just photos), have versioned backups of everything, borg, tarsnap, and BackBlaze b2 are good tools for this. > have some way to manage !@#%^ duplicates (I have several computers, and that makes for all sorts of messes). Simple SHA collisions are easy, and there are many command-line tools that accomplish this. PhotoStructure does detect SHA-duplicate images, but also downsampled, RAW+JPG, and other variations: https://photostructure.com/faq/what-do-you-mean-by-dedupe/ https://photostructure.com/faq/what-do-you-mean-by-dedupe/ > Ideally, I'd also have some way to securely share with people This is coming in a future version of PhotoStructure.

murgindrag 6y ago
> > Your blog post suggest btrfs, ZFS or XFS. Which one should I pick next time around? > All three are available to recent linux distributions, but know they all have very different ergonomics. Spend a minute or two looking at a getting-started guide for each, and pick (or if you have a tech friend that might help you, pick whatever they recommend). All of them work. Tech friend (yes you!): Which one do you recommend? I mostly care about avoiding headaches (so reliability, stability, and ease-of-use). > You may have missed this, but with PhotoStructure, none of your data leaves your computer: it's all sitting in plain files and in a schema-documented SQLite database in your library. I got that, but my goal is to avoid headaches. If your business goes under, I don't want to lose a couple weekends to data recovery. I think open source APIs to get at my data would be the level which would give me confidence. If I have access to the same APIs as your UI/UX uses to get at the data, I'd be okay. > That's an interesting idea, but I think logistics would prove challenging. There are a couple startups that have tried this in recent years, and failed. I wouldn't look at other failures as ways to discredit an idea. Most failures are due to poor execution. That said, hardware business models are very different from software, and if it's outside of your comfort zone, you'll probably fail to execute too. But my point was: 1) There's a definite business proposition in providing a complete solution to the problem you articulated with very close to the technology you're building. You seem to be on the right track. 2) I'm not sure you've stumbled on the right business model yet. I want a solution which minimizes headaches, both current and future potential. I'm glad to pay for that. > I think there are already cloud solutions (like google photos) that provide this. I think PhotoStructure's value proposition is in privacy you only get via self-hosting. Well, in part. I think the other pieces are: 1) Reliability. Google can and does shut down accounts regularly. It loses data too. I'm a statistic. 2) Google downscales images and doesn't handle ones from my big cameras very well (it's great for cell phone picks). 3) Storage. I can buy many TB of disk space. Many TB in the cloud is still expensive. 4) Trust. If you build a tool which (a) lines up incentives right (I pay you) (b) has the right legal code (my data is mine, and unless you receive a subpoena, stays mine) (c) has the right technology so I only trust you when I absolutely have to (I can share files with end-to-end encryption, or openly, depending on what I'm doing) 5) Integration. I'd like RAW files and similar to live locally. All I want in the cloud are pics for showing people on my phone, and nice search tools. I want metadata synced, though. What you're building is a lot more useful than anything else out there, by design, I think. > borg, tarsnap, and BackBlaze b2 are good tools for this. Thank you. A few points, though: 1) These do backups. I want synced file management WITH backups, not just backups. If I edit a Word file on my Mac, copy it to my Windows box, edit it again, and look 6 months later, I have a mess. If I then get rid of both machines, and buy a Linux box and copy both files over unto a RAID, I have a mess with duplicates. 2) Tarsnap is $0.25/GB/mo, so $250/TB/month. a 4TB backup would be $12k per year. Just about the only thing I feel I have well-organized is my code, which lives in git. And a lot of my documents, I author in Markdown or LaTeX, and those live in git too.

fomine3 6y ago

XFS don't support checksum for data but only for metadata. dm-integrity could be used with XFS. https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-detect-it-rhel https://www.redhat.com/ja/blog/what-bit-rot-and-how-can-i-de...

paulie_a 6y ago

If you want data integrity I would avoid butterfs at all costs. It deals with it, by not having any.