5 ms·

Let me tell you an anecdote about Owncloud. Once upon a time, I hosted an Owncloud instance for my family. I had a raid 5 system setup on an old Xserve with aro

Let me tell you an anecdote about Owncloud. Once upon a time, I hosted an Owncloud instance for my family. I had a raid 5 system setup on an old Xserve with around 10 TB of usable space. I was really into learning about encryption at the time, so I figured it would be cool to encrypt all my families files on the server. I made sure that all my family members were properly educated about not forgetting their password, and then enabled the Encryption App[0] by following a tutorial. Everything went smoothly for about 6 months until I was updating Owncloud with a minor version upgrade, and then BAM; nobody (myself included) could decrypt any data on the server. I posted around about it, trying to figure it, ripped apart the source code to the point where I found that manually decrypting the data was resulting in gibberish. All my backups were the encrypted data, so those didn't really help either. I still don't know what happened to this day, but I've been eternally wary of personal fileservers ever since. [0] https://owncloud.org/blog/how-owncloud-uses-encryption-to-protect-your-data/ https://owncloud.org/blog/how-owncloud-uses-encryption-to-pr...

antx 10y ago
Well, sorry about your loss. Then again, please understand that NO, you didn't have a backup. What you had was a single copy (probably a simple rsync, am I right?)... that is worth nothing in case of corruption or voluntary mischief. Learn from your mistakes. Either use a solution that has a retention policy, or don't do backups at all.
- onli 10y ago
  That is not the lesson here. It is that a backup should not backup the encrypted data, but the unencrypted data, and have its own encryption. A simple rsync based bashscript can be fine for that, though I'd use borg.
  - jjnoakes 10y ago
    How do you decrypt the data if it was encrypted client-side by multiple keys?
    - onli 10y ago
      Every client would need to make its own backups, automatically. In such a situation it would be probably best to have two backups, those plus a backup of the encrypted data for the clients that fail to do the backups on their own. But you raise a good point: That's difficult to do.
  - baldfat 10y ago
    Having 2 local copies with local monthly backup for a year and an off site backup (CrashPlan) is the secret. CrashPlan can go back in time for issues like this. As a Systems Librarian I did the daily (31 days worth), weekly (52 weeks worth), monthly (48 months worth) back ups with 131 tapes was a pain but it always paid off. I just had three machines and needed to go to my tapes once a day.
    - onli 10y ago
      That sounds like a lot. Simple because it sounds also interesting: Why did you have to use your tapes each day? Unreliable filesystem?
      - the-dude 10y ago
        To move them offsite presumably.
      - baldfat 10y ago
        It was a library with 500,000 items being used all the time. I had Librarians who were adding and purging our collection everyday full time and students and faculty checking in and out books etc. all the times. Insider information all the Library Management Systems are totally broken and garbage (Open Source Evergreen is doing things better). I was required to reboot my Windows Server (Purchased before I was hired) everyday. I was told this was totally acceptable. The Oracle DB was solid but the rest of the stack was total and complete garbage that cost over $125,000 and service contracts of $30,000. Edit: Off Site aka IT Office Fireproof Case was my monthly and previous weeks. Once a week I walked them over. In the library in a fire case were the daily, and the last week. Tape was it was 2007.
- tomc1985 10y ago
  So, what, everyone is cool with owncloud pushing encryption-breaking updates? What the fuck is that about? The way apologists step up to the defense of stuff like this is appalling. The 'hard' stuff like this is supposed to be dealt with before the public release. Just because one has a backup doesn't mean that other programmers should force them to rely on it (whether they intended to or not... the point is that this kind of thing should never happen) I am beginning to feel like we should start figuratively burning people at the stake for advertently or inadvertently destroying other peoples' data -- it is a sin about as cardinal as they get; and it seems many programmers are clearly not very scared of such an outcome. Wiping/bricking/rendering inaccessible your customers' data should be causing people to shit bricks, not go on the defensive. Normals HATE technology bullshit like this. They just want it to work. Failures like these make the entire field look bad.
  - CaptSpify 10y ago
    I don't think I've seen anyone say owncloud is not responsible. Yes, they fucked up, and royally. But in the end, you can't trust them to not fuck up. You have to trust that they will make a mistake, so be prepared for it. Doing backups right is just really hard.
    - tomc1985 10y ago
      Maybe, but at the time of writing many of the responses here felt like the comment I originally replied to -- splitting hairs over pedantics and blaming the user for not testing his backups well enough. I still feel very strongly that he shouldn't have to
      - CaptSpify 10y ago
        > I still feel very strongly that he shouldn't have to As nice as that idea is, I don't think we have the tech yet to fully trust someone else with your data. You have to test it out that your backups work. Anyone who tells you otherwise is selling snake-oil at best.
    - mcherm 10y ago
mseebach 10y ago
I don't mean to belittle the severity of the bug you encountered -- but the root-cause of your problem was a lack of a backup. A large number of issues, some of them software bugs, many of them not, could lead to data loss. It that vein, too, your RAID-5 system is/was a disaster waiting to happen, especially when you don't have a backup: http://www.zdnet.com/article/raidfail-dont-use-raid-5-on-small-arrays/ http://www.zdnet.com/article/raidfail-dont-use-raid-5-on-sma...
- api 10y ago
  "You're holding it wrong." He did have a backup. Having it also be mysteriously borked certainly violates the principle of least surprise. That's a bug like Fukushima is an industrial accident.
  - mseebach 10y ago
    Backups are hard and you absolutely need to do them right, otherwise they're pointless. The reason you need backups is exactly that you can't trust other people (or yourself) not to epically fuck things up. No matter how unambiguously totally someone else's fault it is, if your data is gone, it's gone.
  - mnutt 10y ago
    If you want to rely on backups, you have to test them periodically and not overwrite older ones with newer ones. It sounds like the backups had probably never worked, or there was some necessary component that was not backed up.
    - Pxtl 10y ago
      Considering the number of problems we describe in backups 1) You need incremental backups, not just a live copy, so you can rollback. 2) You need redundancy both connected/disconnected and offsite. 3) You need to test restoring. All these requirements... how do we not expect the user to screw up? This is like crypto, except every idiot knows "don't roll your own, you'll screw that up and leave a hole". Same problem with OwnCloud itself - it requires too much configuration, as the OP explained. Why are we doing all this garbage manually? Where's the automatic backup solution that provides differential incremental backups so you can rollback to various points, integrates with Google/Dropbox/whatever, integrates with OwnCloud, lets you plug in an external HDD for regular disconnected backups, etc.
jwildeboer 10y ago
I think it is unfair to extend this specific experience to all personal fileservers. And TBH, I think that encrypting the storage shouldn't be done on this level. Setting up the OS that runs owncloud to have encrypted data volumes is the better way. If encryption is needed at all for taht. Transport encryption between the various devices and the fileserver is a must, however. But going beyond that is not something I would consider to be a good thing.
- antx 10y ago
  In-transit encryption is a no-brainer. The most secure (from a trust point of view, at least) method of encrypting file is obviously at rest, client-side. Seafile uses that method, but fails on the implementation on some aspects. "In-place" encryption versus "volume/disk" encryption both have major drawbacks and are hard to implement correctly. In case of "in-place", the private key is a possible vulnerability (like leaving the key under the doormat). In case of disk encryption, while it's mounted, it's worth nothing if (when) you are compromised.
deleted 10y ago
[deleted]
alexggordon 10y ago
Replying to my own comment, because I'm out of the edit time-range and this kind of blew up. If I had to point the discussion of this comment, I'd point it away from what I did wrong personally (I know I f*cked up in probably more than one way). What I learned from this is that there are no situations in which I would personally host a file server of this feature level and control over my documents. I think "all in one" (dropbox/backup/encryption/sharing/file storage) personal solutions like Owncloud try to do too much, and end up with some seriously rough edges that can turn off very large portions of their user base. I guess the question I raise is, given my experience "can you prove to me that I can absolutely trust you with all my data"?
- thomnottom 10y ago
  > I guess the question I raise is, given my experience "can you prove to me that I can absolutely trust you with all my data"? But the answer to that is "no". All of the time. No matter who you are asking it about. There is always the possibility of someone - no matter how much you trust them or pay them - screwing up your data. Including yourself.
  - notatoad 10y ago
    >There is always the possibility of someone - no matter how much you trust them or pay them - screwing up your data. Including yourself. Especially yourself. I trust a product with dedicated administrators who do nothing but keep people's data safe to do a much better job than i could ever do myself. I don't want that kind of responsibility when google and dropbox are willing to do it for me for a couple bucks a month.
    - alexggordon 10y ago
      Yeah, pretty much summarizes my fears with Owncloud. Not only that, but as mentioned before, I no longer have any desire to manage my own large scale storage (NAS/Raid, really anything), so that compounds my own fears with Owncloud.
- mahyarm 10y ago

antx 10y ago

Well, sorry about your loss. Then again, please understand that NO, you didn't have a backup. What you had was a single copy (probably a simple rsync, am I right?)... that is worth nothing in case of corruption or voluntary mischief. Learn from your mistakes. Either use a solution that has a retention policy, or don't do backups at all.

onli 10y ago
That is not the lesson here. It is that a backup should not backup the encrypted data, but the unencrypted data, and have its own encryption. A simple rsync based bashscript can be fine for that, though I'd use borg.
- jjnoakes 10y ago
  How do you decrypt the data if it was encrypted client-side by multiple keys?
  - onli 10y ago
    Every client would need to make its own backups, automatically. In such a situation it would be probably best to have two backups, those plus a backup of the encrypted data for the clients that fail to do the backups on their own. But you raise a good point: That's difficult to do.
- baldfat 10y ago
  Having 2 local copies with local monthly backup for a year and an off site backup (CrashPlan) is the secret. CrashPlan can go back in time for issues like this. As a Systems Librarian I did the daily (31 days worth), weekly (52 weeks worth), monthly (48 months worth) back ups with 131 tapes was a pain but it always paid off. I just had three machines and needed to go to my tapes once a day.
  - onli 10y ago
    That sounds like a lot. Simple because it sounds also interesting: Why did you have to use your tapes each day? Unreliable filesystem?
    - the-dude 10y ago
      To move them offsite presumably.
    - baldfat 10y ago
      It was a library with 500,000 items being used all the time. I had Librarians who were adding and purging our collection everyday full time and students and faculty checking in and out books etc. all the times. Insider information all the Library Management Systems are totally broken and garbage (Open Source Evergreen is doing things better). I was required to reboot my Windows Server (Purchased before I was hired) everyday. I was told this was totally acceptable. The Oracle DB was solid but the rest of the stack was total and complete garbage that cost over $125,000 and service contracts of $30,000. Edit: Off Site aka IT Office Fireproof Case was my monthly and previous weeks. Once a week I walked them over. In the library in a fire case were the daily, and the last week. Tape was it was 2007.
tomc1985 10y ago
So, what, everyone is cool with owncloud pushing encryption-breaking updates? What the fuck is that about? The way apologists step up to the defense of stuff like this is appalling. The 'hard' stuff like this is supposed to be dealt with before the public release. Just because one has a backup doesn't mean that other programmers should force them to rely on it (whether they intended to or not... the point is that this kind of thing should never happen) I am beginning to feel like we should start figuratively burning people at the stake for advertently or inadvertently destroying other peoples' data -- it is a sin about as cardinal as they get; and it seems many programmers are clearly not very scared of such an outcome. Wiping/bricking/rendering inaccessible your customers' data should be causing people to shit bricks, not go on the defensive. Normals HATE technology bullshit like this. They just want it to work. Failures like these make the entire field look bad.
- CaptSpify 10y ago
  I don't think I've seen anyone say owncloud is not responsible. Yes, they fucked up, and royally. But in the end, you can't trust them to not fuck up. You have to trust that they will make a mistake, so be prepared for it. Doing backups right is just really hard.
  - tomc1985 10y ago
    Maybe, but at the time of writing many of the responses here felt like the comment I originally replied to -- splitting hairs over pedantics and blaming the user for not testing his backups well enough. I still feel very strongly that he shouldn't have to
    - CaptSpify 10y ago
      > I still feel very strongly that he shouldn't have to As nice as that idea is, I don't think we have the tech yet to fully trust someone else with your data. You have to test it out that your backups work. Anyone who tells you otherwise is selling snake-oil at best.
  - mcherm 10y ago

mseebach 10y ago

I don't mean to belittle the severity of the bug you encountered -- but the root-cause of your problem was a lack of a backup. A large number of issues, some of them software bugs, many of them not, could lead to data loss. It that vein, too, your RAID-5 system is/was a disaster waiting to happen, especially when you don't have a backup: http://www.zdnet.com/article/raidfail-dont-use-raid-5-on-small-arrays/ http://www.zdnet.com/article/raidfail-dont-use-raid-5-on-sma...

api 10y ago
"You're holding it wrong." He did have a backup. Having it also be mysteriously borked certainly violates the principle of least surprise. That's a bug like Fukushima is an industrial accident.
- mseebach 10y ago
  Backups are hard and you absolutely need to do them right, otherwise they're pointless. The reason you need backups is exactly that you can't trust other people (or yourself) not to epically fuck things up. No matter how unambiguously totally someone else's fault it is, if your data is gone, it's gone.
- mnutt 10y ago
  If you want to rely on backups, you have to test them periodically and not overwrite older ones with newer ones. It sounds like the backups had probably never worked, or there was some necessary component that was not backed up.
  - Pxtl 10y ago
    Considering the number of problems we describe in backups 1) You need incremental backups, not just a live copy, so you can rollback. 2) You need redundancy both connected/disconnected and offsite. 3) You need to test restoring. All these requirements... how do we not expect the user to screw up? This is like crypto, except every idiot knows "don't roll your own, you'll screw that up and leave a hole". Same problem with OwnCloud itself - it requires too much configuration, as the OP explained. Why are we doing all this garbage manually? Where's the automatic backup solution that provides differential incremental backups so you can rollback to various points, integrates with Google/Dropbox/whatever, integrates with OwnCloud, lets you plug in an external HDD for regular disconnected backups, etc.

jwildeboer 10y ago

I think it is unfair to extend this specific experience to all personal fileservers. And TBH, I think that encrypting the storage shouldn't be done on this level. Setting up the OS that runs owncloud to have encrypted data volumes is the better way. If encryption is needed at all for taht. Transport encryption between the various devices and the fileserver is a must, however. But going beyond that is not something I would consider to be a good thing.

antx 10y ago
In-transit encryption is a no-brainer. The most secure (from a trust point of view, at least) method of encrypting file is obviously at rest, client-side. Seafile uses that method, but fails on the implementation on some aspects. "In-place" encryption versus "volume/disk" encryption both have major drawbacks and are hard to implement correctly. In case of "in-place", the private key is a possible vulnerability (like leaving the key under the doormat). In case of disk encryption, while it's mounted, it's worth nothing if (when) you are compromised.

deleted 10y ago

[deleted]

alexggordon 10y ago

Replying to my own comment, because I'm out of the edit time-range and this kind of blew up. If I had to point the discussion of this comment, I'd point it away from what I did wrong personally (I know I f*cked up in probably more than one way). What I learned from this is that there are no situations in which I would personally host a file server of this feature level and control over my documents. I think "all in one" (dropbox/backup/encryption/sharing/file storage) personal solutions like Owncloud try to do too much, and end up with some seriously rough edges that can turn off very large portions of their user base. I guess the question I raise is, given my experience "can you prove to me that I can absolutely trust you with all my data"?

thomnottom 10y ago
> I guess the question I raise is, given my experience "can you prove to me that I can absolutely trust you with all my data"? But the answer to that is "no". All of the time. No matter who you are asking it about. There is always the possibility of someone - no matter how much you trust them or pay them - screwing up your data. Including yourself.
- notatoad 10y ago
  >There is always the possibility of someone - no matter how much you trust them or pay them - screwing up your data. Including yourself. Especially yourself. I trust a product with dedicated administrators who do nothing but keep people's data safe to do a much better job than i could ever do myself. I don't want that kind of responsibility when google and dropbox are willing to do it for me for a couple bucks a month.
  - alexggordon 10y ago
    Yeah, pretty much summarizes my fears with Owncloud. Not only that, but as mentioned before, I no longer have any desire to manage my own large scale storage (NAS/Raid, really anything), so that compounds my own fears with Owncloud.
mahyarm 10y ago