6 ms·

So I'm actually setting up a private cloud for my company right now. It's a company of 1, so the requirements are quite extensive. :-P I'm building a FreeNAS s

So I'm actually setting up a private cloud for my company right now. It's a company of 1, so the requirements are quite extensive. :-P I'm building a FreeNAS server with 6x3TB hard drives in a RAID-Z2 config. My goal is to allow my Mac to use it for Time Machine backups, but to also use AeroFS as my file sync mechanism when I'm both in the office and on the road. Hopefully it works out smoothly. I'll have to figure out how to access the machine from behind my router, and I'll have to determine how to get it to automatically back up to S3 + Glacier. I think there's going to be a lot of details that I'll have to research here.

toomuchtodo 13y ago
How To Backup FreeNAS to S3: http://www.inetpub.dk/index.php/widgetkit/just-going-through-the-motions/60-remote-synchronization-from-freenas-to-amazon-s3 http://www.inetpub.dk/index.php/widgetkit/just-going-through...
- bifrost 13y ago
  Unfortunately it leaves out encrypting the backup. I think there is a tarsnap plugin for FreeNAS which does encrypt the data...
pedrocr 13y ago
>My goal is to allow my Mac to use it for Time Machine backups Note that this is completely unreliable. Time Machine isn't built to work over the network. The network support is badly hacked together and will destroy your backups on network faults. They basically treat the network volume as a block device that gets corrupted when the connection fails. Using it over wifi makes this painfully obvious.
- deleted 13y ago
  [deleted]
  - pedrocr 13y ago
    I'm positive about Time Machine over AFP. iSCSI was indeed pointed out as a potential avenue with less issues, but if I recall didn't really fix it completely. If you're running wired+iSCSI you're probably much less prone to this. Try and yank some network cables during backup and see how it handles it. I ran into more issues than most because the particular AP I have at home doesn't work well with the macbook air I was trying to backup and resulted in a lot of disconnects. So it was a bit of a stress test but the issue is real and well documented. You can find instructions all over the web on how to fsck sparsebundles to try and get back your backups[1]. This is particularly annoying because Time Machine is supposed to be a user friendly version of versioned backups like you can do with rsync+hardlinks. I have a script to do that for me on my personal linux machine and would love to be able to deploy TimeMachine+AFP for the rest of the family. Unfortunately apple decided it really needed to have hard-linked directories for just this case and so decided that the way to do that over the network was to have a single binary blob with a HFS image on the server instead of the proper files in the filesystem (filesystems don't usually support hard linked directories, HFS was hacked to do it). That way you not only lose direct access to the individual files on the server you also lose the whole backups when there's disk corruption... I should probably have a look again to see if iSCSI is a viable solution to this. [1] http://www.garth.org/archives/2011,08,27,169,fix-time-machine-sparsebundle-nas-based-backup-errors.html http://www.garth.org/archives/2011,08,27,169,fix-time-machin...
    - kika 13y ago
      I can confirm that. I've lost a TM backup recently made over the (wired) network. The cable failed, backup in progress hung, TM volume remained locked until I reboot, I rebooted and sparsebundle got screwed up. TM started it all over again.
      - pedrocr 13y ago
        Check the link in my post. If you still have the sparsebundle you can probably recover it with an fsck.
nostromo 13y ago
Why not just use BTSync for syncing? I use it for this purpose and it's fantastic. For backup you can use one of many solutions to archive to S3. I wouldn't recommend Time Machine.
- HorizonXP 13y ago
  I do like BTSync and considered it, but I'm not the biggest fan of the idea that my data is stored on everyone's computer. AeroFS allows me some semblance of control, and as I grow, it will allow me to add users easily. I won't know until I try it though, I could just as easily change my mind and use BTSync.
  - kika 13y ago
    BTSync could be installed everywhere, but it only syncs between the folders with the same key. So if only your computers have your key(s) then only they would have contents of your folder(s).
  - ketralnis 13y ago
    It doesn't put your data on everyone's computer. The data goes onto the computers that are sharing the folder. If you have two computers, the data is on two computers.
- mindslight 13y ago
  Why not use Unison? It only does pairwise synchronization, but is Free software and extremely mature. (sigh, it seems the tech community has become blind to anything that isn't heavily blogvertised.)
  - chrisrogers 13y ago
    Unison requires a bit more administrative work to set up and maintain than BTSync, in my experience. People really like the Dropbox experience.
  - porker 13y ago
    > (sigh, it seems the tech community has become blind to anything that isn't heavily blogvertised.) There's so much released and so much noise, it's hard to keep track of all the good things that come out - especially if they're not mentioned within their tribe of friends/followed ppl. I don't know how to get round that - the web is huge and no one can keep track (or have a network that touches) all of the open source releases.
barrkel 13y ago
Check out the recovery probabilities of raidz2 vs raid10. You're winning ~17% more space at the cost of a substantial performance hit, and almost no difference in reliability, depending on how you estimate HDD failure rates. Raidz2 is not fast when everything is working well, and if any of your drives develop a speed problem, the whole array will be reduced to that speed. I have a 11x4T raidz2, and will be migrating to 3x4x4T raid10. My last array was 2x4x1.5T raidz1, 3 drive failures in its 5 year lifetime, but not concurrent so no data lost. I use CrashPlan to get data from one place to another; I back up NAS to CrashPlan, PCs to CrashPlan, and PCs to NAS. If you don't use CrashPlan's app to back up to their cloud, it doesn't stop you from using it to back up to other machines you own. And it works on Windows, Mac, Linux, Solaris, and with a bit of hacking, FreeBSD.
- bestham 13y ago
  While you might be right about the small space advantage, I would feel more safe if the position of the failing drive (and the next failing drive) does not matter at all. If a drive develops a speed problem, I'll replace it.
  - barrkel 13y ago
    You might feel more safe, but you're not as safe as you feel. Suppose that once one drive has failed, the chance of another drive failing (at least an URE, but potentially worse) while reading it entirely (as necessary during a scrub / resilver) is 10%. I don't think this is very implausible, as RAID HDD failures are often correlated, and not only because it's inconvenient to distribute HDD purchases. The chance of successfully recovering in a mirrored scenario is 90%. You only need to read one drive. In raidz1 with 6 drives, your probability is 0.9^5 - you need to get lucky 5 times, because you need to read every other drive fully to recover. That brings you down to less than 60%. Raidz1 with 5 drives is 0.9^4 - a bit over 65%. Raidz2 for 6 drives combines these two stats. The probability of one failing disk while reading is 1-0.9^5. But you can recover from that. And the probability of that recovery is 0.9^4. So your consecutive probability of failures is (1-0.9^5)(1-0.9^4). Subtract that from 1 and you get your probability of getting lucky. Work out the numbers, and you'll find it's only 86%. It's less than mirroring. There are a couple of simplifications in the maths above (don't forget, a second drive failure means another drive to resilver, and a restart to the procedure), and a big assumption based on reliability %. But frankly, unless you think HDD failure is already fairly rare - and it's not - mirroring is usually ahead on safety, and the lead only increases with number of disks in the array. I used to think like you do, that I'd feel more safe if a single disk failed, knowing I could still tolerate another failure. That's why my array is currently raidz2. But I worked out the numbers, and it changed my mind. And that's why my next array is going to be raid10. Not to mention that it's hard to get much more than 150M/sec for any realistic I/O with raidz2. I have over 1G/sec of potential HDD bandwidth. Unless I'm doing a scrub, I can't touch the potential performance. And of course all disks act like they're on the same spindle, there is zero random I/O advantage for raidz.

toomuchtodo 13y ago

How To Backup FreeNAS to S3: http://www.inetpub.dk/index.php/widgetkit/just-going-through-the-motions/60-remote-synchronization-from-freenas-to-amazon-s3 http://www.inetpub.dk/index.php/widgetkit/just-going-through...

bifrost 13y ago
Unfortunately it leaves out encrypting the backup. I think there is a tarsnap plugin for FreeNAS which does encrypt the data...

pedrocr 13y ago

>My goal is to allow my Mac to use it for Time Machine backups Note that this is completely unreliable. Time Machine isn't built to work over the network. The network support is badly hacked together and will destroy your backups on network faults. They basically treat the network volume as a block device that gets corrupted when the connection fails. Using it over wifi makes this painfully obvious.

deleted 13y ago
[deleted]
- pedrocr 13y ago
  I'm positive about Time Machine over AFP. iSCSI was indeed pointed out as a potential avenue with less issues, but if I recall didn't really fix it completely. If you're running wired+iSCSI you're probably much less prone to this. Try and yank some network cables during backup and see how it handles it. I ran into more issues than most because the particular AP I have at home doesn't work well with the macbook air I was trying to backup and resulted in a lot of disconnects. So it was a bit of a stress test but the issue is real and well documented. You can find instructions all over the web on how to fsck sparsebundles to try and get back your backups[1]. This is particularly annoying because Time Machine is supposed to be a user friendly version of versioned backups like you can do with rsync+hardlinks. I have a script to do that for me on my personal linux machine and would love to be able to deploy TimeMachine+AFP for the rest of the family. Unfortunately apple decided it really needed to have hard-linked directories for just this case and so decided that the way to do that over the network was to have a single binary blob with a HFS image on the server instead of the proper files in the filesystem (filesystems don't usually support hard linked directories, HFS was hacked to do it). That way you not only lose direct access to the individual files on the server you also lose the whole backups when there's disk corruption... I should probably have a look again to see if iSCSI is a viable solution to this. [1] http://www.garth.org/archives/2011,08,27,169,fix-time-machine-sparsebundle-nas-based-backup-errors.html http://www.garth.org/archives/2011,08,27,169,fix-time-machin...
  - kika 13y ago
    I can confirm that. I've lost a TM backup recently made over the (wired) network. The cable failed, backup in progress hung, TM volume remained locked until I reboot, I rebooted and sparsebundle got screwed up. TM started it all over again.
    - pedrocr 13y ago
      Check the link in my post. If you still have the sparsebundle you can probably recover it with an fsck.

nostromo 13y ago

Why not just use BTSync for syncing? I use it for this purpose and it's fantastic. For backup you can use one of many solutions to archive to S3. I wouldn't recommend Time Machine.

HorizonXP 13y ago
I do like BTSync and considered it, but I'm not the biggest fan of the idea that my data is stored on everyone's computer. AeroFS allows me some semblance of control, and as I grow, it will allow me to add users easily. I won't know until I try it though, I could just as easily change my mind and use BTSync.
- kika 13y ago
  BTSync could be installed everywhere, but it only syncs between the folders with the same key. So if only your computers have your key(s) then only they would have contents of your folder(s).
- ketralnis 13y ago
  It doesn't put your data on everyone's computer. The data goes onto the computers that are sharing the folder. If you have two computers, the data is on two computers.
mindslight 13y ago
Why not use Unison? It only does pairwise synchronization, but is Free software and extremely mature. (sigh, it seems the tech community has become blind to anything that isn't heavily blogvertised.)
- chrisrogers 13y ago
  Unison requires a bit more administrative work to set up and maintain than BTSync, in my experience. People really like the Dropbox experience.
- porker 13y ago
  > (sigh, it seems the tech community has become blind to anything that isn't heavily blogvertised.) There's so much released and so much noise, it's hard to keep track of all the good things that come out - especially if they're not mentioned within their tribe of friends/followed ppl. I don't know how to get round that - the web is huge and no one can keep track (or have a network that touches) all of the open source releases.

barrkel 13y ago

Check out the recovery probabilities of raidz2 vs raid10. You're winning ~17% more space at the cost of a substantial performance hit, and almost no difference in reliability, depending on how you estimate HDD failure rates. Raidz2 is not fast when everything is working well, and if any of your drives develop a speed problem, the whole array will be reduced to that speed. I have a 11x4T raidz2, and will be migrating to 3x4x4T raid10. My last array was 2x4x1.5T raidz1, 3 drive failures in its 5 year lifetime, but not concurrent so no data lost. I use CrashPlan to get data from one place to another; I back up NAS to CrashPlan, PCs to CrashPlan, and PCs to NAS. If you don't use CrashPlan's app to back up to their cloud, it doesn't stop you from using it to back up to other machines you own. And it works on Windows, Mac, Linux, Solaris, and with a bit of hacking, FreeBSD.

bestham 13y ago
While you might be right about the small space advantage, I would feel more safe if the position of the failing drive (and the next failing drive) does not matter at all. If a drive develops a speed problem, I'll replace it.
- barrkel 13y ago
  You might feel more safe, but you're not as safe as you feel. Suppose that once one drive has failed, the chance of another drive failing (at least an URE, but potentially worse) while reading it entirely (as necessary during a scrub / resilver) is 10%. I don't think this is very implausible, as RAID HDD failures are often correlated, and not only because it's inconvenient to distribute HDD purchases. The chance of successfully recovering in a mirrored scenario is 90%. You only need to read one drive. In raidz1 with 6 drives, your probability is 0.9^5 - you need to get lucky 5 times, because you need to read every other drive fully to recover. That brings you down to less than 60%. Raidz1 with 5 drives is 0.9^4 - a bit over 65%. Raidz2 for 6 drives combines these two stats. The probability of one failing disk while reading is 1-0.9^5. But you can recover from that. And the probability of that recovery is 0.9^4. So your consecutive probability of failures is (1-0.9^5)(1-0.9^4). Subtract that from 1 and you get your probability of getting lucky. Work out the numbers, and you'll find it's only 86%. It's less than mirroring. There are a couple of simplifications in the maths above (don't forget, a second drive failure means another drive to resilver, and a restart to the procedure), and a big assumption based on reliability %. But frankly, unless you think HDD failure is already fairly rare - and it's not - mirroring is usually ahead on safety, and the lead only increases with number of disks in the array. I used to think like you do, that I'd feel more safe if a single disk failed, knowing I could still tolerate another failure. That's why my array is currently raidz2. But I worked out the numbers, and it changed my mind. And that's why my next array is going to be raid10. Not to mention that it's hard to get much more than 150M/sec for any realistic I/O with raidz2. I have over 1G/sec of potential HDD bandwidth. Unless I'm doing a scrub, I can't touch the potential performance. And of course all disks act like they're on the same spindle, there is zero random I/O advantage for raidz.