7 ms·
Building NAS with ZFS, AFP/Samba for Time Machine (2021)
- riobard 4y agoHave recent versions of macOS fixed the reliability of Time Machine over network (especially over Wi-Fi)? It used to be that after a while (varies from a couple of weeks to months) Time Machine would complain that the spare disk image on the NAS was corrupted and you'd have to start from scratch, which completely defeats the purpose of backup.
- dividedbyzero 4y agoI've been using the same (Synology) share for years with not a single corruption (that I know of). I've set TimeMachine to only run at night though when the laptop tends to sit on my desk where it has ethernet.
- tailspin2019 4y agoInteresting, I have two Macs backing up via Time Machine to a Synology (over ethernet) and both have regularly suffered corruptions where they have to restart the backup from scratch again. It's infuriating. Were there any specific tweaks you made to your setup to make this reliable? I also wonder if the amount of backup data makes a difference. I have tended to use Time Machine to backup pretty much my whole drive. I wonder if it's more reliable when backing up smaller amounts of data perhaps...
- geerlingguy 4y agoI have it do my entire drive and frequently throw 100+ GB of video files onto my main hard drive that get backed up within the hour.
- dividedbyzero 4y agoJust the scheduling via TimeMachineEditor so it only does backups between 2AM and 8AM or so, mostly because the Synology is audible during backups and I find that annoying when I'm in the room. I don't backup over VPN or the like, just via the local LAN (if I'm not at home, no backup gets made). As to amount of data, I run it on my whole drive with a few excluded directories, that's 700-ish gb of non-excluded data and the backup on the Synology is 1.6TB. I've recently migrated the TimeMachine backup from an older Intel MBP to a new M2 Air (I wasn't even aware you could do that) and that seems to have worked just fine as well.
- riobard 4y agoI'm wondering… have you ever tried to restore from that TM backup?
- dividedbyzero 4y agoNot from that one. I've restored once from a TM backup on an external HDD (worked well), but from this one I've only had to restore individual files (always worked). It's kind of hard to test a restore of 700gb of data on a laptop with a 1tb SSD, and since this is my personal laptop it has tons and tons of documents, photos, files for projects, tons data in various apps ... I wouldn't know how to verify more than a small sample of it all.
- nicolas_t 4y agoI've had good experience with Synology and Time machine despite usually using wifi. I've seen corruption only once in the last 5 years across 4 different machines.
- _cenw 4y agoNo. TM is fast/reliable on local drives because of APFS magic. They don't seem to have touched the implementation that uses SMB in any meaningful way since the AirPort Time Capsule was discontinued.
- crest 4y agoHave your tried TimeMachine on sparse images on network file systems or iSCSI targets (or did Apple finally kill off all initiators for macOS?) in recent macOS versions?
- Asmod4n 4y agoCan’t do full system restores from iSCSI
- anecdotal1 4y agoif the iSCSI disk is its own physical disk you can just pull it from your NAS, plug into a USB dock, and plug it into your Mac
- Asmod4n 4y agoThat’s defeating the purpose why you want to use iSCSI. Just get a usb drive and back up to that then.
- jiveturkey 4y agoNot at all. It's typically far more convenient to use a network available drive for backup. You rarely need to restore and even rarer to do a full system restore.
- xoa 4y ago>or did Apple finally kill off all initiators for macOS I would assume that'd work fine, because I run my main workstations and a server full time with my home folder and various data living off APFS or HFS formatted iSCSI targets served over 10 gigabit from a TrueNAS system. I haven't upgraded to macOS 13 yet since I tend to be a lot more conservative these days about Mac upgrades than I used to, but Xtend SAN at least still works beautifully under macOS 12. I really, REALLY hope Apple doesn't remove that capability without a native alternative (or at least officially sanctioned way to do the same thing). Of course in that specific scenario Time Machine is of limited value (and I don't use it) since why bother when you can just use ZFS snapshots on the NAS side instead. TM has always been too flaky for me to feel comfortable with anyway. Still, could be powerful in some instances, particularly when combined with a hardware gadget setup. Which leads to a response to sibling thread: >Asmod4n 3 hours ago: Can’t do full system restores from iSCSI Linux has kernel support for USB gadgets that can emulate mass storage devices, and that means you can actually rig up a little cheap embedded board to translate iSCSI into something that will look to anything else like a standard external USB hard drive. See for example [0]. Like the author there I use it to work around issues with appliances like consoles or the like that don't know how to natively deal with a lot of network stuff, but it's a pretty powerful tool if you've got a good NAS and LAN setup regardless. Get your full network storage flexibility, snapshots, data integrity and so on that pretty much anything ever with a USB port can utilize natively. ---- 0: https://matt.olan.me/making-a-piscsi-usb-drive-part-1/ https://matt.olan.me/making-a-piscsi-usb-drive-part-1/
- savolai 4y agoI've experienced this too. My wifi isn't terribly reliable.
- xenadu02 4y agoI've been using FreeNAS/TrueNAS for a year or so with no issues using multiple Mac machines. It's entirely over SAMBA shares. I set the server up to give each user their own unique TM backup disk.
- wil421 4y agoWhat’s you pool setup for this? How much space do you set aside for each disk?
- xenadu02 4y agoI have it setup to share the space; I have a TM dataset (generic not AFP, Time Machine, or whatever). Then I have a SAMBA per-user Time Machine share on top of that. The first time the user connects from the Mac it creates a child dataset for that user with the relevant ACLs to keep them isolated. A quote forces each client to prune its history. The pool itself is 8x12TB Toshiba enterprise drives. I'll have to check what pool config I used but I did add multiple NVME SSDs for SLOG and ARC. Whichever one is the important one is actually mirrored across the NVME drives for redundancy while the other partition only exists on one NVME drive, which was mildly annoying to configure. I'm still not sure I made the "best" choices for the ZFS config - there are many knobs to turn.
- rtp4me 4y agoSame here. 3 Mac Laptops connected to my TrueNAS server (v13) over wifi. Each laptop has its own separate ZFS dataset. Works great. No issues thus far. That said, I have not tried to restore from TimeMachine yet. I need to try that this week.
- riobard 4y agoInteresting. Could you please share the Samba config for the Time Machine shares? And over what medium do your Mac machines connect to the NAS? Wired or wireless?
- xenadu02 4y agoI did not customize the Samba config, just accepted the defaults. Connections are a mix of wired and wireless. The NAS box has 10GbE to my UDMPro. See my other comment for how the datasets are setup. I forgot to mention that I applied a quota to force each client to prune its history to avoid consuming all disk space.
- jbverschoor 4y agoYeah it's been performant (wifi), but I use TimeMachineEditor, so it will only backup to my NAS once or twice per day. In terms of stability, I haven't had a problem with TM on a NAS for a few years, but yeah the corruption is why I used to use CCC instead.
- atmosx 4y agohm, what reliability issue? I'm using ZFS + netatalk in Gentoo server since 2015 as a backup server for 5 macOS devices without problems.
- bombcar 4y agoThis is basically my setup but isn’t samba + apple doodads and it’s been working fine for years. Early on I had an issue with over provisioning the ZFS slice or whatever but that was resolved.
- furyg3 4y agoI have been running Time Machine backups to a Synology (via SMB, exclusively over wifi except for the initial backup) for years and years. I ran into this issue only once, but it was after moving the backups around and changing the shares on the synology (so may have been my fault). So far I've found it to be pretty reliable.
- geerlingguy 4y agoDitto, just in my case using ASUSTOR. I think the reliability issues aren't too bad especially if you use in the built-in Time Machine support in most NAS OSes these days.
- aeturnum 4y agoI also use a Synology with Time Machine. It does seem pretty functional, but I regularly run into weirdness associated with it (some backups fail for unclear reasons, sometimes I can't ls my /Volumes directory while the TM backup is mounted, etc). It has the feel of an under-maintained system.
- netruk44 4y agoI've had this same issue with my Synology. Randomly Time Machine would stop doing backups because the destination was "corrupted". I've given up on Time Machine over SMB and I'm using Borg + Vorta for my backups. It has the benefit of being readable by any other Linux machine, too.
- memco 4y agoI too was bitten by Synology corrupted TM backups over SMB on WiFi and have stopped trying to back up since I have not found any good documentation on the cause or solution. Where do you store the backups? Did you change to a hardwired connection?
- netruk44 4y agoI haven't changed anything other than swapping out TM for Borg. The backups still reside on the Synology (Borg is available on SynoCommunity: https://synocommunity.com/package/borgbackup https://synocommunity.com/package/borgbackup), and I still backup over WiFi. The Vorta (Mac UI for Borg) client and user experience isn't Apple levels of polished like TM, but I've made it work and I'm happy enough with it. I'm not sure what causes TM to start considering the volume corrupted, because I believe at one point I switched over to a hardwired connection, but still saw a corrupted volume. I compared that to Borg, which doesn't flinch if you SIGKILL the process mid-backup, and will basically just pick up from where it left off if you start another one. I saw no reason to continue with TM at all.
- CharlesW 4y ago> Have recent versions of macOS fixed the reliability of Time Machine over network (especially over Wi-Fi)? FWIW, I've done Time Machine backups over Wi-Fi to a QNAP NAS for several years and have never run into this issue. I'm just using default settings for everything.
- n8henrie 4y agoNo. Mine just borked again last month. Happens once or twice per year. Thankfully this is not my first rodeo. My Time Machine over SMB on ZFS setup has zfs-auto-snapshot set up, so I was able to roll back to a snapshot a month or so prior, which Time Machine happily verified, and it resumed backups without a hitch (keeping several years worth of backups intact).
- pridkett 4y agoHasn’t happened to me recently, probably about 16 months since the last time it claimed everything needed to be reset. Fortunately, I found I can put the Time Machine archive on btrfs and use snapshotting to roll back a day when this happens. Works great. I also found increasing the band size to 50MB from the default of 8MB made things work “better”, but I lack concrete empirical evidence.
- vonseel 4y agoSo, I just did this recently. This is an ok article for beginners. I’d hesitate to even mention l2arc or slog devices to someone that isn’t an experienced technical user, though, as they will probably go buy the wrong drives and may not even need the added complexity. I’m not using them and I see high write and read speeds from a zpool with 2x2 mirrored vdevs (wd reds). If the hardware is remotely capable I’d probably advise running proxmox and spinning up a turnkey Linux fileserver container with bind mounts to the underlying zfs filesystem to use as the main samba share manager. This makes it easier for less technical people to manage the users and permissions and stuff with a web gui. I had some trouble getting zfs sambasharing to work with sharesmb=on (not sure where those configs are stored on proxmox, and if it can do everything samba normally can), but you can try that route to or use sharenfs to set up NFS shares. There are a bunch of options for ACLs in zfs you probably want turned on , acltype and aclmode, and xattr=sa. Also case sensitivity and the case normalization stuff. And atime=off. Does anyone ever actually use access time because I can’t think of any use case for that on my nas!? To go beyond chmod, you probably want to learn about setfacl and getfacl and set some sane ACLs so your folders and files are created with the correct permissions. If you are sharing stuff with other users, this becomes a problem real quick unless you’ve got the ACLs set right and things get created with 775/664 perms. I like to also set chmod g+s so the default group of anything is the group of the parent, not the current users primary group. Lastly, if you’re on zfs you can setup something like sanoid to automate snapshots. I just did that, hopefully it will save me next time I run rm -rf against the wrong folder by accident!
- vardump 4y agoI had really bad "find ." performance until setting up l2arc. Used a Samsung 980 1 TB NVMe for the purpose.
- denkmoon 4y agoBecause your metadata blocks are sitting in the l2arc, presumably. That's not guaranteed to be the case though. A more modern zfs setup can use that nvme flash as a dedicated device for metadata/"small" blocks, without evicting them at the whim of the caching algo. https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954 https://forum.level1techs.com/t/zfs-metadata-special-device-...
- eatbitseveryday 4y ago> An important note is that you CANNOT add extra disks to an existing VDEV. Not true for mirror vdev, but you don’t get the added capacity. You may add vdevs later, however. Also some good arguments are made for using mirror vdevs over RAID-Z [1]. The extra space afforded by a 1:3 RAIDZ compared to 1:2 mirror isn’t much. Given the performance gain over a mirror resilvering and reduced recovery time, I prefer mirroring. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
- Nursie 4y agoPretty sure this is no longer true for RAIDZ as well, though the space usage is non-optimal at first, until files are read/written enough (it fixes itself over time). No idea if this has filtered through to mainstream release yet, but it was announced almost 18 months ago. (Edit — AFAICT it is still in the pipeline but not yet quite released)
- eatbitseveryday 4y ago> No idea if this has filtered through to mainstream release yet, but it was announced almost 18 months ago. What was announced?
- Nursie 4y agoThat you will, when the PR is considered fully stable and good to go, be able to add drives to a RAIDZ vdev. Here we are - https://arstechnica.com/gadgets/2021/06/raidz-expansion-code-lands-in-openzfs-master/amp/ https://arstechnica.com/gadgets/2021/06/raidz-expansion-code...
- reportingsjr 4y agoWhen I built my home storage server I made a mistake, and thought I could fit an additional drive, but I ran out of SATA ports. I kept the extra drive as a cold swap and just set up a raidz array. I ended up freeing up that SATA port I needed, but I'm waiting just a bit longer to expand ZFS on to that last drive so this code can get juuusstt a bit more testing!
- jgalt212 4y agoI've heard advice against doing RAID as disks tend to fail at the same time/age. Has anyone tried to mix and match vendors to counter this spikiness in time to failure distributions?
- rtp4me 4y agoI think the preferred method is to buy the same drive type from different vendors. This should help avoid getting drives from the same batch from the factory.
- gclawes 4y agoIsn't AFP deprecated?
- hnbear 4y agoThat's addressed in the article, which recommends and explains SMB too. --- Using Samba instead of AFP (recommended in 2021) As pointed out by many people on Reddit that Apple has effectively deprecated AFP, I would like to add this section telling you how to use Samba instead of AFP for Time Machine and file sharing. ---
- hsbauauvhabzb 4y agoAre there any good zero toads expert guides or books? I understand the basics but feel like there’s so much beyond my current understanding. I want to know how to calculate expected performance, understand the process for disk monitoring, what to do in the event of a failure (my guess is to not power down the cluster in case disks don’t spin back up), etc.
- wil421 4y agoLook into TrueNAS (formally freenas). The community forums have a huge depth of knowledge. I’d caution you first, read a lot of their docs and understand the basics before asking questions. They come off as angry system admins. https://www.truenas.com/community/ https://www.truenas.com/community/
- ksec 4y agoTaking this opportunity to ask a slightly off topic question. Are there any Wireless Storage solution that doesn't use the "Network", i.e Behind a Router. If not, are there any NAS solution that only works for "Intranet" or a Single User only. Basically I want an external storage solution that doesn't require a cable. And offer ZFS / BTRFS error protection. And it has to be consumer friendly, i.e Plug and Play.
- rtp4me 4y agoMaybe something like an RPi4 with external storage and wifi? With a quick google search, I am sure you can find a bunch of SBC (single board computers) with wifi that can be used to host something like TrueNas or Debian + ZFS.
- wil421 4y agoYea this is all great until it stops working. I’ve had a FreeNAS build sitting for 3 years and one day I notice Time Machine says it hasn’t backed up. Time Machine refuses to use the disk and says the samba disk is incompatible (code 3 in the logs). It’s the same Samba share I’ve had working on 3 Macs for 3 years but one day my M1 had enough. Apple support is less than helpful.
- hrdwdmrbl 4y agoCan anyone comment on the reliability of a setup like that? I've used OpenMediaVault for a few years and I've had terrible network reliability. I believe that it's basically just a GUI on top of those packages, eh?
- rtp4me 4y agoBeen using a combination of TrueNAS and Debian+ZFS for a long time providing backup services for both production and non-production (personal) systems. As mentioned above, I am using TrueNAS v13 as a Time Machine repository server for 3 Mac laptops at home. Are you using WiFi or wired network connectivity using OMV? Feel to reach out via email if you need help (see my profile...).
- hrdwdmrbl 4y agoThe OMV is wired but my computer is wireless, though it’s within 2 meters of the router.
- aquova 4y agoSlightly off topic, but can someone explain what happens when you use multiple vdevs in the same zfs pool as shown in their diagram? Does any data in the zpool get written to both vdev1 and vdev2 in its entirety, part to each, or entirely to one or the other? It always seemed to me that it would be better to have two zpools with one vdev each rather than one zpool with two vdevs, but maybe there's something I don't understand.
- yabones 4y agoMultiple VDEVs in one pool = striping. To build a traditional RAID10 on ZFS, you'd create several mirror vdev's, then add them to the same pool. This writes one strip to vdev1, the next to vdev2, etc. Should also be noted that it doesn't rebalance, so adding a vdev to a pool won't re-distribute data throughout the pool; new data will concentrate on the new vdev and potentially degrade performance until they're roughly at the same fill level. (edit) Real world example: this is a NAS that has 3 pairs of mirrors. Writes are a little faster than a single disk since it can hit three at a time. Data correction is possible because every chunk of data has a mirror. And, since it's three mirrors I could potentially lose 1 disk in any vdev at the same time without data loss. % zpool status NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada2 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 da0 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0
- radiowave 4y agoAFAICT, the diagram is just there is illustrate the differences between the types of vdev. It doesn't represent a sensible configuration, nor is it what the rest of the article is talking about actually doing.
- kbumsik 4y agoWhy combining RAID 1 mirror and RAID 6 array in the same pool?
- sandreas 4y agoIn 2022 / 2023 you could also consider using Proxmox VE or TrueNAS Scale, where you get much more than a simple NAS. I've recently built a small Home-Server with an old Fujitsu D3417-B2 Board, 32GB ECC RAM and Xeon E3 1225v5, 2TB Samsung Pro and Fractal Define Mini C for about 400 Bucks total (used) and can now enjoy fully encrypted ZFS Proxmox with LXC containers and Device Passthrough running 9.4W Idle. I did not expect this to work that well, but I'm pretty happy with this config and using zfs-auto-snapshot from xshock[1] and OpenMediaVault LXC container from tteck[2] just for shares I don't think I would switch to a pure NAS again. You have to teach yourself some zfs command line kung fu, though. [1]: https://github.com/extremeshok/xshok-proxmox https://github.com/extremeshok/xshok-proxmox [2]: https://github.com/tteck/Proxmox https://github.com/tteck/Proxmox
- mustache_kimono 4y ago> I'm pretty happy with this config and using zfs-auto-snapshot from xshock zfs-auto-snapshot is cool, but it's all in how you use those snapshots? httm [0]. [0]: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
- sandreas 4y agoI use the Samba Share option for file based history of zfs and the .zfs directory option to have access to the files. But httm looks cool, I'll definetely take a look, thank you very much.
- abrookewood 4y agoThis isn't strictly correct "The only way to expand your ZFS pool is by adding a new VDEV into it" and it is the reason I primarily run RAID 1 mirrors. If you run out of space, you swap out the disks one at a time, wait for the drives to sync and then expand the pool.