3 ms·

The tool I ended up writing does essentially something very similar: - It uses a block size that matches the disk - It uses 50% of my RAM as a ring buffer -

by sly010 6y ago

The tool I ended up writing does essentially something very similar: - It uses a block size that matches the disk - It uses 50% of my RAM as a ring buffer - On start (and on buffer underrun) it waits for the buffer to fill before it starts writing to tape. - It adjust the tape write speed as an exponential function of the buffer fill level. I am proud of this last one. It is very simple and automatically converges to the best average speed to keep both the tape speed and the buffer level relatively stable. I have been considering making the reading multi threaded to make use of more disks, but that would be a bit of an overkill for my personal 2 disk NAS. edit:formatting

js2 6y ago
- How many TB are you backing up? - How frequently do you perform backups? - How often do you do fulls vs incremental? - What drive/autoloader are you using, and what was the total investment in drive + tapes? I keep considering a used LTO but I've only got 20 TB and it doesn't seem worth the trouble. I should just buy another 20 TB of drives and keep them offline when not backing up.
- foobiekr 6y ago
  I back up around 20TB and the easiest solution I found was to just buy 4x12TB drives and mirror to cloud (yielding 2x offline copies). For stuff that's really really important, around half of that, and which if restore is needed fast access is required, I also back it up to SSD. It's not the greatest solution, but when you're hovering at some small multiple of common single drives, it's certainly the easy answer.
- sly010 6y ago
  I only have ~1.5TB stuff that is really important which fits on a single LTO5 tape. (I said LTO6 before, I was mistaken) I archive every ~3 months, generally before I make bigger changes. I always do a full tar archive, it typically takes the whole day, definitely an event my family is aware of. I use a desktop HP Ultrium LTO5 drive. I fit on a single tape, but even if I didn't, changing tape twice a day would not be an big ask. Months of researching, ~$900 total on 2 used drives (first one was broken) + enclosure + SAS controller + 5 tapes. You need the enclosure with forced air cooling, you can't put the bare drive in a PC tower, it will overheat. I cannot imagine archiving 20TB with my cheapo setup, but I also cannot imagine spending ~3K for a newer drive, this is a sweet spot for me. After buying a broken drive first, I am now always nervous that this drive will quit on me. It is definitely a passion project at this point.
  - js2 6y ago
    Thanks for the info. For 1.5 TB I'd just use an extra drive. My 20 TB is 11 x 3 TB used SAS drives in a raidz3 config. You can pickup a 3 TB drive off eBay for $18-$24. I should've probably just bought four 10 TB WD red drives, but building this system has been, uh, a passion project I guess. :-) It's still cheaper than if I'd bought new red drives, but I haven't factored in the electricity cost of running 11 SAS drives vs four red drives... the breakeven is probably less than a year. :-(
axiolite 6y ago
Sounds like you reinvented mbuffer. http://www.maier-komor.de/mbuffer.html http://www.maier-komor.de/mbuffer.html Why not publish yours as well? For other file systems (not ZFS), spd_readdir.so can offer a good performance boost. https://kernel.googlesource.com/pub/scm/linux/kernel/git/djwong/e2fsprogs/+/748924621fdd0e70a191091915fc9724d42e4796/contrib/spd_readdir.c https://kernel.googlesource.com/pub/scm/linux/kernel/git/djw... It's probably possible to do the same on ZFS, but CoW certainly makes realizing this concept more difficult.

by sly010 6y ago

js2 6y ago

- How many TB are you backing up? - How frequently do you perform backups? - How often do you do fulls vs incremental? - What drive/autoloader are you using, and what was the total investment in drive + tapes? I keep considering a used LTO but I've only got 20 TB and it doesn't seem worth the trouble. I should just buy another 20 TB of drives and keep them offline when not backing up.

foobiekr 6y ago
I back up around 20TB and the easiest solution I found was to just buy 4x12TB drives and mirror to cloud (yielding 2x offline copies). For stuff that's really really important, around half of that, and which if restore is needed fast access is required, I also back it up to SSD. It's not the greatest solution, but when you're hovering at some small multiple of common single drives, it's certainly the easy answer.
sly010 6y ago
I only have ~1.5TB stuff that is really important which fits on a single LTO5 tape. (I said LTO6 before, I was mistaken) I archive every ~3 months, generally before I make bigger changes. I always do a full tar archive, it typically takes the whole day, definitely an event my family is aware of. I use a desktop HP Ultrium LTO5 drive. I fit on a single tape, but even if I didn't, changing tape twice a day would not be an big ask. Months of researching, ~$900 total on 2 used drives (first one was broken) + enclosure + SAS controller + 5 tapes. You need the enclosure with forced air cooling, you can't put the bare drive in a PC tower, it will overheat. I cannot imagine archiving 20TB with my cheapo setup, but I also cannot imagine spending ~3K for a newer drive, this is a sweet spot for me. After buying a broken drive first, I am now always nervous that this drive will quit on me. It is definitely a passion project at this point.
- js2 6y ago
  Thanks for the info. For 1.5 TB I'd just use an extra drive. My 20 TB is 11 x 3 TB used SAS drives in a raidz3 config. You can pickup a 3 TB drive off eBay for $18-$24. I should've probably just bought four 10 TB WD red drives, but building this system has been, uh, a passion project I guess. :-) It's still cheaper than if I'd bought new red drives, but I haven't factored in the electricity cost of running 11 SAS drives vs four red drives... the breakeven is probably less than a year. :-(

axiolite 6y ago

Sounds like you reinvented mbuffer. http://www.maier-komor.de/mbuffer.html http://www.maier-komor.de/mbuffer.html Why not publish yours as well? For other file systems (not ZFS), spd_readdir.so can offer a good performance boost. https://kernel.googlesource.com/pub/scm/linux/kernel/git/djwong/e2fsprogs/+/748924621fdd0e70a191091915fc9724d42e4796/contrib/spd_readdir.c https://kernel.googlesource.com/pub/scm/linux/kernel/git/djw... It's probably possible to do the same on ZFS, but CoW certainly makes realizing this concept more difficult.