5 ms·

Not that weird, but I've thought about RAIDing some USB thumb drives together. Might actually be a good way to get practice with your choice of RAID software fo

Not that weird, but I've thought about RAIDing some USB thumb drives together. Might actually be a good way to get practice with your choice of RAID software for cheap, come to think of it.

buildbot 3y ago
I recall reading a great intro to ZFS article that did this! They had like 16 USB drives!
hsbauauvhabzb 3y ago
Virtual machines with many virtual disks would be much easier, or maybe I’m just lazy…
- _joel 3y ago
  loopback devices would be even easier... but not as fun
dspillett 3y ago
> but I've thought about RAIDing some USB thumb drives together I had a netbook with an irritatingly slow (and a bit small) eMMC drive for its main storage. I had a pair of microSD cards lying around that were faster for many IO patterns, so I bought a third and strung the three together in mini-USB readers (like https://www.amazon.co.uk/dp/B001JPPQS6/ https://www.amazon.co.uk/dp/B001JPPQS6/) and run Linux off that using its 3-drive RAID10 support. Leaving a bit of space on each for a smaller RAID1 array for /boot (booting direct form RAID10 didn't work), IIRC the 3x8Gb SD cards give a small touch under 12GiB, larger than the built-in 8Gb, and it was significantly faster (more so than I expected). The slow eMMC got used as backup storage, syncing copies of bits from /home and config from elsewhere, so I could rebuild easily should two cards die at the same time. The main problem with this arrangement was that is consumed all three of the USB ports, if I wanted to access another drive locally rather than over wireless I'd have to shut down and reboot with at least one of the drives hung off a small USB hub. For some reason it didn't work with one of the drives in the built-in SD reader (the arrays would always start in a degraded state on boot) which I never got around to looking into working around (perhaps a kernel module was missing from initrd that was needed to be able to access the hardware?). I experimented with RAID5 and RAID0 also. 5 gave a bit more space but was a lot slower for write operations, at least in artificial benchmarks, I assume due to read-before-write issues and the machine having little RAM spare for cache that might mitigate this. 0 did not give notable read performance benefits (though measurably faster writes, and obviously gave much more space), perhaps all the ports running of a single internal USB2 hub was a limiting factor.
Nextgrid 3y ago
On Linux, you can expose raw image files as storage devices using "loop devices" - the "losetup" command will turn any flat file into an attached storage device as /dev/loop*, you can then run your RAID of choice on that. By the way, if you want to play with ZFS, it supports using flat files as backing "devices" naturally, so you don't even need to fake a drive using loop devices.
jaclaz 3y ago
Well, it has already been done (JFYI): https://fitzcarraldoblog.wordpress.com/2019/10/17/creating-a-raid-of-usb-pendrives-in-linux/ https://fitzcarraldoblog.wordpress.com/2019/10/17/creating-a...
_joel 3y ago
Used to do this with mdadm/zfs etc for playing, it works well. They're just block devices at the end of the day :)
M95D 3y ago
I'm doing it right now on a router with OpenWrt and btrfs!
Someone 3y ago
If you want to get such practice, creating filesystems inside files on your disk and then mounting them is more convenient. https://askubuntu.com/questions/85977/how-do-i-create-a-file-and-mount-it-as-a-filesystem https://askubuntu.com/questions/85977/how-do-i-create-a-file... should, I think, work on any Unix-like system. On the Mac, create .dmg files for extra convenience.

Not that weird, but I've thought about RAIDing some USB thumb drives together. Might actually be a good way to get practice with your choice of RAID software for cheap, come to think of it.

buildbot 3y ago

I recall reading a great intro to ZFS article that did this! They had like 16 USB drives!

hsbauauvhabzb 3y ago

Virtual machines with many virtual disks would be much easier, or maybe I’m just lazy…

_joel 3y ago
loopback devices would be even easier... but not as fun

dspillett 3y ago

> but I've thought about RAIDing some USB thumb drives together I had a netbook with an irritatingly slow (and a bit small) eMMC drive for its main storage. I had a pair of microSD cards lying around that were faster for many IO patterns, so I bought a third and strung the three together in mini-USB readers (like https://www.amazon.co.uk/dp/B001JPPQS6/ https://www.amazon.co.uk/dp/B001JPPQS6/) and run Linux off that using its 3-drive RAID10 support. Leaving a bit of space on each for a smaller RAID1 array for /boot (booting direct form RAID10 didn't work), IIRC the 3x8Gb SD cards give a small touch under 12GiB, larger than the built-in 8Gb, and it was significantly faster (more so than I expected). The slow eMMC got used as backup storage, syncing copies of bits from /home and config from elsewhere, so I could rebuild easily should two cards die at the same time. The main problem with this arrangement was that is consumed all three of the USB ports, if I wanted to access another drive locally rather than over wireless I'd have to shut down and reboot with at least one of the drives hung off a small USB hub. For some reason it didn't work with one of the drives in the built-in SD reader (the arrays would always start in a degraded state on boot) which I never got around to looking into working around (perhaps a kernel module was missing from initrd that was needed to be able to access the hardware?). I experimented with RAID5 and RAID0 also. 5 gave a bit more space but was a lot slower for write operations, at least in artificial benchmarks, I assume due to read-before-write issues and the machine having little RAM spare for cache that might mitigate this. 0 did not give notable read performance benefits (though measurably faster writes, and obviously gave much more space), perhaps all the ports running of a single internal USB2 hub was a limiting factor.

Nextgrid 3y ago

On Linux, you can expose raw image files as storage devices using "loop devices" - the "losetup" command will turn any flat file into an attached storage device as /dev/loop*, you can then run your RAID of choice on that. By the way, if you want to play with ZFS, it supports using flat files as backing "devices" naturally, so you don't even need to fake a drive using loop devices.

jaclaz 3y ago

Well, it has already been done (JFYI): https://fitzcarraldoblog.wordpress.com/2019/10/17/creating-a-raid-of-usb-pendrives-in-linux/ https://fitzcarraldoblog.wordpress.com/2019/10/17/creating-a...

_joel 3y ago

Used to do this with mdadm/zfs etc for playing, it works well. They're just block devices at the end of the day :)

M95D 3y ago

I'm doing it right now on a router with OpenWrt and btrfs!

Someone 3y ago

If you want to get such practice, creating filesystems inside files on your disk and then mounting them is more convenient. https://askubuntu.com/questions/85977/how-do-i-create-a-file-and-mount-it-as-a-filesystem https://askubuntu.com/questions/85977/how-do-i-create-a-file... should, I think, work on any Unix-like system. On the Mac, create .dmg files for extra convenience.