7 ms·

Raspberry Pi – UASP, Trim, and Boot Performance via USB

220 points by geerlingguy 6y ago

geerlingguy 6y ago
tl;dr - the Pi 4 now has USB boot enabled by default, and you can get way better performance from a fast USB 3.0 drive than the built-in microSD. I like the Arcanite for large files and basic storage needs, or an NVMe + USB 3.0 adapter for the best performance for a reasonable price. UASP can speed up operations by up to 50%, and many SSDs and high end flash drives do support it. TRIM support is less prevalent, and you often need to manually enable it (there's a guide linked in the post). Weirdest finding: the Arcanite flash drive firmware indicates it supports trim, but when I tried running fstrim something blew up in the drive, and I could no longer read/mount/initialize it from Mac/Linux/Windows (I had to buy a replacement to finish the testing).
- fuzzybear3965 6y ago
  In hindsight, how would you have used the documentation of the Raspberry Pi architecture to determine whether UASP is supported? Also, I'm really curious if your only problem with that Arcanite was only that it was a defective unit. If anyone reading has any other ideas for diagnosing that problem then I'd love to know them. Great guide, Jeff. Really enjoyable (and informative!).
  - ranma42 6y ago
    > In hindsight, how would you have used the documentation of the Raspberry Pi architecture to determine whether UASP is supported? Doesn't really depend on the Pi hardware at all, it depends on the kernel driver (kernel must be new enough) and the USB storage device (does it implement UASP in addition to legacy bulk-only-transport?). https://www.devever.net/~hl/usbuas https://www.devever.net/~hl/usbuas has a great explanation of the nitty-gritty details.
- 867-5309 6y ago
  it would be interesting to see some NVMe benchmarks for the Pi 4, both enclosure and drive
  - geerlingguy 6y ago
    I've got you covered, at least for an XPG NVMe in a TDBT enclosure: https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-options-raspberry-pi https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o...
- bigiain 6y ago
  Thanks for doing all that testing! Nice to see that answer pretty much boils down to "Use something decent-ish, mostly it doesn't matter which choice you make"... (I would have guessed nVME would have been worth spending extra on, turns out not by enough to care about, at least not in a Pi4.)
  - geerlingguy 6y ago
    Yeah the Pi 4's USB 3.0 bus isn't able to take advantage of the speeds faster drives can put out; I'm hoping the next compute module can support NVMe or SATA more directly (something Eben Upton hinted at in a podcast earlier this summer). A Pi 4 B+ or Pi 5 could even fit a slot underneath?
    - Zenst 6y ago
      I'd expect the compute flavour to expose the PCIe bus, so SATA or NVMe would be easily added as no USB3 chip hogging it. Indeed I'd say that is not only a given avenue, but the angle Mr Upton was hinting towards given the I/O side of things currently from the SOC. Like many - pre Pi4, was wishing for a SATA port, but overall, if they did do some upgrade to the Pi4 in another B+ or 5 flavour - then I'd be more interested in a faster USB flavour, like 3.2 or USB-C. Reason being is that it would be more flexible and those who want SATA or NVMe would be able to tap into that. Certainly more flexible in usage and that does seem to be key.
    - The_Hermit 6y ago
      Yes, I'm also anxiously waiting for the CM4, but wouldn't M.2, NVMe, & etc. be located on the carrier boards, instead of the Computer Module?
- dvdbloc 6y ago
  Do you mean the Arcanite flash drive literally let the smoke out or did it just stop working?
  - rwmj 6y ago
    Not likely to be smoke. However TRIM failures were very common with SSDs early on and even today some distros disable or partially disable it by default (see for example this Debian warning: https://wiki.debian.org/SSDOptimization#WARNING https://wiki.debian.org/SSDOptimization#WARNING). Upstream Linux does not enable TRIM for MD-raid devices unless you use a special kernel boot parameter. "Continuous TRIM" (which means setting the discard parameter on the filesystem so that files are automatically trimmed on deletion) is also a good way to reveal bugs and problems, some of which will merely kill performance when you delete files, and others will nuke your data.
  - geerlingguy 6y ago
    No smoke, just stopped working entirely, and can't mount it or partition it at all. Closest I can get is with Disk Utility on the Mac, if I partition it, it acts like it's working, then it gives an error on the last block of the device, something like that.
zaroth 6y ago
Wow, the animated gif of the Defrag running sent me way back. I used to run that like it was religion, watching the blocks all move together and change color, and knowing the machine would run just a bit faster after it was all done was a bit of magic.
- xattt 6y ago
  It felt like you were cleaning and tidying up.
- joosters 6y ago
  It felt oddly soothing watching the animation - ironic, really, since back when running defrag I'd be madly impatient for it to finish.
- bitwize 6y ago
  It gave me flashbacks to high school computer class, in which we were taught to use a tool misnamed COMPRESS -- actually a defragger -- to maintain our hard disks. It displayed a similar animation in text mode as it worked. This was on OG IBM 5150s -- ancient even at the time. It's funny how, in these late-stage Moore's Law days, a 10-year-old PC is old certainly, but not as hopelessly outmoded now as the 5150 was then.
- mobilio 6y ago
  No worries! IT'S ALIVE! http://hultbergs.org/defrag/ http://hultbergs.org/defrag/
- lloydatkinson 6y ago
  It's a shame Windows didn't show the date in the system clock until Vista, because it would have been fun to see when that GIF was recorded/made.
avian 6y ago
Does anyone have any experience how reliable is such modern USB attached storage when run 24/7? Some years back I made a box (x86, not RPi) with a bunch of USB connected drives (from Western Digital, if I remember correctly). After a few days of uptime the drives would simply drop from the bus. I was guessing there was a bug in the drive’s USB controller. These devices are usually meant to be used as a carrier that’s only connected to a computer to load/off load data. Long-term operation probably isn’t a priority when developing them.
- adrian_b 6y ago
  "USB attached storage" includes not only USB sticks, but many other devices, e.g. memory cards or SSD's connected through UDB adapters. I have run 24/7 many SSD's attached via USB, with no reliability problems. Usually buying SATA or NVME SSD's and separate USB adapters for them is cheaper than buying USB SSD's, which actually contain an internal USB adapter.
- Zenst 6y ago
  I ran 5 USB drivers (4x USB3 and 1x USB2) of a powered USB3 hub attached to a RPi4 a few weeks back as was clearing up some drives and data juggling between them. Had that running for few days with no issues at all and was thrashed a few times with moves/copies across all the drives. So indications are good, and the drives in this case was a WD SSD, 3x Toshiba bus powered drives and an ancient 320gb usb externally powered WD affair. So a but of a mix. Though I did have it plugged into a UPS, which like all things tech - kinda helps smooth out the random chaos of electricity and tech.
  - imtringued 6y ago
    Most storage corruption problems related Rasberry PIs are just PSU problems. Flash drives and SD cards draw their power directly from the Rasberry PI and since those only use a crappy micro USB port or a buggy USB C port you're going to run into issues, especially if you use a random USB charger instead of the official power supplies. External storage like an SSD that uses its own power supply (via the USB hub) will be far more reliable.
- 6y ago
alexellisuk 6y ago
Good to see you up here on the front page Jeff, well-deserved for all your RPi tutorials and videos.
- geerlingguy 6y ago
  Ha, good to see you in the comments too!
jeswin 6y ago
Samsung Evo+ cards with the same speed rating (say U1) and model number can vary in performance quite a bit. I have tried three of them on my rpi4, and found one which was considerably faster than the others. To be sure I verified it with crystaldiskmark and indeed, it was up to 2x faster in writes and reads. Samsung cards tend to outperform other brands in that price range.
megous 6y ago
How does this change with LUKS enabled? I don't see a point of storing data unencrypted these days.
- dijit 6y ago
  For normal laptops it won’t be so much difference in performance, as LUKS by default uses AES-XT as the encryption type, and most modern CPUs have AES accelerators in them. For rPi though? Not sure they have an AES accelerator, and if they don’t then it’s going to be a huge cost to performance. Not only because USB is using a CPU core to operate (because USB uses CPU not dedicated hardware like FireWire) but also because it has to transparently translate and encrypt every access. It seems like the raspberry actually has zero crypto extensions anyway: https://www.raspberrypi.org/forums/viewtopic.php?t=207888 https://www.raspberrypi.org/forums/viewtopic.php?t=207888
  - I_Byte 6y ago
    >Not sure they have an AES accelerator The ARM chips in the all of the Raspberry pis do not have AES accelerators. Encrypting your pis drive is going to be a serious hit on performance like you said. I don't recommend it. If you really really really want to encrypt your Rpi then you need to use a disk cipher like Adiantum. It is based off of ChaCha12 and was made specifically for low end CPUs that lack AES accelerators.
  - megous 6y ago
    I know it doesn't have AES encryption acceleration (at least in the CPU). That's why it's an interesting thing to measure, how much the competition between boot tasks and AES encryption affect the boot. It may not be much because I suspect boot is IO bound, but I don't know.
intrasight 6y ago
I've a rpi4 that I'm setting up as a media player. Files come from a NAS. In the past (rpi3) I've just used an SD card in the Pi. I'm curious if I'd get better performance using USB. Also curious if I did use USB if I could setup automatic caching to the drive.
- NortySpock 6y ago
  I'm using an RPi 4 as a NAS/home server and the difference between SD speeds and USB speeds was pretty noticeable (subjectively); even simple linux commands (ls, cp -r, mv) sped up after switching to a SanDisk Ultra Flair. Except for one USB drive... Recently I had the misfortune of having a debian update screw up my docker daemon and I ran over to Best Buy to grab any-old-USB drive to use as a new OS drive. As it happens the SD copy over went fine, but as I performed basic "setting the server up" operations like copying over the old data (50 GB of various sizes) and firing up docker-compose scripts, performance deteriorated rapidly to the point where a docker install of a lighttpd container took at least 30 minutes and the system became effectively unusable. The poor-performing drive was the PNY Elite-X Fit, USB 3.0, 128GB, purchased for ~$22 I've switched back to a SanDisk SD card for the moment which is perfectly adequate for the next few months but I hope to get back to running off a USB soon when I'm not so busy.
- geerlingguy 6y ago
  For the Pi 3 B+ and previous generations, the difference will be much less pronounced, but still can be measurable. For the Pi 4, a faster USB drive (e.g. SSD or NVMe in adapter) will be a massive improvement for many things (though some operations like launching a browser won't be vastly improved).
NortySpock 6y ago
Just wanted to say thanks for doing this research on flash speeds and posting your results for the last few years, they've helped me pick both reasonable sd-cards and mid-cost USB thumb drives that work for my budget and RPi 4.
londons_explore 6y ago
These reports always comment on raw performance without checking if the drive actually meets the storage contract with the host. That's like having a 'hall of fame' for the fastest marathon runners, but not having anyone keep an eye on if they took a taxi half the route... These benchmarks should always have a section where a drive has its power pulled repeatedly while under heavy load. If, ever, a single sector becomes overwritten, rolled back, or unreadable, except those being written between a pair of write barriers, then the drive has failed the test, and all other performance results thrown out. That drive took the taxi.
destitude 6y ago
Is there a reason the XPG SX8200 256GB SSD with 3500MB/s wasn't tested? Would there be a performance boost compared to what was tested on the Pi 4?
brodouevencode 6y ago
As an end user of Raspis and all their improvements over the past few years, to anyone connected to or working on the project, thank you.

220 points by geerlingguy 6y ago

tl;dr - the Pi 4 now has USB boot enabled by default, and you can get way better performance from a fast USB 3.0 drive than the built-in microSD. I like the Arcanite for large files and basic storage needs, or an NVMe + USB 3.0 adapter for the best performance for a reasonable price. UASP can speed up operations by up to 50%, and many SSDs and high end flash drives do support it. TRIM support is less prevalent, and you often need to manually enable it (there's a guide linked in the post). Weirdest finding: the Arcanite flash drive firmware indicates it supports trim, but when I tried running fstrim something blew up in the drive, and I could no longer read/mount/initialize it from Mac/Linux/Windows (I had to buy a replacement to finish the testing).

fuzzybear3965 6y ago
In hindsight, how would you have used the documentation of the Raspberry Pi architecture to determine whether UASP is supported? Also, I'm really curious if your only problem with that Arcanite was only that it was a defective unit. If anyone reading has any other ideas for diagnosing that problem then I'd love to know them. Great guide, Jeff. Really enjoyable (and informative!).
- ranma42 6y ago
  > In hindsight, how would you have used the documentation of the Raspberry Pi architecture to determine whether UASP is supported? Doesn't really depend on the Pi hardware at all, it depends on the kernel driver (kernel must be new enough) and the USB storage device (does it implement UASP in addition to legacy bulk-only-transport?). https://www.devever.net/~hl/usbuas https://www.devever.net/~hl/usbuas has a great explanation of the nitty-gritty details.
867-5309 6y ago
it would be interesting to see some NVMe benchmarks for the Pi 4, both enclosure and drive
- geerlingguy 6y ago
  I've got you covered, at least for an XPG NVMe in a TDBT enclosure: https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-options-raspberry-pi https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o...
bigiain 6y ago
Thanks for doing all that testing! Nice to see that answer pretty much boils down to "Use something decent-ish, mostly it doesn't matter which choice you make"... (I would have guessed nVME would have been worth spending extra on, turns out not by enough to care about, at least not in a Pi4.)
- geerlingguy 6y ago
  Yeah the Pi 4's USB 3.0 bus isn't able to take advantage of the speeds faster drives can put out; I'm hoping the next compute module can support NVMe or SATA more directly (something Eben Upton hinted at in a podcast earlier this summer). A Pi 4 B+ or Pi 5 could even fit a slot underneath?
  - Zenst 6y ago
    I'd expect the compute flavour to expose the PCIe bus, so SATA or NVMe would be easily added as no USB3 chip hogging it. Indeed I'd say that is not only a given avenue, but the angle Mr Upton was hinting towards given the I/O side of things currently from the SOC. Like many - pre Pi4, was wishing for a SATA port, but overall, if they did do some upgrade to the Pi4 in another B+ or 5 flavour - then I'd be more interested in a faster USB flavour, like 3.2 or USB-C. Reason being is that it would be more flexible and those who want SATA or NVMe would be able to tap into that. Certainly more flexible in usage and that does seem to be key.
  - The_Hermit 6y ago
    Yes, I'm also anxiously waiting for the CM4, but wouldn't M.2, NVMe, & etc. be located on the carrier boards, instead of the Computer Module?
dvdbloc 6y ago
Do you mean the Arcanite flash drive literally let the smoke out or did it just stop working?
- rwmj 6y ago
  Not likely to be smoke. However TRIM failures were very common with SSDs early on and even today some distros disable or partially disable it by default (see for example this Debian warning: https://wiki.debian.org/SSDOptimization#WARNING https://wiki.debian.org/SSDOptimization#WARNING). Upstream Linux does not enable TRIM for MD-raid devices unless you use a special kernel boot parameter. "Continuous TRIM" (which means setting the discard parameter on the filesystem so that files are automatically trimmed on deletion) is also a good way to reveal bugs and problems, some of which will merely kill performance when you delete files, and others will nuke your data.
- geerlingguy 6y ago
  No smoke, just stopped working entirely, and can't mount it or partition it at all. Closest I can get is with Disk Utility on the Mac, if I partition it, it acts like it's working, then it gives an error on the last block of the device, something like that.

zaroth 6y ago

Wow, the animated gif of the Defrag running sent me way back. I used to run that like it was religion, watching the blocks all move together and change color, and knowing the machine would run just a bit faster after it was all done was a bit of magic.

xattt 6y ago
It felt like you were cleaning and tidying up.
joosters 6y ago
It felt oddly soothing watching the animation - ironic, really, since back when running defrag I'd be madly impatient for it to finish.
bitwize 6y ago
It gave me flashbacks to high school computer class, in which we were taught to use a tool misnamed COMPRESS -- actually a defragger -- to maintain our hard disks. It displayed a similar animation in text mode as it worked. This was on OG IBM 5150s -- ancient even at the time. It's funny how, in these late-stage Moore's Law days, a 10-year-old PC is old certainly, but not as hopelessly outmoded now as the 5150 was then.
mobilio 6y ago
No worries! IT'S ALIVE! http://hultbergs.org/defrag/ http://hultbergs.org/defrag/
lloydatkinson 6y ago
It's a shame Windows didn't show the date in the system clock until Vista, because it would have been fun to see when that GIF was recorded/made.

avian 6y ago

Does anyone have any experience how reliable is such modern USB attached storage when run 24/7? Some years back I made a box (x86, not RPi) with a bunch of USB connected drives (from Western Digital, if I remember correctly). After a few days of uptime the drives would simply drop from the bus. I was guessing there was a bug in the drive’s USB controller. These devices are usually meant to be used as a carrier that’s only connected to a computer to load/off load data. Long-term operation probably isn’t a priority when developing them.

adrian_b 6y ago
"USB attached storage" includes not only USB sticks, but many other devices, e.g. memory cards or SSD's connected through UDB adapters. I have run 24/7 many SSD's attached via USB, with no reliability problems. Usually buying SATA or NVME SSD's and separate USB adapters for them is cheaper than buying USB SSD's, which actually contain an internal USB adapter.
Zenst 6y ago
I ran 5 USB drivers (4x USB3 and 1x USB2) of a powered USB3 hub attached to a RPi4 a few weeks back as was clearing up some drives and data juggling between them. Had that running for few days with no issues at all and was thrashed a few times with moves/copies across all the drives. So indications are good, and the drives in this case was a WD SSD, 3x Toshiba bus powered drives and an ancient 320gb usb externally powered WD affair. So a but of a mix. Though I did have it plugged into a UPS, which like all things tech - kinda helps smooth out the random chaos of electricity and tech.
- imtringued 6y ago
  Most storage corruption problems related Rasberry PIs are just PSU problems. Flash drives and SD cards draw their power directly from the Rasberry PI and since those only use a crappy micro USB port or a buggy USB C port you're going to run into issues, especially if you use a random USB charger instead of the official power supplies. External storage like an SSD that uses its own power supply (via the USB hub) will be far more reliable.
6y ago

alexellisuk 6y ago

Good to see you up here on the front page Jeff, well-deserved for all your RPi tutorials and videos.

geerlingguy 6y ago
Ha, good to see you in the comments too!

jeswin 6y ago

Samsung Evo+ cards with the same speed rating (say U1) and model number can vary in performance quite a bit. I have tried three of them on my rpi4, and found one which was considerably faster than the others. To be sure I verified it with crystaldiskmark and indeed, it was up to 2x faster in writes and reads. Samsung cards tend to outperform other brands in that price range.

megous 6y ago

How does this change with LUKS enabled? I don't see a point of storing data unencrypted these days.

dijit 6y ago
For normal laptops it won’t be so much difference in performance, as LUKS by default uses AES-XT as the encryption type, and most modern CPUs have AES accelerators in them. For rPi though? Not sure they have an AES accelerator, and if they don’t then it’s going to be a huge cost to performance. Not only because USB is using a CPU core to operate (because USB uses CPU not dedicated hardware like FireWire) but also because it has to transparently translate and encrypt every access. It seems like the raspberry actually has zero crypto extensions anyway: https://www.raspberrypi.org/forums/viewtopic.php?t=207888 https://www.raspberrypi.org/forums/viewtopic.php?t=207888
- I_Byte 6y ago
  >Not sure they have an AES accelerator The ARM chips in the all of the Raspberry pis do not have AES accelerators. Encrypting your pis drive is going to be a serious hit on performance like you said. I don't recommend it. If you really really really want to encrypt your Rpi then you need to use a disk cipher like Adiantum. It is based off of ChaCha12 and was made specifically for low end CPUs that lack AES accelerators.
- megous 6y ago
  I know it doesn't have AES encryption acceleration (at least in the CPU). That's why it's an interesting thing to measure, how much the competition between boot tasks and AES encryption affect the boot. It may not be much because I suspect boot is IO bound, but I don't know.

intrasight 6y ago

I've a rpi4 that I'm setting up as a media player. Files come from a NAS. In the past (rpi3) I've just used an SD card in the Pi. I'm curious if I'd get better performance using USB. Also curious if I did use USB if I could setup automatic caching to the drive.

NortySpock 6y ago
I'm using an RPi 4 as a NAS/home server and the difference between SD speeds and USB speeds was pretty noticeable (subjectively); even simple linux commands (ls, cp -r, mv) sped up after switching to a SanDisk Ultra Flair. Except for one USB drive... Recently I had the misfortune of having a debian update screw up my docker daemon and I ran over to Best Buy to grab any-old-USB drive to use as a new OS drive. As it happens the SD copy over went fine, but as I performed basic "setting the server up" operations like copying over the old data (50 GB of various sizes) and firing up docker-compose scripts, performance deteriorated rapidly to the point where a docker install of a lighttpd container took at least 30 minutes and the system became effectively unusable. The poor-performing drive was the PNY Elite-X Fit, USB 3.0, 128GB, purchased for ~$22 I've switched back to a SanDisk SD card for the moment which is perfectly adequate for the next few months but I hope to get back to running off a USB soon when I'm not so busy.
geerlingguy 6y ago
For the Pi 3 B+ and previous generations, the difference will be much less pronounced, but still can be measurable. For the Pi 4, a faster USB drive (e.g. SSD or NVMe in adapter) will be a massive improvement for many things (though some operations like launching a browser won't be vastly improved).

NortySpock 6y ago

Just wanted to say thanks for doing this research on flash speeds and posting your results for the last few years, they've helped me pick both reasonable sd-cards and mid-cost USB thumb drives that work for my budget and RPi 4.

londons_explore 6y ago

These reports always comment on raw performance without checking if the drive actually meets the storage contract with the host. That's like having a 'hall of fame' for the fastest marathon runners, but not having anyone keep an eye on if they took a taxi half the route... These benchmarks should always have a section where a drive has its power pulled repeatedly while under heavy load. If, ever, a single sector becomes overwritten, rolled back, or unreadable, except those being written between a pair of write barriers, then the drive has failed the test, and all other performance results thrown out. That drive took the taxi.

destitude 6y ago

Is there a reason the XPG SX8200 256GB SSD with 3500MB/s wasn't tested? Would there be a performance boost compared to what was tested on the Pi 4?

brodouevencode 6y ago

As an end user of Raspis and all their improvements over the past few years, to anyone connected to or working on the project, thank you.