6 ms·

I tried to back up a 60% full Ubuntu Linux RAID to a 4TB USB3 hard drive. I set the backup going and left it a few hours. I was shocked to come back and find

I tried to back up a 60% full Ubuntu Linux RAID to a 4TB USB3 hard drive. I set the backup going and left it a few hours. I was shocked to come back and find the RAID 100% full and only a few GB copied. It took me a few minutes to figure out what happened. Maddeningly, the USB drive has a feature that it goes to sleep after 15 minutes or so, even if Ubuntu is actively using it to write files! Maybe the drivers on Windows do something that keep it awake, but on Linux it just goes to sleep in the middle of being used. Now here's the thing, the drive DOES realize it has more to do and wake back up. But this sleep-wake causes a USB disconnect and reconnect. Which causes Ubuntu to unmount and remount the drive. Now here's the problem, perhaps because the backup program is still making the the original mount point "busy", Ubuntu doesn't re-mount the media to the path. Instead it, gets mounted at "/media/path-1". Since Linux uses regular folders as mount points, the old mount point at "/media/path" becomes a valid folder on the local disk. So the backup program keeps going, but now it's filling up the local disk. I haven't found a solution for this problem that will allow me to complete a backup (or even complete a long-duration manual copy).

nightfly 5y ago
Most linux tools have an option that keeps you from crossing filesystem boundaries
- Filligree 5y ago
  In this case he needs the opposite -- an option keeping it from not crossing filesystem boundaries.
- ta988 5y ago
  I knew that for reads how do you do that for writes?
  - p1mrx 5y ago
    chattr +i /mnt/foo This makes the mount point immutable when nothing is mounted there.
    - blooalien 5y ago
      OOooo, that's a nice trick! I'm totally gonna use that in a few places in my current setup. A little shocked at myself for never havin' thought of usin' the immutable bit in that specific way. Seems so very sensible and logical now that I see it. :)
    - ta988 5y ago
      Thanks, that will save me some serious headache!
benlivengood 5y ago
My personal opinion is that if the USB enclosure is that flaky then it's not worth trusting it to operate when it's time to do a restore. You could shuck the drive and connect it directly or find a reliable USB enclosure. Putting the disk in /etc/fstab by uuid should keep it mounted to the same directory at least, but I'll be surprised if the backup software properly handles the potential errors well. The following link has an example to remount the drive with udev when it goes away, and if the backup software isn't running as root you can mount with uid,gid set to the backup user (or chown an ext filesystem) but make the mountpoint directory 0700 owned by root to prevent the backup from writing to it while unmounted. https://blog.backslasher.net/automatically-mounting-usb-drives-and-surviving-reconnects.html https://blog.backslasher.net/automatically-mounting-usb-driv...
- indiv0 5y ago
  +1 for shucking the drive. I got a few WD externals (14 TB). They exhibited the exact behaviour described by OP. Essentially the drive slept every 15 minutes or so. Smartctl on a timer didn’t work. Touch on a timer didn’t work. I can’t recall if hdparm or some other tuning of sleep and head parking settings worked. Taking the drive out of the enclosure worked. It got so much quieter too.
  - boring_twenties 5y ago
    This is so weird. I have a few of the 10TB and 12TB WD EasyStores. I've been using them regularly for years and have never experienced anything like this. Just to be clear, the drive does go to sleep when it's actually idle. Not while it's being used, though. IOW, they work as expected. What distro/kernel version are you on? I mostly use Debian stable, which means a pretty old kernel. I wonder if it could be a regression in more recent ones? I also use the drives with FreeBSD and have never had an issue there, either.
    - benlivengood 5y ago
      I'm also curious if it's aggressive USB power saving attempts with autosuspend. From my limited exploration autosuspend is usually off and e.g. requires powertop to enable (on Ubuntu LTS) or manual fiddling, but may be worth checking.
naikrovek 5y ago
the controller in the enclosure is maybe overheating and either switching into a failsafe mode or a failure mode until it cools down again. sustained data transfers crush those little controllers, so spend more on a better enclosure, actively cool the one you have, or just connect the bare hard drive directly to your PC if that is an option. I've had 2 USB-3 to mSATA enclosures fail permanently after 5 minutes of sustained data transfers.
ta988 5y ago
can you give the brand and model so we all avoid that?
deleted 5y ago
[deleted]
depingus 5y ago
Alternatively, I have a Debian file server with 24Tb in internal drives and 3 x 8Tb external USB drives (WD EasyStores). The 3 USB drives are connected all the time and setup with MergeFS to appear as one massive 24Tb mount (ext4 underlying). I use rsync to do backups. The initial backup of 16Tb took many hours. About once a month I initiate a manual backup that often runs over an hour. I’ve never had this problem.
deathanatos 5y ago
I have a USB 2 external HDD enclosure that will drop off the bus if it is connected to a USB 3 port and under any respectable load. Only workaround I know of is to connect it to a USB port while giving the logo a stern, backwards-compatible glare.
Abishek_Muthian 5y ago
I'm certain that this is a problem with the drive firmware or USB controller and not Linux per se. I use several USB storage devices, enclosures on Linux everyday including using separate USB ssd caches. Can you check with your storage device managing software on Windows on whether it has the sleep option or spin down set? I remember seeing one in the Seagate SW on mac although it has an OS level option to put HDD to sleep when not in use. If that doesn't work you can try using hdparm or sdparm to modify the power management and spin down timer. If everything else fails then disable USB autosuspend in the kernel boot option.
- ithkuil 5y ago
  Sleep is one problem. The other problem is how the backup software reacts to the possible failure modes. A backup tool should identify the backup destination in a reliable way and nountpoints is clearly not a reliable way to identify a backup destination
  - Abishek_Muthian 5y ago
    True but the parent didn't say how the backup is happening, It could have even been a rsync.
    - ithkuil 5y ago
      a shell script running rsync is "a backup automation software"
boring_twenties 5y ago
This seems like it must be an issue with the drive you're using. I use a mix of Western Digital and Seagate USB drives and have never had a problem like this.
john2010 5y ago
Perhaps it is SMR drive. Ubuntu or linux is generally poor at sleeping USB. Ubuntu always mounts at /media/username/drivename - I am sure something is wrong with your setup.
teekert 5y ago
I have a Pi3 at my parents home and I rsync over ssh to a 5 TB 2.5" spinning disc attached via USB, I added the disc to fstab. The first sync took over 24 hrs for ~550 GB [0]. We both have 100/100 fiber, but USB is limiting in this case, which is nice because it kept both our connections usable during transfer. Anyway, I had 0 issues. Well, apart from the fact that I initially used vfat as the FS of the target disc so my partner or parents could just pull out the disc and use it (on Windows/Mac) when something would happen to me. But this leads to rsync issues and needs workarounds so I switched it to ext4 and told my loved ones to find a Linux nerd in the event of my demise. [0] rsync -ahv --progress -e 'ssh -p2222' /data/0/Pictures bu.mydomain.tld:/mnt/data/backups/ --info=progress2 #(assumes cert based auth set up)
brador 5y ago
Why do you refuse to blame Linux for this problem?
- blooalien 5y ago
  Because not everything that happens on Linux is Linux's fault, any more than every thing that goes wrong on Windows is Microsoft's fault? Sometimes it's user error, sometimes it's faulty hardware, sometimes it is the operating system. You don't just lay blame on the operating system for everything that's not working though (although I do see a lot of people do exactly that, despite it not being logical). As an example; Many folks commenting above appear to have never experienced a problem with hard drives spinning down during a backup, or it causing a remount at a different mount point. This leads to thinking it might not be an operating system issue, but rather something strange about that specific external hard drive.
AnIdiotOnTheNet 5y ago
Huh, it's almost like treating separate volumes as part of the same file hierarchy is a bad abstraction.
beagle3 5y ago
I had related issues. My solution since then was to work through a soft link to a directory inside the mounted drive. Whatever the failure mode, when the storage disappears, even if the users do a “mkdir -p” (which seems to be the underlying issue you are describing)

nightfly 5y ago

Most linux tools have an option that keeps you from crossing filesystem boundaries

Filligree 5y ago
In this case he needs the opposite -- an option keeping it from not crossing filesystem boundaries.
ta988 5y ago
I knew that for reads how do you do that for writes?
- p1mrx 5y ago
  chattr +i /mnt/foo This makes the mount point immutable when nothing is mounted there.
  - blooalien 5y ago
    OOooo, that's a nice trick! I'm totally gonna use that in a few places in my current setup. A little shocked at myself for never havin' thought of usin' the immutable bit in that specific way. Seems so very sensible and logical now that I see it. :)
  - ta988 5y ago
    Thanks, that will save me some serious headache!

benlivengood 5y ago

My personal opinion is that if the USB enclosure is that flaky then it's not worth trusting it to operate when it's time to do a restore. You could shuck the drive and connect it directly or find a reliable USB enclosure. Putting the disk in /etc/fstab by uuid should keep it mounted to the same directory at least, but I'll be surprised if the backup software properly handles the potential errors well. The following link has an example to remount the drive with udev when it goes away, and if the backup software isn't running as root you can mount with uid,gid set to the backup user (or chown an ext filesystem) but make the mountpoint directory 0700 owned by root to prevent the backup from writing to it while unmounted. https://blog.backslasher.net/automatically-mounting-usb-drives-and-surviving-reconnects.html https://blog.backslasher.net/automatically-mounting-usb-driv...

indiv0 5y ago
+1 for shucking the drive. I got a few WD externals (14 TB). They exhibited the exact behaviour described by OP. Essentially the drive slept every 15 minutes or so. Smartctl on a timer didn’t work. Touch on a timer didn’t work. I can’t recall if hdparm or some other tuning of sleep and head parking settings worked. Taking the drive out of the enclosure worked. It got so much quieter too.
- boring_twenties 5y ago
  This is so weird. I have a few of the 10TB and 12TB WD EasyStores. I've been using them regularly for years and have never experienced anything like this. Just to be clear, the drive does go to sleep when it's actually idle. Not while it's being used, though. IOW, they work as expected. What distro/kernel version are you on? I mostly use Debian stable, which means a pretty old kernel. I wonder if it could be a regression in more recent ones? I also use the drives with FreeBSD and have never had an issue there, either.
  - benlivengood 5y ago
    I'm also curious if it's aggressive USB power saving attempts with autosuspend. From my limited exploration autosuspend is usually off and e.g. requires powertop to enable (on Ubuntu LTS) or manual fiddling, but may be worth checking.

naikrovek 5y ago

the controller in the enclosure is maybe overheating and either switching into a failsafe mode or a failure mode until it cools down again. sustained data transfers crush those little controllers, so spend more on a better enclosure, actively cool the one you have, or just connect the bare hard drive directly to your PC if that is an option. I've had 2 USB-3 to mSATA enclosures fail permanently after 5 minutes of sustained data transfers.

ta988 5y ago

can you give the brand and model so we all avoid that?

deleted 5y ago

[deleted]

depingus 5y ago

Alternatively, I have a Debian file server with 24Tb in internal drives and 3 x 8Tb external USB drives (WD EasyStores). The 3 USB drives are connected all the time and setup with MergeFS to appear as one massive 24Tb mount (ext4 underlying). I use rsync to do backups. The initial backup of 16Tb took many hours. About once a month I initiate a manual backup that often runs over an hour. I’ve never had this problem.

deathanatos 5y ago

I have a USB 2 external HDD enclosure that will drop off the bus if it is connected to a USB 3 port and under any respectable load. Only workaround I know of is to connect it to a USB port while giving the logo a stern, backwards-compatible glare.

Abishek_Muthian 5y ago

I'm certain that this is a problem with the drive firmware or USB controller and not Linux per se. I use several USB storage devices, enclosures on Linux everyday including using separate USB ssd caches. Can you check with your storage device managing software on Windows on whether it has the sleep option or spin down set? I remember seeing one in the Seagate SW on mac although it has an OS level option to put HDD to sleep when not in use. If that doesn't work you can try using hdparm or sdparm to modify the power management and spin down timer. If everything else fails then disable USB autosuspend in the kernel boot option.

ithkuil 5y ago
Sleep is one problem. The other problem is how the backup software reacts to the possible failure modes. A backup tool should identify the backup destination in a reliable way and nountpoints is clearly not a reliable way to identify a backup destination
- Abishek_Muthian 5y ago
  True but the parent didn't say how the backup is happening, It could have even been a rsync.
  - ithkuil 5y ago
    a shell script running rsync is "a backup automation software"

boring_twenties 5y ago

This seems like it must be an issue with the drive you're using. I use a mix of Western Digital and Seagate USB drives and have never had a problem like this.

john2010 5y ago

Perhaps it is SMR drive. Ubuntu or linux is generally poor at sleeping USB. Ubuntu always mounts at /media/username/drivename - I am sure something is wrong with your setup.

teekert 5y ago

I have a Pi3 at my parents home and I rsync over ssh to a 5 TB 2.5" spinning disc attached via USB, I added the disc to fstab. The first sync took over 24 hrs for ~550 GB [0]. We both have 100/100 fiber, but USB is limiting in this case, which is nice because it kept both our connections usable during transfer. Anyway, I had 0 issues. Well, apart from the fact that I initially used vfat as the FS of the target disc so my partner or parents could just pull out the disc and use it (on Windows/Mac) when something would happen to me. But this leads to rsync issues and needs workarounds so I switched it to ext4 and told my loved ones to find a Linux nerd in the event of my demise. [0] rsync -ahv --progress -e 'ssh -p2222' /data/0/Pictures bu.mydomain.tld:/mnt/data/backups/ --info=progress2 #(assumes cert based auth set up)

brador 5y ago

Why do you refuse to blame Linux for this problem?

blooalien 5y ago
Because not everything that happens on Linux is Linux's fault, any more than every thing that goes wrong on Windows is Microsoft's fault? Sometimes it's user error, sometimes it's faulty hardware, sometimes it is the operating system. You don't just lay blame on the operating system for everything that's not working though (although I do see a lot of people do exactly that, despite it not being logical). As an example; Many folks commenting above appear to have never experienced a problem with hard drives spinning down during a backup, or it causing a remount at a different mount point. This leads to thinking it might not be an operating system issue, but rather something strange about that specific external hard drive.

AnIdiotOnTheNet 5y ago

Huh, it's almost like treating separate volumes as part of the same file hierarchy is a bad abstraction.

beagle3 5y ago

I had related issues. My solution since then was to work through a soft link to a directory inside the mounted drive. Whatever the failure mode, when the storage disappears, even if the users do a “mkdir -p” (which seems to be the underlying issue you are describing)