5 ms·

ZFS is pretty great, but it's not perfect. Occasionally you have to dump and re-create a whole pool to fix corruption: [nsivo@news.ycombinator.com ~]$ zpoo

by kogir 12y ago

ZFS is pretty great, but it's not perfect. Occasionally you have to dump and re-create a whole pool to fix corruption: [nsivo@news.ycombinator.com ~]$ zpool status -v pool: arc state: ONLINE scan: scrub repaired 0 in 0h43m with 0 errors on Wed Aug 13 20:14:53 2014 config: NAME STATE READ WRITE CKSUM arc ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/arc0 ONLINE 0 0 0 gpt/arc1 ONLINE 0 0 0 errors: No known data errors pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 5h3m with 0 errors on Tue Aug 26 00:19:05 2014 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/tank0 ONLINE 0 0 0 gpt/tank1 ONLINE 0 0 0 gpt/tank2 ONLINE 0 0 0 gpt/tank3 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 aacd5p1 ONLINE 0 0 0 aacd6p1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /usr/nginx/var/log/nginx-access-appengine.log.0 I can't delete that file or recover it from backup, and don't really trust the pool or our disk controller anymore. So we're moving to a new server, again. Good times. I imagine this would be disastrous for someone who's filled a 24 disk array with data, and has to purchase a duplicate to safely restore to.

insaneirish 12y ago
Do you have ECC memory on that host? That corruption is not normal and suggests a problem likely with your hardware. Do you have snapshots? Can you read that file in a snapshot?
- Filligree 12y ago
  I've had the same thing happen on a similar system, with ECC. In that case I eventually tracked it down to a dodgy power supply, but the damage was done... it was time to dump and reload. Corruption happens. A good filesystem needs to cope.
- kogir 12y ago
  The memory is ECC. No data I care about was affected. I'm just surprised there's no "do whatever you need to make it consistent so I can move on with life" option and instead I have to bring up a whole new pool.
  - insaneirish 12y ago
    You should post on the illumos zfs-discuss mailing list. The most amount of experts will be there to provide comments. I just saw the hostname of the machine; your "clout" may get you some attention. :-) You very well may have some flaky hardware which is causing the issue, in which case, it probably needs to be fixed sooner than later. And maybe with some magic, you can get the pool into a happy state without rebuilding.
  - takeda 12y ago
    If you are having random errors like that it might be a sign of something bigger. As mentioned in my other response I had random bit flips in memory. After I replaced my broken NIC crashes stopped. But I got a nasty surprise when I removed a filesystem - kernel panic due to failed assertion, and then kernel panic for the same assertion every single time I tried to import the pool. I suspect it possibly could be resolved somehow without losing data, but decided to simply restore it since my backup was relatively fresh. I suspect you might have something similar to what I experienced, and even though you use ECC memory I doubt it would help much when corruption is happening on a bus.
dsturnbull2049 12y ago
This is one of the key strengths of ZFS. Your log file would be silently corrupted without checksumming.
- kogir 12y ago
  I agree completely. Much better to know than not. But having no option to knowingly just lose inconsistent data and move on is a bummer.
  - dsturnbull2049 12y ago
    A corrupt file in ZFS indicates multiple broken disks. Copy the file in from a backup, use raidz3 and/or set copies=2 for critical zfs volumes. https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p...
- rodgerd 12y ago
  Regardless, it's still frustrating that so many failure modes for ZFS have an answer of "blow it all away and start again". And that's with Solaris/ZFS.
  - laumars 12y ago
    There are other ways to fix issues with ZFS pools without having to "blow it all away and start again". In the example that started this conversation fork, there's a simple parameter to pass to the zpool utility that would have saved the OP from dropping his pool: zpool clear tank ZFS does actually come with a decent set of tools for fixing and clearing faults - which I'm starting to realise isn't as widely known (possibly because people rarely run into a situation where they need it?) Anecdotally speaking; I've ran the same pool for years on consumer hardware and that pool has even survived a failing SATA controller that was randomly dropping disks during heavy IO and causing kernel panics. So I've lost count of the number times ZFS has saved my storage pool from complete data loss where other file systems would have just given up.
takeda 12y ago
I got ZFS corruption myself. Although another issue (which actually brought my attention) was also common system crashes, whenever ARC was enabled. When inspecting cored dump it looked like random bit flips. In my case the problem turned out to be a broken WiFi card (PCI). After I replaced it no issues for 2 years now.
laumars 12y ago
You don't need to drop the pool to clear those error messages: zpool clear tank Source: http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kqod/index.html http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kqod/index... (unfortunately there's no tag ID's to hash to, so you'll have to [ctrl]+[F] to find the specific section on 'clear')
- kogir 12y ago
  Sadly, that only clears device errors. Tried it with no change.
  - laumars 12y ago
    Dropping the pool and recreating it only clears the errors as well.

by kogir 12y ago

insaneirish 12y ago

Do you have ECC memory on that host? That corruption is not normal and suggests a problem likely with your hardware. Do you have snapshots? Can you read that file in a snapshot?

Filligree 12y ago
I've had the same thing happen on a similar system, with ECC. In that case I eventually tracked it down to a dodgy power supply, but the damage was done... it was time to dump and reload. Corruption happens. A good filesystem needs to cope.
kogir 12y ago
The memory is ECC. No data I care about was affected. I'm just surprised there's no "do whatever you need to make it consistent so I can move on with life" option and instead I have to bring up a whole new pool.
- insaneirish 12y ago
  You should post on the illumos zfs-discuss mailing list. The most amount of experts will be there to provide comments. I just saw the hostname of the machine; your "clout" may get you some attention. :-) You very well may have some flaky hardware which is causing the issue, in which case, it probably needs to be fixed sooner than later. And maybe with some magic, you can get the pool into a happy state without rebuilding.
- takeda 12y ago
  If you are having random errors like that it might be a sign of something bigger. As mentioned in my other response I had random bit flips in memory. After I replaced my broken NIC crashes stopped. But I got a nasty surprise when I removed a filesystem - kernel panic due to failed assertion, and then kernel panic for the same assertion every single time I tried to import the pool. I suspect it possibly could be resolved somehow without losing data, but decided to simply restore it since my backup was relatively fresh. I suspect you might have something similar to what I experienced, and even though you use ECC memory I doubt it would help much when corruption is happening on a bus.

dsturnbull2049 12y ago

This is one of the key strengths of ZFS. Your log file would be silently corrupted without checksumming.

kogir 12y ago
I agree completely. Much better to know than not. But having no option to knowingly just lose inconsistent data and move on is a bummer.
- dsturnbull2049 12y ago
  A corrupt file in ZFS indicates multiple broken disks. Copy the file in from a backup, use raidz3 and/or set copies=2 for critical zfs volumes. https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p...
rodgerd 12y ago
Regardless, it's still frustrating that so many failure modes for ZFS have an answer of "blow it all away and start again". And that's with Solaris/ZFS.
- laumars 12y ago
  There are other ways to fix issues with ZFS pools without having to "blow it all away and start again". In the example that started this conversation fork, there's a simple parameter to pass to the zpool utility that would have saved the OP from dropping his pool: zpool clear tank ZFS does actually come with a decent set of tools for fixing and clearing faults - which I'm starting to realise isn't as widely known (possibly because people rarely run into a situation where they need it?) Anecdotally speaking; I've ran the same pool for years on consumer hardware and that pool has even survived a failing SATA controller that was randomly dropping disks during heavy IO and causing kernel panics. So I've lost count of the number times ZFS has saved my storage pool from complete data loss where other file systems would have just given up.

takeda 12y ago

I got ZFS corruption myself. Although another issue (which actually brought my attention) was also common system crashes, whenever ARC was enabled. When inspecting cored dump it looked like random bit flips. In my case the problem turned out to be a broken WiFi card (PCI). After I replaced it no issues for 2 years now.

laumars 12y ago

You don't need to drop the pool to clear those error messages: zpool clear tank Source: http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kqod/index.html http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kqod/index... (unfortunately there's no tag ID's to hash to, so you'll have to [ctrl]+[F] to find the specific section on 'clear')

kogir 12y ago
Sadly, that only clears device errors. Tried it with no change.
- laumars 12y ago
  Dropping the pool and recreating it only clears the errors as well.