9 ms·
I thought you shouldn't be running ZFS inside a virtualized environment?
by briankwest 10y ago
I thought you shouldn't be running ZFS inside a virtualized environment?
- zorked 10y agoWhy?
- briankwest 10y agoLooks like its not an issue, Was just reading the warnings a year or so ago specifically relating to FreeNAS and how it would corrupt your data.
- jlgaddis 10y agoIdeally, ZFS has exclusive control over the storage. When it's virtualized, it doesn't and there may be various HBAs, raid controllers, etc., in between ZFS and the actual disks. These can (do) "get in the way" and you (can) lose one of the biggest features of ZFS (data integrity and error correction).
- simcop2387 10y agoFor a little more detail about this, it's somewhat common for these controllers to lie about when thinks hit the disk. They'll cache it and then write either out but if they lose power or a write gets corrupted zfs will have no way of discovering it until it's too late.
- bsuh 10y agoAre other file systems better off with recovery when interacting with virtual disks and host power failure?
- ryao 10y agoThe short answer is no.
- pkaye 10y agoSearch online for published papers related to "IRON File Systems." Some researchers injected errors into various parts of common file systems and see how well they recovered. I think ZFS was the best of the bunch though that research is from a few years back and things may have improved elsewhere.
- deleted 10y ago[deleted]
- fulafel 10y agoThe IRON paper doesn't mention ZFS, I found some citing papers that focus on ZFS but have no comparisons.
- pkaye 10y agoYes you are right. I believe it was a different paper by the same research group... http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf
- IgorPartola 10y agoThey are not, but they are better at swallowing the errors and not bothering you with such details. ZFS fails fast & early, while EXT4 will fail when you realize your Postgres DB is borked. I guess it's possible that some type of disk command timing could cause unexpected lockups or slowdowns that you wouldn't get with a system that doesn't try to control the hardware to quite the same extent as ZFS, but my (cursory) understanding is that it's rare/hardware specific. My personal take is that running ZFS on hardware that lies is no worse than running EXT4 on it. YMMV as I'm not a storage expert.
- aidenn0 10y agoShort answer "no" with a longer answer of "ZFS is sometimes better than most other options"
- fulafel 10y agoIf the storage lies about syncs, the best you can hope for is replaying a consistent state somewhere in the past. Log structured filesystems with checksums would be a good bet here.
- dekhn 10y agothere's nothing ZFS specific about this. Your admins should be ensuring that you're not doing concurrent access to shared storage, or if you are, you're using a lock-based cluster manager.
- lscotte 10y agoThe keyword is definitely "ideally" :-) With a cloud provider like AWS, storage is always virtualized - so we've always got that working against us. I see ZFS in AWS more about flexibility than data integrity, although having said that ZFS should do just as well or better (certainly no worse) than EXT4, XFS, or BTRFS for reliability. The ability to add storage dynamically without having to move bits around is powerful.
- drvdevd 10y agoAnd I've actually had (a couple times now) random hardware failings, with random notification emails from Amazon, on EBS storage on AWS, with accompanying data loss. Might as well treat your zpool like it's on real hardware and configure raidz accordingly. The cloud does have real, problematic hardware behind it, and it's important to remember that. [edit] especially if you can configure your block devices such that you know they're sitting on different physical hardware at the cloud data center, you will gain that benefit of ZFS
- deelowe 10y agoThere are virtualization solutions that provide exclusive access to storage. Hyper-V surprisingly is one such solution and I've been running ZFS inside of Hyper-V instances for a couple of years now with no issue.
- beachstartup 10y agopass-through device virtualization of various types provide this facility.
- agumonkey 10y agoSomehow it's a good thing ZFS gets to be tried on AWS, it might trigger some bugs.
- scoot 10y ago"there may be various HBAs, raid controllers, etc., in between ZFS and the actual disks." Or in the case if EBS - NFS and a network stack. EBS is "Emulated Block Storage" despite what AWS may like you to think.
- cthalupa 10y agoSource on EBS being NFS?
- Unklejoe 10y agoDoes this same idea apply when running ZFS on top of the Linux LUKS mapper (whatever the proper terminology is)?
- lscotte 10y agoI haven't tested it with /boot in ZFS, but I have a laptop that has an EXT /boot partition and a LUKS partition with ZFS for everything else (/, /home, etc). It works just fine - and I've never noticed any performance issues with this setup - obviously LUKS has some overhead, but running ZFS on LUKS works as well as EXT4 or BTRFS on LUKS, with all the advantages of ZFS.
- jlgaddis 10y agoIn an ideal perfect world, you wouldn't. In reality, however, plenty of people do and, personally, I've not experienced any issues with it.
- kernelbro 10y agoI've always heard the same. Is there a filesystem that _is_ recommended for virtual envs?
- napsterbr 10y agoActually with pci pass-through it's quite safe to run zfs on top of a virtual machine. Take a look at this great post: http://www.freenas.org/blog/yes-you-can-virtualize-freenas/ http://www.freenas.org/blog/yes-you-can-virtualize-freenas/