12 ms·
ZFSin: A Windows kernel-mode ZFS driver
- XorNot 9y agoThis is exciting to see. I currently have a separate NTFS disk for Windows on home machine - it would be awesome in the future to share a single mirrored zpool between Linux and Windows.
- jacobush 9y agowait what? Can you do that - do you mean with both operating systems live?
- jpgvm 9y agoI believe he means dual-boot. Thus only one would be live at a time.
- georgyo 9y agoIt is unlikely that you could install Windows on anything but NTFS. That doesn't mean you can't put your home directory on some other filesystem. Still, there is a fair amount of fragmentation in the ZFS world. It's possible that it will work fine (provided you got all the feature flags to match), but I am very skeptical that being able to mount in both will happen any time soon.
- nailer 9y ago> It is unlikely that you could install Windows on anything but NTFS. That doesn't mean you can't put your home directory on some other filesystem. Even Microsoft's own https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview https://docs.microsoft.com/en-us/windows-server/storage/refs... can't be used as a boot drive currently.
- exikyut 9y agoMost definitely no. You'd still need to go for SMB or NFS for that. If this is possible I would consider it one of the most impressive things I'd ever seen.
- Filligree 9y agoZFS supports connecting a pool to multiple computers at once, but only by way of locking the pool as a whole, and as others have pointed out it's unlikely to work even then. There's too much feature divergence. NFS is what you're "supposed" to use.
- exikyut 9y agoHah. ZFS really does have every single almost-reasonable feature baked into the design :) Problem with NFS is that it's really glitchy. For example I cannot disconnect my laptop's (higher-power) USB Wi-Fi dongle when I want to move around, because the PHY disappearing from under my NFS mount produces a kworker that consumes 100% CPU. The good news: I found a Red Hat bugzilla thread with repro info that perfectly matched what I was seeing, which identified the root cause and generated an upstream kernel patch that completely eliminated the problem. This all happened a couple years ago, for kernel 3.8 or so. The bad news: I'm using kernel 4.1.6. (I'll upgrade eventually, and see if the problem has conveniently disappeared. If it hasn't, I'll make the effort to switch to SMB. I'll also see what happens if I trigger this bug on a kernel with NFS compiled as a module at some point... >:D)
- _joel 9y agoThere's also interesting technologies that use ZFS as a base layer too such as MarFS which effectively provides network based object storage (or lustre etc) - https://www.usenix.org/system/files/login/articles/login_spring17_07_inman.pdf https://www.usenix.org/system/files/login/articles/login_spr...
- the8472 9y ago> ZFS really does have every single almost-reasonable feature baked into the design :) Except some of the nice features btrfs has. Like almost arbitrary online restriping; on-demand deduplication, defragmentation and compression of existing files; reflink copy; nocow-flag, ...
- takeda 9y agoI have feeling he just means for the data. ZFS doesn't allow two systems to use it at the same time (unless it is shared through NFS or similar) and booting Windows from ZFS would be another challenge (I bet much more difficult one). Currently there's no good FS that can work together with Windows and Linux/Unix. I mean the one that works on both would be ExFAT but its age is showing up. I currently found a workaround to have FreeNAS installed on VirtualBox and then share the volume through Samba, it works, but it is hacky. A native solution would be far better.
- exikyut 9y agoWow. Of the 26 items in the Milestone section, three (near the bottom) are currently not done. All the rest are. (The last one should read "swim". And I hope that last one definitely happens!) The first substantial commit (https://github.com/lundman/ZFSin/commit/6ea8093d48da0a0abc3b24f3e985b525052de27e https://github.com/lundman/ZFSin/commit/6ea8093d48da0a0abc3b...) includes a large pile of code. It's clear this was (wisely) built out privately to begin with. I wonder how long it took to get to the above commit point. While unrelated, thanks very much to the author for the TIL on how to do Windows kernel development! TIL about the free Windows VMs too. I've opened https://github.com/lundman/ZFSin/issues/6 https://github.com/lundman/ZFSin/issues/6 to hopefully prompt a license file to turn up.
- matharmin 9y ago> TIL about the free Windows VMs too. Not quite free, just an evaluation version that expires after a while.
- exikyut 9y agoAh. I'd middle-clicked that link but not actually looked at it before hitting Submit. :) That page says the images expire 12/1/17. So that's... two months from now? Huh. That's... 341.33MB per day in bandwidth (20GB over 60 days). Weird way to measure it (the cost of continually reconfiguring would theoretically be higher), but I had a really tiny Internet cap for a few years some time ago.
- moron4hire 9y agoWindows licenses aren't that expensive. If you need a copy of Windows more often than the trial image makes feasible, you should probably buy a copy.
- exikyut 9y agoFair point. Glances at nonexistant bank account - and computer from 2006 :P
- laumars 9y agoDoes anyone know anything about this author? Is this likely something that's been developed for a business or is it "just" a hobby project (albeit a seriously impressive hobby project!!)?
- roboyoshi 9y agoJorgen Lundman is also a main coder for the OpenZFS project afaik. I've seen his name many times since the very early betas of the OpenZFS macOS port that I've been using back in 2012/2013. I don't know how it's financed, but I'd think it's not just a hobby.
- laumars 9y agoOh that's good. Even if this turns out just to be a hobby project then at least the brains behind it is someone with real experience hacking ZFS.
- lundman 9y agoJust a hobby, working on it because I want to :)
- technomalogical 9y agoYour hobbies make my hobbies look pathetic. Kudos :D
- barrkel 9y agovdev_disk.c spawns a thread to get around that IoCompletionRoutine is called in a different context, to sleep until signalled. Is there a better way to do async in Windows? Instead of creating a thread and blocking until being signalled in the continuation of an IO, enqueue a work item to continue work at PASSIVE_LEVEL (i.e. schedule a continuation). Async means continuations, otherwise you're blocking. https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/guidelines-for-writing-dpc-routines https://docs.microsoft.com/en-us/windows-hardware/drivers/ke...
- Filligree 9y agoI'm not ZFSin's creator, and there's no guarantee they're reading this. If you want to inform him, you should probably file an issue.
- takeda 9y agoLooks like the author is here and already responded to his comments.
- exikyut 9y agoThis would be perfect to dump into a new issue
- lundman 9y agoI tried that way first, but because it called the callback at different level, and ZFS grabs a mutex, it just died. Which why I changed it to spawn a taskq to wait for the callback to come back, then tell ZFS about it. So it is async right now, I'm just not happy it has to create yet another taskq to handle it, for each IO request.
- barrkel 9y agoI'm far from expert at NT kernel programming, but I do know that there's not much you can do in the callbacks from interrupts, which is why you use a deferred procedure call (using Io/ExQueueWorkItem) to continue at a level where you can do more things, like potentially blocking calls. The work item execution of the DPC proceeds on a system worker thread, rather than a thread you had to spawn yourself. But perhaps that's what you did. If you're still stuck, I suggest you email Larry Osterman (Firstname.Lastname@microsoft.com) for a clue; Larry has worked on the NTFS driver and he's responded to my emails in the past. For an improved probability of a useful response, I'd send him a link to a Stack Overflow question, so that other people benefit from the reply also. (This is a general rule for emailing people for developer help, you'd be surprised how many random people email system / compiler developers for coding assistance.)
- Philipp__ 9y agoI was kinda bummed when Apple announced that they are going with APFS instead of ZFS. I think ZFS is one of the biggest engineering feats in last decade or so of software engineering. It still needs wider adoption.
- ohtwenty 9y agoHow do you think they could achieve wider adoption?
- kawsper 9y agoThe problem is the license, which ZFS on Linux covers a bit here: https://github.com/zfsonlinux/zfs/wiki/FAQ#licensing https://github.com/zfsonlinux/zfs/wiki/FAQ#licensing You can see some of the fun hoops you have to go through to get ZFS as your root filesystem here: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-... For now I just have 4 disks in a ZFS-raid for storage, and then I have installed my OS on a separate SSD.
- RantyDave 9y agoUbuntu has it built in and ready to go. But not for the root drive (that way lies madness).
- lscotte 9y agoIt's quite excellent for the root drive, you just have to install via debootstrap. I personally do this on all my 1/2 dozen boxes (Straight Debian usually, but some Ubuntu). I've also written up how to build an AWS AMI this way. Good stuff, It's only going to get easier.
- mst 9y ago> I've also written up how to build an AWS AMI this way If you've written it up in public, any chance of a link?
- ianai 9y agoBut why? I’m thinking for data recovery. I’m hoping people wouldn’t try to set up a ZFS fileserver within windows.
- daphreak 9y agoI would love to have an advanced file system that works on portable drives across multiple OSes. ZFS is not designed for this use case, but at least it is something. Having compression, encryption, integrity checking, and data duplication on a potentially flaky external drive would be killer features.
- ianai 9y agoThat would be awesome. I’d love to have c or root on zfs as well. That could make clients really easy to back up.
- krnlpk 9y agoAre they ever going to implement built-in encryption in this Windows ZFS driver? I can't see it in the Milestones listed by the Github project. My pools largely make use of native encryption, so this is quite useless to me without it being able to import keys and decrypt the drives.
- lundman 9y agoJust a matter of cherry-picking it in. I'll get around it it soon
- e12e 9y agoFantastic. If/when encryption stabilizes that give a solid cross-platform fs with both encryption and snapshot support. Hopefully there'll be something for Apple users as well. Would be ironic if Linux has so-so fatx support because of patents, and OS X lacks zfs because of patents..
- jlgaddis 9y agoI'm sure "they" would appreciate any help.
- netrap 9y agoIs this not an IFS driver?
- rbanffy 9y agoBut wait! WinFS (or whatever its name) is just around the corner! Sorry. Couldn't resist. Microsoft promises something better than NTFS - something "object oriented" or "database driven" - since before I moved to Windows NT 3.5.
- jellund 9y agoHe's doing as a hobby but he isn't against any support you can give him https://www.patreon.com/lundman https://www.patreon.com/lundman :)