10 ms·
Ntfs2btrfs does in-place conversion of NTFS filesystem to the open-source Btrfs
- johnea 2y ago[flagged]
- snapplebobapple 2y agobtrfs isn't that terrible for desktop use right now. I mean, I wouldn't personally use it, I lost data on it a couple times four plus years ago, but it's come a long way since then. (my preference is keep everything I care about on a fileserver like truenas running zfs with proper snapshotting, replication and backup and live dangerously on the desktop testing out bcachefs, but I recognize not everyone can live my life and some people just want a laptop with a reasonable filesystem resistant to bit rot.
- jeroenhd 2y agoI recently found out Fedora defaults to btrfs with zstd compression enabled by default. Seems to work well enough. On my personal devices I prefer BTRFS' snapshotting ability over the risk of having to restore from backup at some point.
- emacsen 2y agoIs there decent encryption support, or are we stuck using full disk encryption at the block level?
- RussianCow 2y agoThe latter at this point.
- jeroenhd 2y agoI don't know, I just use FDE because I don't trust filesystem level encryption to protect against the many side channel attacks one can inject into a running system. LUKS is good enough for me. It looks like there's slow but steady progress on FSCRYPT support on BTRFS: https://bkhome.org/news/202403/linux-kernel-btrfs-supports-fscrypt.html https://bkhome.org/news/202403/linux-kernel-btrfs-supports-f... No idea what kind of state it is in currently.
- arjvik 2y agoHow is bcachefs for personal use these days?
- chasil 2y agoThe political situation for bcachefs is far from good, with pressure from Linus and a CoC violation. The net effect will likely delay stability. https://www.phoronix.com/news/Bcachefs-Fixes-Two-Choices https://www.phoronix.com/news/Bcachefs-Fixes-Two-Choices https://www.phoronix.com/news/Linux-CoC-Bcachefs-6.13 https://www.phoronix.com/news/Linux-CoC-Bcachefs-6.13
- koverstreet 2y agoHonestly the political situation will probably be a /good/ thing for long term stability, because I get a few months without any stupid arguments with upstream and finally get to write code in peace :) It sucks for users though, because now you have to get my tree if you want the latest fixes, and there's some stuff that should be backported for forwards compatibility with the scalability improvements coming soon [1]. [1]: https://www.patreon.com/posts/more-expensive-116975457 https://www.patreon.com/posts/more-expensive-116975457
- chasil 2y agoI'm hoping to use your filesystem when it's ready. Everyone wishes that this were easier for you.
- mappu 2y agoEchoing the sibling comment Kent, bcachefs is a really wonderful and important project. The whole world wants your filesystem to become the de-facto standard Linux filesystem for the next decade. One more month of LKML drama is a small price for that (at LKML prices).
- homebrewer 2y agoDon't worry about the users, we'll manage somehow, it's such a tiny burden compared to the actual development. I'm just really happy to see you're not discouraged by the petty political mess and keep pushing through. Thank you!
- tombert 2y agoAfter a bit of peer pressure from a friend, I decided ended up using btrfs with my laptop about three months ago. It’s been fine thus far.
- aucisson_masque 2y agoEmphasize on ‘this far’… To me that kind of experiment is the equivalent of changing your car brake for something ‘open source’, maybe better, maybe not. But when you need them you’re going to want to make sure they’re working.
- kccqzy 2y agoHow much of an assurance do you want? I've been using btrfs on my desktop for 3+ years. And I've also experienced no issues thus far.
- kalleboo 2y agoThe btrfs volume on my Synology NAS is 6 years old now, with 24/7 operation (including hosting web sites with constant read/write database activity) and going through several volume resize operations. No issues.
- fragmede 2y agohonestly I think btrfs isn't bloated enough for today's VM-enabled world. ext4 and xfs and hell, exfat haven't gone anywhere, and if those fulfill your needs, just use those. but if you need more advanced features that btrfs or zfs bring, those added features are quite welcome. imo, btrfs could use the benefits of being a cluster filesystem on top of everything it already does because having a VM be able to access a disk that is currently mounted by the host or another VM would useful. imagine if the disk exported to the VM could be mounted by another VM, either locally or remote simultaneously. arguably ceph fills this need, but having a btrfs-native solution for that would be useful.
- ChocolateGod 2y agoRunning VMs (and database servers) on btrfs performs really bad so you have to disable CoW for them. Otherwise you'll get situations where your 100GB VM image will use over a TB of physical disk space. It's a shame really that this still isn't solved.
- Joe_Cool 2y agoI don't think thin provisioning btrfs makes a lot of sense. Before disabling CoW I'd rather use a different filesystem. Are you sure your TRIM is working and the VM disk image is compacting properly? It's working for me but not really great for fragmentation.
- ComputerGuru 2y agoWhat’s the underlying issue? I used VMs with ZFS for storage for well over a decade with no issue.
- dmm 2y agoZFS performs much better than btrfs with the many small writes that VMs produce. Why exactly is a great question. Maybe it has to do with the optimizations around the ZIL, the temporary area where sync writes are accumulated before they are written to the long-term spot.
- 2y ago
- do_not_redeem 2y agoWhat filesystem would you suggest that has data checksums, efficient snapshots, and doesn't require compiling an out of tree kernel module?
- freeone3000 2y agozfs is licensed too freely to be in-tree, but it’s still an excellent choice.
- do_not_redeem 2y agoYeah the situation is unfortunate. There's a decent chance I'd be using ZFS if not for the licensing issues, but as a practical matter I'm getting too old to be futzing with kernel modules on my daily driver.
- telgareith 2y agoDKMS solved these "licensing issues." Dell is mum on official motivation- but it provides a licensing demarcation point, and a way for kernels to update without breaking modules- so it's easier for companies to develop for Linux. _Windows Drivers work the same way and nobody huffs and puffs about that_ I'd love to have an intelligent discussion on how one person's opinion on licensing issues stacks up against the legal teams of half the fortune 50's. Licensing doesn't work on "well, I didn't mean it THAT way."
- do_not_redeem 2y agoI admit I'm not fully up to date on whether it's actually "license issues" or something else. I'm not a lawyer. As a layman here's what I know. I go to the Arch wiki (https://wiki.archlinux.org/title/ZFS https://wiki.archlinux.org/title/ZFS) and I see this warning under the DKMS section (as you advised): > Warning: Occasionally, the dkms package might not compile against the newest kernel packages in Arch. Using the linux-lts kernel may provide better compatibility with out-of-tree kernel modules, otherwise zfs-dkms-staging-gitAUR backports compatibility patches and fixes for the latest kernel package in Arch on top of the stable zfs branch So... my system might fail to boot after updates. If I use linux-lts, it might break less often. Or I can use zfs-dkms-staging-git, and my system might break even less often... or more often, because it looks like that's installing kernel modules directly from the master branch of some repo. As a practical matter I could care less if my system fails to boot because of "license issues" or some other reason, I just want the lawyers to sort their shit out so I don't have to risk my system becoming unbootable at some random inopportune time. Until then, I've never hit a btrfs bug, so I'm going to keep on using it for every new build.
- dang 2y ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- the_hoser 2y agoThe degree of hold-my-beer here is off the charts.
- koverstreet 2y agoIt's not quite as dangerous as you'd think. The standard technique is to reserve a big file on the old filesystem for the new filesystem metadata, and then walk all files on the old filesystem and use fiemap() to create new extents that point to the existing data - only writing to the space you reserved. You only overwrite the superblock at the very end, and you can verify that the old and new filesystems have the same contents before you do.
- jeroenhd 2y agoI believe that is also the method [btrfs-convert](https://btrfs.readthedocs.io/en/latest/Convert.html https://btrfs.readthedocs.io/en/latest/Convert.html) uses. A cool trick that tool uses is to keep the ext4 structures on disk (as a subvolume), which allows reverting to ext4 if the conversion didn't go as planned (as long as you don't do anything to mess with the ext4 extents, such as defragmenting or balancing the filesystem, and you can't revert after deleting the subvolume of course).
- Joe_Cool 2y agoI believe you are right. You can only convert back to the metadata from before. So any new or changed (different extents) files will be lost or corrupted. So best to only mount ro when considering to rollback. Otherwise it's pretty risky.
- heftig 2y agoNo, it also covers the data. As long as you don't delete the rollback subvolume, all the original data should still be there, uncorrupted. Even if you disable copy-on-write, as long as the rollback subvolume is there to lay claim to the old data, it's considered immutable and any modification will still have to copy it.
- Dwedit 2y agoI would be very surprised if it supported files that are under LZX compression. (Not to be confused with Windows 2000-era file compression, this is something you need to activate with "compact.exe /C /EXE:LZX (filename)")
- ruined 2y agoit seems to contain code that handles LZX, among other formats https://github.com/search?q=repo%3Amaharmstone%2Fntfs2btrfs%20lzx&type=code https://github.com/search?q=repo%3Amaharmstone%2Fntfs2btrfs%...
- cryptonector 2y agoThinking of how I'd do this for ZFS... I think I'd do something like: add a layer that can read other filesystem types and synthesize ZFS block pointers, then ZFS could read other filesystems, and as it writes it could rewrite the whole thing slowly. If ZFS had block pointer rewrite (and I've explained here before why it does not and cannot have BP rewrite caoabilities, not being a proper CAS filesystem), one could just make it rewrite the whole thing to finish the conversion.
- deleted 2y ago[deleted]
- rurban 2y agoWhy would someone do that? NTFS is stable, faster than btrfs and has all the same features.
- rnd0 2y agoThe only reason I can think of is so that they can use the same FS in both windows and linux -but with ntfs, they already can. Mind you, with openzfs (https://openzfsonwindows.org/ https://openzfsonwindows.org/) you get windows (flakey), freebsd, netbsd and linux but -as I said; I'm not sure zfs is super reliable on windows at this point. Mind you, I just stick with ntfs -linux can see it, windows can see it and if there's extra features btrfs provides they're not ones I am missing.
- mistaken 2y agoWith ntfs you have to create a separate partition though. With btrfs you could create a subvolume and just have one big partition for both linux and windows.
- ComputerGuru 2y agoI’m a die-hard ZFS fan and heavy user since the Solaris days (and counting) but I believe the WinBtrfs project is in better (more useable) shape than the OpenZFS for Windows project.
- paines 2y agowhat ?!?! NTFS has no case sensitivity no compression. And I guess a couple of more things I do not want to miss.
- casey2 2y agoVery cool, but nobody will hear about this until at least a week after they format their ntfs drives that they have been putting off formatting for 2 years
- deleted 2y ago[deleted]
- npn 2y agoI tried this one before, resulted in a read-only disk. Hope it improves since then.
- quasarj 2y agoThat's 50% better than losing all your data!
- tronicdude 2y ago[flagged]
- mort96 2y ago"Endless errors"? Are you talking about disk errors? Or are you referring to that one time long ago that it had a bug wrt a specific uncommon RAID setup? APFS and ZFS aren't very interesting to me honestly, because neither are, or can be, in the Linux kernel. I also don't understand why APFS is in the same conversation as ZFS and BTRFS.
- deleted 2y ago[deleted]
- oDot 2y agoIs anyone here using BTRFS and can comment on its current-day stability? I used to read horror stories about it
- quotemstr 2y agoI've used it for my personal machines and backups (via btrbk) for years without any issues
- einsteinx2 2y agoI’ve been using it for a few years on my NAS for the all the data drives (with Snapraid for parity and data validation), and as the boot drive on a few SBCs that run various services. Also use it as the boot drive for my Linux desktop PC. So far no problems at all and I make heavy use of snapshots, I have also had various things like power outages that have shut down the various machines multiple times. I’ve never used BTRFS raid so can’t speak to that, but in my personal experience I’ve found BTRFS and the snapshot system to be reliable. Seems like most (all?) stories I hear about corruption and other problems are all from years ago when it was less stable (years before I started using it). Or maybe I just got lucky ¯\_(ツ)_/¯
- AtlasBarfed 2y agoWhy use BTRfS raid rather than good old MDadm raid?
- einsteinx2 2y agoI don’t use BTRFS raid, I don’t actually use any RAID. I use SnapRAID which is really more of a parity system than real RAID. I have a bunch of data disks that are formatted BTRFS, then 2 parity disks formatted using ext4 since they don’t require any BTRFS features. Then I use snapraid-btrfs which is a wrapper around SnapRAID to automatically generate BTRFS snapshots on the data disks when doing a SnapRAID sync. Since the parity is file based, it’s best to use it with snapshots, so that’s the solution I went with. I’m sure you could also use LVM snapshots with ext4 or ZFS snapshots, but BTRFS with SnapRAID is well supported and I like how BTRFS snapshots/subvolumes works so I went with that. Also BTRFS has some nice features over ext4 like CoW and checksumming. I considered regular RAID but I don’t need the bandwidth increase over single disks and I didn’t ever want the chance of losing a whole RAID pool. With my SnapRAID setup I can lose any 2 drives and not lose any data, and if I lose 3 drives, I only lose the data on any lost data drives, not all the data. Also it’s easy to add a single drive at a time as I need more space. That was my thought process when choosing it anyway and it’s worked for my use case (I don’t need much IOPS or bandwidth, just lots of cheap fairly resilient and easy to expand storage).
- ComputerGuru 2y agoI found the link to Quibble, an open and extensible reverse engineering of the Windows kernel bootloader to be much more intriguing: https://github.com/maharmstone/quibble https://github.com/maharmstone/quibble
- SomeoneOnTheWeb 2y agoI would have needed that like 2 months ago, when I had to format a hard drive with more than 10TB of data into from NTFS... ^^ Nic project!