3 ms·
So does anyone know of a way to take a set of files and write them to an HDD, in NTFS or exFAT format, in a single sequential write? Essentially building the FS
by quasarj 8y ago
So does anyone know of a way to take a set of files and write them to an HDD, in NTFS or exFAT format, in a single sequential write? Essentially building the FS on-the-fly (because we're talking about datsets that are much too large to fit into memory)?
- Dylan16807 8y agoSo basically a zip file? I will note that you could easily build the MFT before you start transferring data. That's really your active 'dataset' here, and it's not very big.
- quasarj 8y agoBuilding the MFT first is pretty much what I want to do. But I'm not aware of any utilities that can handle it, nor where to start with writing it myself... I have a project where I routinely need to copy large amounts of data (3 to 8 TB) to a hard drive. Problem is, my files are all 512kb. So this is much slower than it could be... If I write it as a single tar file I get excellent throughput, but the users who need to be able to work with the drive are unable to handle a tar file. They need to be able to plug the drive into a Windows computer and have it "just work".. which presents some problems.
- AgentME 8y agoMaybe you could use the same type of filesystem that file CD/DVDs use (UDF?). They're written sequentially and are commonly supported.
- proverbialbunny 8y agoUsing dd. http://man7.org/linux/man-pages/man1/dd.1.html http://man7.org/linux/man-pages/man1/dd.1.html
- Dylan16807 8y agoThat's how you copy an existing filesytem. It's not how you take an arbitrary subset of files and put them into a new filesystem with minimal write amplification.
- deleted 8y ago[deleted]
- rini17 8y agoROMFS (there are several variants in linux kernel) are for this use case,but these are only read-only.