7 ms·
ext4 seems always to be the best.. no?
by jeffbe 11y ago
ext4 seems always to be the best.. no?
- dijit 11y agountil you get to many small files (multiple terabytes) or very large volumes. then it's metadata overheads consume more I/O than the data itself.
- chubot 11y agoCan you elaborate on this or provide a pointer? I'm storing a lot of small files on an ext4 drive (on a 48 GB Linode). Right now it's 5M files, and they take up 40G or so. I formatted it to use at least twice as many inodes as the default (otherwise I would have run out), and then lowered the block size to 2K rather than 4K, which reduced a ton of wasted space because the files are so small. That's not terabytes, but it's definitely a lot of small files for the size. I probably will end up using 50M or so files up from 5M (on a bigger drive). So is ext4 less inefficient for this use case than other file systems? Why? I hear you can turn on options like dir_index but I haven't tried it yet.
- nadams 11y ago> Can you elaborate on this or provide a pointer? In my experience ext4 becomes very unhappy/slow when you have many files/folders in the same folder (I noticed this with backuppc). dir_index may have helped me with this.
- dijit 11y agoHappy to. :) Talk by dchinner @ Redhat https://www.youtube.com/watch?v=FegjLbCnoBw https://www.youtube.com/watch?v=FegjLbCnoBw
- pwg 11y agoDo you have a reason for keeping them stored as lots of very small files in a filesystem? If not, you might take a look at this page on the Sqlite home site: https://www.sqlite.org/intern-v-extern-blob.html https://www.sqlite.org/intern-v-extern-blob.html and consider storing the small items within a sqlite database file instead.
- chubot 11y agoThanks for the pointer. I actually am using sqlite for the metadata, but not the data. I was thinking that sqlite wouldn't be happy with 40+ GB of files, though I haven't tested it. This page seems to say it's doable with some tuning [1]. (Although as mentioned I may go up to 500 GB or more later) It's nice to see that intuitive performance characteristic actually tested -- small files perform better when stored internally, but large files perform better when stored on the file system. I want to use rsync to sync files between machines, so that's a consideration. Otherwise I would have to roll my own. With sqlite you'll also get duplication of the OS buffer cache in user space. Not sure how much of an issue that is in practice. I like sqlite a lot, but it seems to have more nontrivial data structures than a file system, so it's harder to reason about performance and space overhead. It also has more concurrency issues than a file system. One thing I like about the file system is that I was able to account for the waste within 1%. I wrote a simple script to compare the waste with 2K ext4 block size vs 4K block size, and it predicted 7 GB of saved space (out of 40 GB or so), and that turned out to be exactly what happened in practice. I don't think I'd be able to account for space overheads so exactly in sqlite (in theory or in practice). The other dimension is IOPS as mentioned here, but I think that will be easier with just a file system too. There are more observability and debugging tools with file systems. It's possible that because the inode issue, I could go for storing small files internally and big files externally though. The distribution of sizes is pretty much Zipfian. That isn't too hard to code, and the complexity may be justified. [1] http://stackoverflow.com/questions/784173/what-are-the-performance-characteristics-of-sqlite-with-very-large-database-file http://stackoverflow.com/questions/784173/what-are-the-perfo...
- chubot 11y agoActually I just checked, and the bottom 1.91M files out of 5M are less than 1024 bytes, and sums to only 1 GB ! The other 3.1 M files sum to 34 G (largest file being 2GB or so). The distribution was more skewed than I thought. So I think the hybrid solution with internal sqlite for small files may be a good idea. Thanks!
- hrez 11y agoNote that since linux 3.8 [1] ext4 can embed small files into inodes thus using less space and allow faster reads. Using that feature and adjusting inode size you can manipulate which file sizes are inlined. [1] http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd0114ae97f889f67ef46a http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd...
- chubot 11y agoOK interesting, so this capability is on by default then? I didn't change the default inode size, but it looks like I could SAVE space by increasing it (from 256 or something). Since I have 1.91M files less than 2048 bytes, I guess I could increase it enough to store most/many of those. And I guess it halves the IOPS for serving those files, since with a cold cache you need to read block holding the inode and block holding the data.
- TillE 11y agoext4 is comparable to NTFS or HFS+. It mostly works, but other filesystems like btrfs or ZFS give you proper data integrity (checksumming) and other features.
- 0x0 11y agoIsn't btrfs still plagued with stability issues/bugs though? (and zfs isn't readily available in default kernels?) Reading planet.debian.org I seem to get the impression that btrfs it's not 100% there yet, unless you feel a little adventurous.
- noja 11y agoI've been running raid-1 btrfs on workstations and a server for a while, haven't seen any issues at all. I did have an issue on a btrfs laptop, it didn't cope with on/off all that well, but that was more than a year ago, maybe I will retest.
- moviuro 11y agoBtrFS will stop being "unstable" when development is done, in 20 years (much like ext*). I run BtrFS on my 2 laptops (one with SSD, one with HDD): the only issue I ran into is metadata space exhaustion: I have well over 100 snapshots and suddenly, the drive becomes unwritable. Once you know that can happen, you're more careful about backups and when to remove the old snapshots. Also, Debian might not be the best place for bleeding-edge stuff like BtrFS: you should consider a distribution with a more recent version of kernel and utils (archlinux, gentoo...)
- joonoro 11y agoDepends on the use case, but ext4 is a pretty good default. I like using IBM's JFS on my laptops to prolong battery life since it minimizes CPU usage [0][1][2]. [0] http://fsbench.netnation.com/ http://fsbench.netnation.com/ [1] https://www.debian-administration.org/article/388/Filesystems_ext3_reiser_xfs_jfs_comparison_on_Debian_Etch https://www.debian-administration.org/article/388/Filesystem... [2] http://linuxgazette.net/122/TWDT.html#piszcz http://linuxgazette.net/122/TWDT.html#piszcz
- pantalaimon 11y agothose benchmarks are 10+ years old - are you sure they are still valid?
- ck2 11y agoEXT4 is arguably the most mature but even recently they found a long term bug in EXT4 on RAID that no-one noticed for years https://www.phoronix.com/scan.php?page=news_item&px=Linux-4-EXT4-RAID-Issue-Found https://www.phoronix.com/scan.php?page=news_item&px=Linux-4-...
- __david__ 11y agoActually it looks like they found a bug in md (RAID) that was causing fs corruption. To my reading it looks like ext4 was just incidental.
- laumars 11y agoext4 has it's place - and deservedly so. But picking the right file system really depends on your requirements. For example if you're building a storage server then ZFS would be a better fit. XFS is also worth taking notice of since the benchmarks I've read rate it for having faster read and write speeds than ext4. However I don't have extensive first hand experience running XFS (something which I'm currently addressing).
- __david__ 11y agoI'm running XFS on a couple of my disks. It seems quick and reliable so far. My only complaint is that you can expand a volume (online, too), but you can't shrink it. Shrinking volumes comes in handy occasionally, and when you need it, you need it. But so far it hasn't bitten me yet.
- wyldfire 11y agoXFS is hands down the best at sustained write speeds. Each time I've stacked it up against the competition it wins.
- wazoox 11y agoI've set up several hundred storage servers with XFS (up to 250 TB per FS) in the past 15 years. There were sour times, 10 years ago or more, but in recent times it's been rock solid and beat the crap out of other FS. Notice that XFS always hated crappy hardware, because it pushes it to the limits.
- nickpsecurity 11y agoUnsurprising given it was built by the same people who built these: https://www.sgi.co.jp/features/2001/dec/fleet_numerical/images/lg_origin.jpg https://www.sgi.co.jp/features/2001/dec/fleet_numerical/imag... And more recently this: https://www.sgi.com/pdfs/4555.pdf https://www.sgi.com/pdfs/4555.pdf Because pushing the envelope of HPC awesomeness was just another day in the office for folks at SGI. :)
- mtanski 11y agoext4 is really good at the single user (single process / single thread) use case. Other file systems shine in different categories like number of features (btrfs) snapshots (btrfs) very large devices (xfs) really high concurrency (xfs).
- otterley 11y agoext4 is usually fine, but it has really bad corner case behavior with respect to large directories (> 10k dentries). Iterating through them becomes increasingly slow as they grow larger -- worse than O(n), which is the expected case. What's worse is that even if you then delete all the files in the directory, performance does not return to normal; iterating becomes just as slow at that point with 10 files. The only solution is to remove the directory and recreate it. We've been running XFS on our 1000+ hosts for 4 years without issue. There are old stories of XFS issues, but it's been rock solid for quite some time, at least since RHEL 5.