4 ms·
ext4 has some pretty sharp edge cases in terms of performance degradation. For example, readdir() performance is unacceptable in directories that contain lots
by otterley 12y ago
ext4 has some pretty sharp edge cases in terms of performance degradation. For example, readdir() performance is unacceptable in directories that contain lots of dentries (on the order of 100,000s) -- or that ever once did. (Directory read performance was just as bad even with 5 dentries after deleting the rest.) We had to switch to XFS for just that reason.
- mitchty 12y agoActually that is a libc issue. It only reads in 32k of directory entries at a time. If you compile your own lame ls implementation and read in say a 1 meg buffer vs 1024 bytes at a time, you can print out directories with over 4 million entries in about a second versus 20ish hours with 1024 byte buffers (yeah I timed it). I'm not saying ext4 doesn't have issues, just that its not as simple as it seems on the surface.
- otterley 12y agoIf it were a libc issue, the problem would also occur with XFS. The reason we know it's a filesystem issue is that on ext4, readdir() still takes an eternity even after we delete all the files in the subdirectory.
- jfindley 12y agoYeah, because ext4 never cleans up the dentry table after it's been made. This is a pain. Unfortunately XFS also sucks for handling larger numbers of files - at the hundreds of millions or larger scale, you're largely metadata bound - and XFS is very slow at this for a number of reasons. RH did an interesting study a while back [0] on storing 1 billion files, which came out in favor of ext4. If you're able to work around the lack of dentry cleanup, I still think ext4 is probably the best choice here (ideally with the journal on a small SSD) - assuming you can't store them in a database or something more appropriate like mogileFS [1] [0]: http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf http://www.redhat.com/summit/2011/presentations/summit/decod... [1]: https://code.google.com/p/mogilefs/wiki/Start https://code.google.com/p/mogilefs/wiki/Start
- otterley 12y agoSlide 32 suggests that XFS performance took the lead again after RHEL 6.2. Am I mistaken? See also https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/6.2_Release_Notes/#filesystem https://access.redhat.com/site/documentation/en-US/Red_Hat_E...
- justin66 12y agoI wonder if you know about Dave Chinner's talk, which addresses the metadata performance issue. (they fixed it) https://www.youtube.com/watch?v=FegjLbCnoBw https://www.youtube.com/watch?v=FegjLbCnoBw
- jfindley 12y agoI hadn't seen that - that's interesting. Thanks.
- rbanffy 12y ago> Yeah, because ext4 never cleans up the dentry table after it's been made. Why is that? It doesn't seem to make much sense.
- deleted 12y ago[deleted]
- mitchty 12y agoAh, makes sense then. What I've seen (mind you this is mostly Suse), is that by adjusting the sys call buffer size I can get ext4/3 to reach pretty good performance. Generally though I hate having 1 million + entries in a directory. I'm kind of curious what the use case is for that kind of layout?
- otterley 12y agoIn our case we're storing copies of incoming email messages, one per file, into a date-stamped directory. At the beginning of the next day, we add all the previous day's message files into a single .zip file and remove the original files.
- deleted 12y ago[deleted]
- kbenson 12y agoI thought there was some ext4 specific mount option you could set to tell it to speed that up significantly? I can't seem to find it now. In any case, the classic workaround (which can be a PITA) is to create a tree of subfolders and distribute files between them. E.g. first letter of filename, second letter of filename, or for more even distribution use first characters from hashing the filename.
- js2 12y agodir_index. Funny this is being concurrently discussed on the "Tell HN: The site was offline. What changed?" thread. https://news.ycombinator.com/item?id=7872306 https://news.ycombinator.com/item?id=7872306
- otterley 12y agodir_index is on by default for ext4 filesystems. It doesn't help in this case.