5 ms·

Actually that is a libc issue. It only reads in 32k of directory entries at a time. If you compile your own lame ls implementation and read in say a 1 meg buff

by mitchty 12y ago

Actually that is a libc issue. It only reads in 32k of directory entries at a time. If you compile your own lame ls implementation and read in say a 1 meg buffer vs 1024 bytes at a time, you can print out directories with over 4 million entries in about a second versus 20ish hours with 1024 byte buffers (yeah I timed it). I'm not saying ext4 doesn't have issues, just that its not as simple as it seems on the surface.

otterley 12y ago
If it were a libc issue, the problem would also occur with XFS. The reason we know it's a filesystem issue is that on ext4, readdir() still takes an eternity even after we delete all the files in the subdirectory.
- jfindley 12y ago
  Yeah, because ext4 never cleans up the dentry table after it's been made. This is a pain. Unfortunately XFS also sucks for handling larger numbers of files - at the hundreds of millions or larger scale, you're largely metadata bound - and XFS is very slow at this for a number of reasons. RH did an interesting study a while back [0] on storing 1 billion files, which came out in favor of ext4. If you're able to work around the lack of dentry cleanup, I still think ext4 is probably the best choice here (ideally with the journal on a small SSD) - assuming you can't store them in a database or something more appropriate like mogileFS [1] [0]: http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf http://www.redhat.com/summit/2011/presentations/summit/decod... [1]: https://code.google.com/p/mogilefs/wiki/Start https://code.google.com/p/mogilefs/wiki/Start
  - otterley 12y ago
    Slide 32 suggests that XFS performance took the lead again after RHEL 6.2. Am I mistaken? See also https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/6.2_Release_Notes/#filesystem https://access.redhat.com/site/documentation/en-US/Red_Hat_E...
  - justin66 12y ago
    I wonder if you know about Dave Chinner's talk, which addresses the metadata performance issue. (they fixed it) https://www.youtube.com/watch?v=FegjLbCnoBw https://www.youtube.com/watch?v=FegjLbCnoBw
    - jfindley 12y ago
      I hadn't seen that - that's interesting. Thanks.
  - rbanffy 12y ago
    > Yeah, because ext4 never cleans up the dentry table after it's been made. Why is that? It doesn't seem to make much sense.
  - deleted 12y ago
    [deleted]
- mitchty 12y ago
  Ah, makes sense then. What I've seen (mind you this is mostly Suse), is that by adjusting the sys call buffer size I can get ext4/3 to reach pretty good performance. Generally though I hate having 1 million + entries in a directory. I'm kind of curious what the use case is for that kind of layout?
  - otterley 12y ago
    In our case we're storing copies of incoming email messages, one per file, into a date-stamped directory. At the beginning of the next day, we add all the previous day's message files into a single .zip file and remove the original files.
deleted 12y ago
[deleted]

by mitchty 12y ago

otterley 12y ago

If it were a libc issue, the problem would also occur with XFS. The reason we know it's a filesystem issue is that on ext4, readdir() still takes an eternity even after we delete all the files in the subdirectory.

jfindley 12y ago
Yeah, because ext4 never cleans up the dentry table after it's been made. This is a pain. Unfortunately XFS also sucks for handling larger numbers of files - at the hundreds of millions or larger scale, you're largely metadata bound - and XFS is very slow at this for a number of reasons. RH did an interesting study a while back [0] on storing 1 billion files, which came out in favor of ext4. If you're able to work around the lack of dentry cleanup, I still think ext4 is probably the best choice here (ideally with the journal on a small SSD) - assuming you can't store them in a database or something more appropriate like mogileFS [1] [0]: http://www.redhat.com/summit/2011/presentations/summit/decoding_the_code/thursday/wheeler_t_0310_billion_files_2011.pdf http://www.redhat.com/summit/2011/presentations/summit/decod... [1]: https://code.google.com/p/mogilefs/wiki/Start https://code.google.com/p/mogilefs/wiki/Start
- otterley 12y ago
  Slide 32 suggests that XFS performance took the lead again after RHEL 6.2. Am I mistaken? See also https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/6.2_Release_Notes/#filesystem https://access.redhat.com/site/documentation/en-US/Red_Hat_E...
- justin66 12y ago
  I wonder if you know about Dave Chinner's talk, which addresses the metadata performance issue. (they fixed it) https://www.youtube.com/watch?v=FegjLbCnoBw https://www.youtube.com/watch?v=FegjLbCnoBw
  - jfindley 12y ago
    I hadn't seen that - that's interesting. Thanks.
- rbanffy 12y ago
  > Yeah, because ext4 never cleans up the dentry table after it's been made. Why is that? It doesn't seem to make much sense.
- deleted 12y ago
  [deleted]
mitchty 12y ago
Ah, makes sense then. What I've seen (mind you this is mostly Suse), is that by adjusting the sys call buffer size I can get ext4/3 to reach pretty good performance. Generally though I hate having 1 million + entries in a directory. I'm kind of curious what the use case is for that kind of layout?
- otterley 12y ago
  In our case we're storing copies of incoming email messages, one per file, into a date-stamped directory. At the beginning of the next day, we add all the previous day's message files into a single .zip file and remove the original files.

deleted 12y ago

[deleted]