16 ms·

No Space Left on Device

66 points by morecoffee 10y ago

nikbackm 10y ago
Huh, max 32000 files in a directory seems awfully low. Even crusty old NTFS is supposed to support 4294967295 files. Explorer will not be too happy displaying such a folder though!
- RyJones 10y ago
  as a practical matter, NTFS used to choke at around a million files in a single directory. This practical limit may be higher now.
- fnj 10y ago
  ZFS, which is known for having crazy high limits, will fit somewhat over 280 trillion entries in a single directory. Curiously, though, you can have over 18 million trillion file systems in a single storage pool.
Neil44 10y ago
I thought this changed from ext3 to ext4? I know I've run into this issue before and moving to ext4 was the easiest way to resolve it. Ext4 is much quicker at doing operations on very large directories too.
JonRB 10y ago
During my young naïve programming days (which coming to think of it haven't really ended) I thought to myself "Why bother using some third party database when I could just store user information in text files, the OS handles that sort of thing really well, right?". I looked into it as much as I could be bothered and found no huge red flags. After a few years of this server merrily creating a new file for each user I found that my disk was full, as this poster did. After a lot of hunting around, I found out that I had hit my inode limit, a concept I was completely unaware of. I know for sure I won't be making THAT mistake again.
- bArray 10y ago
  "After a lot of hunting around, I found out that I had hit my inode limit, a concept I was completely unaware of." What was the number you hit? I have a flat file system for something I'm prototyping. The files are in a database with a random SHA256 filename associated with them. I was thinking about doing what Git does with creating folders for parts of the string to alleviate the number of files in one single directory. I'm sure that makes file retrieval a prettier process. Do all the files for have to be in the same directory for the iNode limit to be reached or just there in general? NOTE: I ask because the prototype isn't ever meant to go into production, but my customer keeps getting confused and I'm concerned this very important system may inherit (or even use) my design.
  - LukeShu 10y ago
    It's a global limit per filesystem. You can check what the limits are for your filesysems, and what the current use is with `df -i`. The thing that git does is for speed.
    - bArray 10y ago
      I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressable space for those nodes. I knew Git did it for speed, but I was wondering whether it was also about the number of files that can exist in a single directory. EDIT: Corrected percentage.
      - LukeShu 10y ago
        It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).
Animats 10y ago
Classic UNIX problem. Not enough error codes, and too many errors map to the same thing.
rshm 10y ago
No date on the post. Probably old and fixed now. Just checked with a random folder, it happily supported 782292 files.
- Hnrobert42 10y ago
  I think his point still stands. Basically, if you do something far enough outside the norm, you may hit really time consuming bugs.
  - bArray 10y ago
    Although the exception would probably be rare if it used a large enough number, do you think the "copying utility" (whatever it may be) should handle those extreme edge cases? Or is it just "disk full" that's used as a fall-back?
- ryanbertrand 10y ago
  There is mention of 2016. "The lesson here is that even in 2016, filesystems still are finicky with large numbers of files in a single directory."
- Xylakant 10y ago
  Since the root cause is a hash collision in the dir_index code you might just have been lucky and the author of the article unlucky.
  - caf 10y ago
    They would have had to have been exceptionally lucky - the chance of getting no collisions in a 32 bit hash among 782292 inputs is about 10^-62.
  - rshm 10y ago
    Probably lucky. File names are md5 of the content, i am not sure if it helped on collision on 32 bit md4. The linked post mentions dir_index default and tune2fs -l lists the dir_index under Filesystem features only. Not knowing this default behavior, i was considering enabling dir_index explicitly to aid chocking rsync. EDIT: I ran the reproduction code with 200K instead of 100K just to be sure. There was no problem creating file as well as copying all the files to a different folder. The disk was formatted about 3 months ago and mounted with default options "ext4 noatime,barrier=0 0 0" . On original as well as copied folder `ls | wc -l` still gives 200000.
vacri 10y ago
"If you're working with huge files or a huge number of files, use XFS" went the slogan. I was hit by this recently - with our 500ish (!!) entries in /node_modules and their dependencies of dependencies of dependencies, a standard 8GB VM formatted with ext4 just simply didn't have enough inodes for us to have two copies of our app (deployment swapover). Had to increase the size to 20GB, just to get the extra inodes. XFS would have handled it much better (from testing on a parallel machine). It was inode exhaustion, though the 'disk full' error message was the same - confusing at first, as df showed 40% free space...
- fnj 10y ago
  You could have also tuned ext4 when you created it with mke2fs. You can specify with -N the max # of inodes desired. Granted, it's not something very many people consider when creating their filesystems; for one thing it's hard to predict how many you will need.
  - guroot 10y ago
    Also most distro installers don't make it very easy to do this.
  - watt 10y ago
    It would be great if it had switch that simply says "double the default number of inodes". Per stories in comments, seems that on large file systems it's easier to run out of inodes, the default must be too low.
  - vacri 10y ago
    Unfortunately it was on a pre-baked image on AWS, and you apparently can't retrofit it.
anilgulecha 10y ago
A good filesystem to use as pseudo-db is ZFS. It's designed to have practically limitless number of files, and it's snapshotting/cloning come in really handy for things like backups etc. Also you get full checksumming.
- fnj 10y ago
  ZFS does have a hard limit of about 280 trillion files in a single directory. It's actually just about the most confining limit it does have.
  - anilgulecha 10y ago
    Correct. Hence, practically limitless.
    - bbcbasic 10y ago
      No one will ever need in 280 trillion files. Ever.
      - skykooler 10y ago
        "So, we tried to use ZFS to back up the entire Internet, and we're running into a weird error..."
      - majewsky 10y ago
        And 640k RAM ought to be enough for everyone. ;)
      - liotier 10y ago
        "No one will ever need in 280 trillion files. Ever." ~bbcbasic on Hacker News, September 2016 Found in someone's email .sig in the year 2046...
        bbcbasic 10y ago
        That typo is immortalised :-o
jobigoud 10y ago
I came to the same conclusion about sharding files from the get go. I had a similar experience where my host wouldn't accept to synchronize a directory with more than 5000 files in it. (Software producing the files is my own note-taking application where each page is stored as an individual file to speed up backup of only the pages that changes).
- vog 10y ago
  Wouldn't 5000 files of note-taking easily fit into RAM? Why bother dealing with thousands of small files? Also, rsync does quite a good job with intransferring multi-MB (or even larger) files efficiently while copying only what's actually changed.
  - jobigoud 10y ago
    I feel more secure by dealing with many individual files. For example in case of a conflict during sync, I can visually diff between the two pages very easily. This happens once in a while. Also for example if more than 5% of files have changed, the sync is stopped and asks for manual confirmation to prevent catastrophies.
cyberferret 10y ago
Brings back memories of the old DOS days, when there was quite a low limit on the number of files that you could store in the root directory. I can't remember the exact numbers, but it lead to my early habits of creating a new sub directory for every little thing so nowadays (over 3 decades later) nearly everything I need is at least 5 levels deep! ;) NB: My current working folders are now in Google Drive, but is essentially the same folder structure as I started on our Novell server about 25 years ago...
- jaredsohn 10y ago
  > when there was quite a low limit on the number of files that you could store in the root directory. I can't remember the exact numbers 512? https://en.wikipedia.org/wiki/File_Allocation_Table#Final_FAT16 https://en.wikipedia.org/wiki/File_Allocation_Table#Final_FA... (see the last paragraph.)
  - cyberferret 10y ago
    That sounds about right. It was a ridiculously low number, and even back in the days of 10MB hard drives, it was easy to bust that number with just batch and text files...
- pjc50 10y ago
  .. aided by CHKDSK, which would find orphaned allocation units and create files for them in the root directory. Could quite easily fill up your root directory on its own.
  - cyberferret 10y ago
    YES! That's right... all those FILEnnnn.CHK files exploding all over your root directory! Ah, those were the days!
guroot 10y ago
Things like this is why I use XFS anywhere I'm not using btrfs, or ZFS.
- lake99 10y ago
  I've had some bad experience with btrfs. When I had to hard-reset a hung computer, some directories became unremovable and unrenameable. All else failed, and I attempted to --repair, it did so much damage that I had to reformat the disk. This was probably about 3-4 years ago. Have you had such issues on your preferred filesystems in the last couple of years? The reason I stay with ext4 is that it has proven amazingly robust in the event of hard reboots.
  - guroot 10y ago
    In my experience XFS is rock solid. We use it almost everywhere at work. I've never had an issue with hard reboots with it. Only recently started playing with btrfs, so can't speak to it's reliability from personal experience.
    - lake99 10y ago
      I hard-reboot about once a week, caused by my carelessness and running out of 16GB of RAM. I always recommend against btrfs for system partitions, and if you don't make frequent backups, against user partitions too. Read through "man btrfsck", that should caution you sufficiently. I'll try out XFS on an external HDD I'll buy soon.
      - guroot 10y ago
        You really shouldn't need to be hard rebooting just because you ran out of ram... did you try a sysrq f (invokes the out of memory killer) first? If that doesn't work you should at least be able to force a sync and remount read-only before taking it down. If you have access to a keyboard and the kernel is still responding to sysrq you should never just hard power off without at least trying to regain control of the system. Some reading if you're not familiar with sysrq: https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key Depending on your distro you might have to enable it, I believe ubuntu (for some dumb reason) ships with emergency sync as the only command available. Enabling it should amount to adding: kernel.sysrq=1 to /etc/sysctl.conf on older machines or editing /etc/sysctl.d/10-magic-sysrq.conf on newer ones. also if you are running out of ram that often you might want to consider a swap partition. Or if you have one already try setting a more aggressive swapiness value.
Nux 10y ago
Yes, this "silly" kind of thing bit us a few times as well. It's good RHEL/CentOS 7 defaults to XFS now which doesn't suffer (as much) from this problem. https://gagor.pl/2013/11/inodes-on-xfs/ https://gagor.pl/2013/11/inodes-on-xfs/
ajuc 10y ago
I've had similar error in production. We had linux server with C++ program doing many things, and in the process it created temporary files for a barcode printer. The files were named uniquely, the server had lots of free space, we left them there to allow easier debugging if sth wrong happened (it happened a few times each year - we created the labels to print by glueing escape codes and sometimes there were labels that didn't worked for one reason or another), and only cleaned the old files when there was not much space left. At some point there was a bug when sometimes users got "wrong permissions or no space left" errors (sorry don't remember the exact error and IIRC our app just tried to save a file and failed with unspecific error if it couldn't write the new temporary file, so we haven't had the exact system error). What was weird is - there was still lots of space on that disk, permissions were OK, and I've tried creating new files there, both from our app and by hand and it worked. It turned out after a few days of frantic debugging, that when there's too many files with similar names in one directory - creating a new file can fail because of hash conflict, depending on the filename only. So, ls A1234.txt S1234.txt # no results touch S12345.txt # fails touch S1234.txt # works :) It was one of the biggest WTF moments I had while programming.
caf 10y ago
This does not seem like the exactly correct diagnosis. ext4 doesn't care if there's collisions in the dir_index hash - they're entirely expected. The dir_index hashes are just used to look up in a tree, the lowest level of which points you to the directory block that contains all the directory entries for names with hashes in a given range. However, those hashes do get used for the directory position used by telldir() / seekdir() - and those are only 32 bits on 32 bit architectures. So I suspect that something involving those calls is generating the ENOSPC.
crgwbr 10y ago
> The technical reasons for this are boring and I really don’t care why; I just want to trust that my filesystem will do the right thing. Statements like this make me trust a developer significantly less. It's our job to care about "technical reasons."

66 points by morecoffee 10y ago

Huh, max 32000 files in a directory seems awfully low. Even crusty old NTFS is supposed to support 4294967295 files. Explorer will not be too happy displaying such a folder though!

RyJones 10y ago
as a practical matter, NTFS used to choke at around a million files in a single directory. This practical limit may be higher now.
fnj 10y ago
ZFS, which is known for having crazy high limits, will fit somewhat over 280 trillion entries in a single directory. Curiously, though, you can have over 18 million trillion file systems in a single storage pool.

Neil44 10y ago

I thought this changed from ext3 to ext4? I know I've run into this issue before and moving to ext4 was the easiest way to resolve it. Ext4 is much quicker at doing operations on very large directories too.

JonRB 10y ago

During my young naïve programming days (which coming to think of it haven't really ended) I thought to myself "Why bother using some third party database when I could just store user information in text files, the OS handles that sort of thing really well, right?". I looked into it as much as I could be bothered and found no huge red flags. After a few years of this server merrily creating a new file for each user I found that my disk was full, as this poster did. After a lot of hunting around, I found out that I had hit my inode limit, a concept I was completely unaware of. I know for sure I won't be making THAT mistake again.

bArray 10y ago
"After a lot of hunting around, I found out that I had hit my inode limit, a concept I was completely unaware of." What was the number you hit? I have a flat file system for something I'm prototyping. The files are in a database with a random SHA256 filename associated with them. I was thinking about doing what Git does with creating folders for parts of the string to alleviate the number of files in one single directory. I'm sure that makes file retrieval a prettier process. Do all the files for have to be in the same directory for the iNode limit to be reached or just there in general? NOTE: I ask because the prototype isn't ever meant to go into production, but my customer keeps getting confused and I'm concerned this very important system may inherit (or even use) my design.
- LukeShu 10y ago
  It's a global limit per filesystem. You can check what the limits are for your filesysems, and what the current use is with `df -i`. The thing that git does is for speed.
  - bArray 10y ago
    I've apparently used 1241165 out of 19005440, or about 6.53% on my development machine. It's not inconceivable to have 19 million files, though, if you had one per user. That's not great. How is that number calculated? It's less than 2^25 but greater than 2^24, so I imagine it's even awkward to store. I guess that has something to do with the file table size for HDD - you can't have more nodes than you have addressable space for those nodes. I knew Git did it for speed, but I was wondering whether it was also about the number of files that can exist in a single directory. EDIT: Corrected percentage.
    - LukeShu 10y ago
      It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).

Animats 10y ago

Classic UNIX problem. Not enough error codes, and too many errors map to the same thing.

rshm 10y ago

No date on the post. Probably old and fixed now. Just checked with a random folder, it happily supported 782292 files.

Hnrobert42 10y ago
I think his point still stands. Basically, if you do something far enough outside the norm, you may hit really time consuming bugs.
- bArray 10y ago
  Although the exception would probably be rare if it used a large enough number, do you think the "copying utility" (whatever it may be) should handle those extreme edge cases? Or is it just "disk full" that's used as a fall-back?
ryanbertrand 10y ago
There is mention of 2016. "The lesson here is that even in 2016, filesystems still are finicky with large numbers of files in a single directory."
Xylakant 10y ago
Since the root cause is a hash collision in the dir_index code you might just have been lucky and the author of the article unlucky.
- caf 10y ago
  They would have had to have been exceptionally lucky - the chance of getting no collisions in a 32 bit hash among 782292 inputs is about 10^-62.
- rshm 10y ago
  Probably lucky. File names are md5 of the content, i am not sure if it helped on collision on 32 bit md4. The linked post mentions dir_index default and tune2fs -l lists the dir_index under Filesystem features only. Not knowing this default behavior, i was considering enabling dir_index explicitly to aid chocking rsync. EDIT: I ran the reproduction code with 200K instead of 100K just to be sure. There was no problem creating file as well as copying all the files to a different folder. The disk was formatted about 3 months ago and mounted with default options "ext4 noatime,barrier=0 0 0" . On original as well as copied folder `ls | wc -l` still gives 200000.

vacri 10y ago

"If you're working with huge files or a huge number of files, use XFS" went the slogan. I was hit by this recently - with our 500ish (!!) entries in /node_modules and their dependencies of dependencies of dependencies, a standard 8GB VM formatted with ext4 just simply didn't have enough inodes for us to have two copies of our app (deployment swapover). Had to increase the size to 20GB, just to get the extra inodes. XFS would have handled it much better (from testing on a parallel machine). It was inode exhaustion, though the 'disk full' error message was the same - confusing at first, as df showed 40% free space...

fnj 10y ago
You could have also tuned ext4 when you created it with mke2fs. You can specify with -N the max # of inodes desired. Granted, it's not something very many people consider when creating their filesystems; for one thing it's hard to predict how many you will need.
- guroot 10y ago
  Also most distro installers don't make it very easy to do this.
- watt 10y ago
  It would be great if it had switch that simply says "double the default number of inodes". Per stories in comments, seems that on large file systems it's easier to run out of inodes, the default must be too low.
- vacri 10y ago
  Unfortunately it was on a pre-baked image on AWS, and you apparently can't retrofit it.

anilgulecha 10y ago

A good filesystem to use as pseudo-db is ZFS. It's designed to have practically limitless number of files, and it's snapshotting/cloning come in really handy for things like backups etc. Also you get full checksumming.

fnj 10y ago
ZFS does have a hard limit of about 280 trillion files in a single directory. It's actually just about the most confining limit it does have.
- anilgulecha 10y ago
  Correct. Hence, practically limitless.
  - bbcbasic 10y ago
    No one will ever need in 280 trillion files. Ever.
    - skykooler 10y ago
      "So, we tried to use ZFS to back up the entire Internet, and we're running into a weird error..."
    - majewsky 10y ago
      And 640k RAM ought to be enough for everyone. ;)
    - liotier 10y ago
      "No one will ever need in 280 trillion files. Ever." ~bbcbasic on Hacker News, September 2016 Found in someone's email .sig in the year 2046...
      - bbcbasic 10y ago
        That typo is immortalised :-o

jobigoud 10y ago

I came to the same conclusion about sharding files from the get go. I had a similar experience where my host wouldn't accept to synchronize a directory with more than 5000 files in it. (Software producing the files is my own note-taking application where each page is stored as an individual file to speed up backup of only the pages that changes).

vog 10y ago
Wouldn't 5000 files of note-taking easily fit into RAM? Why bother dealing with thousands of small files? Also, rsync does quite a good job with intransferring multi-MB (or even larger) files efficiently while copying only what's actually changed.
- jobigoud 10y ago
  I feel more secure by dealing with many individual files. For example in case of a conflict during sync, I can visually diff between the two pages very easily. This happens once in a while. Also for example if more than 5% of files have changed, the sync is stopped and asks for manual confirmation to prevent catastrophies.

cyberferret 10y ago

Brings back memories of the old DOS days, when there was quite a low limit on the number of files that you could store in the root directory. I can't remember the exact numbers, but it lead to my early habits of creating a new sub directory for every little thing so nowadays (over 3 decades later) nearly everything I need is at least 5 levels deep! ;) NB: My current working folders are now in Google Drive, but is essentially the same folder structure as I started on our Novell server about 25 years ago...

jaredsohn 10y ago
> when there was quite a low limit on the number of files that you could store in the root directory. I can't remember the exact numbers 512? https://en.wikipedia.org/wiki/File_Allocation_Table#Final_FAT16 https://en.wikipedia.org/wiki/File_Allocation_Table#Final_FA... (see the last paragraph.)
- cyberferret 10y ago
  That sounds about right. It was a ridiculously low number, and even back in the days of 10MB hard drives, it was easy to bust that number with just batch and text files...
pjc50 10y ago
.. aided by CHKDSK, which would find orphaned allocation units and create files for them in the root directory. Could quite easily fill up your root directory on its own.
- cyberferret 10y ago
  YES! That's right... all those FILEnnnn.CHK files exploding all over your root directory! Ah, those were the days!

guroot 10y ago

Things like this is why I use XFS anywhere I'm not using btrfs, or ZFS.

lake99 10y ago
I've had some bad experience with btrfs. When I had to hard-reset a hung computer, some directories became unremovable and unrenameable. All else failed, and I attempted to --repair, it did so much damage that I had to reformat the disk. This was probably about 3-4 years ago. Have you had such issues on your preferred filesystems in the last couple of years? The reason I stay with ext4 is that it has proven amazingly robust in the event of hard reboots.
- guroot 10y ago
  In my experience XFS is rock solid. We use it almost everywhere at work. I've never had an issue with hard reboots with it. Only recently started playing with btrfs, so can't speak to it's reliability from personal experience.
  - lake99 10y ago
    I hard-reboot about once a week, caused by my carelessness and running out of 16GB of RAM. I always recommend against btrfs for system partitions, and if you don't make frequent backups, against user partitions too. Read through "man btrfsck", that should caution you sufficiently. I'll try out XFS on an external HDD I'll buy soon.
    - guroot 10y ago
      You really shouldn't need to be hard rebooting just because you ran out of ram... did you try a sysrq f (invokes the out of memory killer) first? If that doesn't work you should at least be able to force a sync and remount read-only before taking it down. If you have access to a keyboard and the kernel is still responding to sysrq you should never just hard power off without at least trying to regain control of the system. Some reading if you're not familiar with sysrq: https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key Depending on your distro you might have to enable it, I believe ubuntu (for some dumb reason) ships with emergency sync as the only command available. Enabling it should amount to adding: kernel.sysrq=1 to /etc/sysctl.conf on older machines or editing /etc/sysctl.d/10-magic-sysrq.conf on newer ones. also if you are running out of ram that often you might want to consider a swap partition. Or if you have one already try setting a more aggressive swapiness value.

Nux 10y ago

Yes, this "silly" kind of thing bit us a few times as well. It's good RHEL/CentOS 7 defaults to XFS now which doesn't suffer (as much) from this problem. https://gagor.pl/2013/11/inodes-on-xfs/ https://gagor.pl/2013/11/inodes-on-xfs/

ajuc 10y ago

I've had similar error in production. We had linux server with C++ program doing many things, and in the process it created temporary files for a barcode printer. The files were named uniquely, the server had lots of free space, we left them there to allow easier debugging if sth wrong happened (it happened a few times each year - we created the labels to print by glueing escape codes and sometimes there were labels that didn't worked for one reason or another), and only cleaned the old files when there was not much space left. At some point there was a bug when sometimes users got "wrong permissions or no space left" errors (sorry don't remember the exact error and IIRC our app just tried to save a file and failed with unspecific error if it couldn't write the new temporary file, so we haven't had the exact system error). What was weird is - there was still lots of space on that disk, permissions were OK, and I've tried creating new files there, both from our app and by hand and it worked. It turned out after a few days of frantic debugging, that when there's too many files with similar names in one directory - creating a new file can fail because of hash conflict, depending on the filename only. So, ls A1234.txt S1234.txt # no results touch S12345.txt # fails touch S1234.txt # works :) It was one of the biggest WTF moments I had while programming.

caf 10y ago

This does not seem like the exactly correct diagnosis. ext4 doesn't care if there's collisions in the dir_index hash - they're entirely expected. The dir_index hashes are just used to look up in a tree, the lowest level of which points you to the directory block that contains all the directory entries for names with hashes in a given range. However, those hashes do get used for the directory position used by telldir() / seekdir() - and those are only 32 bits on 32 bit architectures. So I suspect that something involving those calls is generating the ENOSPC.

crgwbr 10y ago

> The technical reasons for this are boring and I really don’t care why; I just want to trust that my filesystem will do the right thing. Statements like this make me trust a developer significantly less. It's our job to care about "technical reasons."