4 ms·

It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but b

by LukeShu 10y ago

It's a property of the filesystem that is set when it was created. In the case of the mkfs.ext{2,3,4}, you can set the number of inodes with the -N flag, but by default, it is the size of the filesystem divided by inode_ratio (probably 4096).

bArray 10y ago
Seems to be about 16k for the ratio? 19,000,000 * 16,000 ~= 300GB - which is about the size of my disk partition. That's crazy. Thanks for that information.
- danieldk 10y ago
  Your filesystem's block size is probably 4KiB. So, it's one inode per for blocks. That's crazy. Why? Do you expect to store many files less than 16KiB?
  - bArray 10y ago
    Each user has some small arbitrary information associated with them, with potentially tens of millions of users (and no doubt more in the future). There's not really a structure due to the nature of the project, each user needs personalised information and structure associated to them. I don't think that amount of information is really suitable for a database? It may be larger in some cases too, so there's not guarantees I can even make about size.
    - danieldk 10y ago
      Ah, sorry, I missed that you were still referring to the 'filesystem as a database'-project :).
    - Xylakant 10y ago
      Document stores shine in that case. Depending on the exact case something like postgres's HSTORE, couchdb, couchbase, mongodb or similar would work. They're all capable of storing arbitrary json docs under a given key and efficiently retrieve it. Partial indexing is possible as well.
- deleted 10y ago
  [deleted]
caf 10y ago
This seems to be one of those bits of old UNIX lore that's being lost - it used to be fairly well-known in sysadmin circles that if you were going to be storing a lot of tiny files, you should create your filesystem with a higher than usual number of inodes (the classic case was an NNTP spool mount).
- bArray 10y ago
  "This seems to be one of those bits of old UNIX lore that's being lost" Hopefully not now, this is why we all appreciate people sharing these sorts of information :)
- gioele 10y ago
  > if you were going to be storing a lot of tiny files, you should create your filesystem with a higher than usual number of inodes (the classic case was an NNTP spool mount). I wonder how many people understand what the term "news server" of the partition wizard in the Debian installer means.
- technofiend 10y ago
  I literally just solved an issue this weekend that was inode exhaustion: there were years of snapshots and traces laying around and the DBAs were confused as to why the database couldn't create a directory to start and was exiting with "No space left on device" when there was 5% storage free. That lead to the second discussion of why you don't fix it by piping filenames one by one to rm, but either use find's inbuilt unlink/rm if there is one or xargs to rm if not.
  - greggyb 10y ago
    Is that second discussion just a performance thing, or are there filesystem implications for one or the other?
    - technofiend 10y ago
      Just speed: you don't spin up a new process for each file removed if you use xargs.

by LukeShu 10y ago

bArray 10y ago

Seems to be about 16k for the ratio? 19,000,000 * 16,000 ~= 300GB - which is about the size of my disk partition. That's crazy. Thanks for that information.

danieldk 10y ago
Your filesystem's block size is probably 4KiB. So, it's one inode per for blocks. That's crazy. Why? Do you expect to store many files less than 16KiB?
- bArray 10y ago
  Each user has some small arbitrary information associated with them, with potentially tens of millions of users (and no doubt more in the future). There's not really a structure due to the nature of the project, each user needs personalised information and structure associated to them. I don't think that amount of information is really suitable for a database? It may be larger in some cases too, so there's not guarantees I can even make about size.
  - danieldk 10y ago
    Ah, sorry, I missed that you were still referring to the 'filesystem as a database'-project :).
  - Xylakant 10y ago
    Document stores shine in that case. Depending on the exact case something like postgres's HSTORE, couchdb, couchbase, mongodb or similar would work. They're all capable of storing arbitrary json docs under a given key and efficiently retrieve it. Partial indexing is possible as well.
deleted 10y ago
[deleted]

caf 10y ago

This seems to be one of those bits of old UNIX lore that's being lost - it used to be fairly well-known in sysadmin circles that if you were going to be storing a lot of tiny files, you should create your filesystem with a higher than usual number of inodes (the classic case was an NNTP spool mount).

bArray 10y ago
"This seems to be one of those bits of old UNIX lore that's being lost" Hopefully not now, this is why we all appreciate people sharing these sorts of information :)
gioele 10y ago
> if you were going to be storing a lot of tiny files, you should create your filesystem with a higher than usual number of inodes (the classic case was an NNTP spool mount). I wonder how many people understand what the term "news server" of the partition wizard in the Debian installer means.
technofiend 10y ago
I literally just solved an issue this weekend that was inode exhaustion: there were years of snapshots and traces laying around and the DBAs were confused as to why the database couldn't create a directory to start and was exiting with "No space left on device" when there was 5% storage free. That lead to the second discussion of why you don't fix it by piping filenames one by one to rm, but either use find's inbuilt unlink/rm if there is one or xargs to rm if not.
- greggyb 10y ago
  Is that second discussion just a performance thing, or are there filesystem implications for one or the other?
  - technofiend 10y ago
    Just speed: you don't spin up a new process for each file removed if you use xargs.