What Is The Maximum Number of Subdirectories Allowed in Ext4

What is the maximum number of subdirectories allowed in Ext4?

The limit is 64000. Until you enable the file system feature flag dir_nlink

http://man7.org/linux/man-pages/man5/ext4.5.html

http://www.ispexperts.com.np/?p=797

what is the max files per directory in EXT4?

It depends upon the MKFS parameters used during the filesystem creation. Different Linux flavors have different defaults, so it's really impossible to answer your question definitively.

What is the maximum allowed depth of sub-folders?

The limit is not on the depth of the nested subdirectories (you could have dozens of them, even more), but on the file systems and its quotas.

Also having very long file paths is inconvenient (and could be slightly inefficient). Programmatically, a file path of several hundreds or even thousands of characters is possible; but the human brain is not able to remember such long file paths.

Most file systems (on Linux) have a fixed limit to their number of inodes.

Some file systems behave poorly with directories containing ten thousand entries (e.g. because the search is linear not dichotomic). And you have hard time to deal with them (e.g. even ls * gives too long output). Hence, it could be wise to have /somepath/a/0001 ... /somepath/z/9999 instead of /somepath/a0001 ... /somepath/z9999

If you have many thousands of users each with his directory, you might want to e.g. group the users by their initials, e.g. have /some/path/A/userAaron/images/foobar and /some/path/B/userBasile/images/barfoo etc. So /some/path/A/ would have only hundreds of subdirectories, etc...

A convenient rule of thumb might be: avoid having more than a few hundreds entries -either subdirectories or files- in each directory.

Some web applications store small data chunk in individual rows of a SQL databases and use files (whose name might be generated) for larger data chunks, storing the filepath in the database. Having millions of files with only a few dozen bytes in most is probably not efficient.

Some sysadmins are also using quotas on filesystems.

How many files can I put in a directory?

FAT32:

• Maximum number of files: 268,173,300
• Maximum number of files per directory: 2¹⁶ - 1 (65,535)
• Maximum file size: 2 GiB - 1 without LFS, 4 GiB - 1 with
  

NTFS:

• Maximum number of files: 2³² - 1 (4,294,967,295)
  
• Maximum file size
  
  
  • Implementation: 2⁴⁴ - 2⁶ bytes (16 TiB - 64 KiB)
  • Theoretical: 2⁶⁴ - 2⁶ bytes (16 EiB - 64 KiB)
• Maximum volume size
  
  
  • Implementation: 2³² - 1 clusters (256 TiB - 64 KiB)
  • Theoretical: 2⁶⁴ - 1 clusters (1 YiB - 64 KiB)

ext2:

• Maximum number of files: 10¹⁸
• Maximum number of files per directory: ~1.3 × 10²⁰ (performance issues past 10,000)
• Maximum file size
  
  
  • 16 GiB (block size of 1 KiB)
  • 256 GiB (block size of 2 KiB)
  • 2 TiB (block size of 4 KiB)
  • 2 TiB (block size of 8 KiB)
• Maximum volume size
  
  
  • 4 TiB (block size of 1 KiB)
  • 8 TiB (block size of 2 KiB)
  • 16 TiB (block size of 4 KiB)
  • 32 TiB (block size of 8 KiB)

ext3:

• Maximum number of files: min(volumeSize / 2¹³, numberOfBlocks)
  
• Maximum file size: same as ext2
• Maximum volume size: same as ext2

ext4:

• Maximum number of files: 2³² - 1 (4,294,967,295)
• Maximum number of files per directory: unlimited
• Maximum file size: 2⁴⁴ - 1 bytes (16 TiB - 1)
• Maximum volume size: 2⁴⁸ - 1 bytes (256 TiB - 1)

Ideal number of files per folder for large sets of files

If performance is what you're worrying about, this will depend the
type of filesystem you are using. Older filesystems like
ext2 kept
directory entries in a linear list. Looking up a particular file in a
directory could be very expensive.

Modern filesystems such as ext4,
btrfs,
xfs
and others typically have indexed directories, the access time of a
single file in a huge directory isn't going to be appreciably
different from accessing a single file in a small directory. In fact,
spreading millions of files over many subdirectories may give you
slower lookup performance than having them all in a single directory!

If you are writing your own software that will do a lot of linear
scans of the entire set of files or access individual files by name,
it probably doesn't matter which way you go about it (as long as you
access it the right
way.

I would worry more about managing the file system outside of the
application. Typical system utilities (like ls) may use readdir() or
linear scans of directories. To prevent the sysadmins from having
terrible headaches when diagnosing issues within the directory
structure, I'd go with something agreeably bushy, and 10k-20k entries
per directory (assuming indexed directories) would work.

When choosing your layout, you may wish to watch out for limits on the
number of subdirectories allowed per directory (i.e. 64000 on ext4).

AWS EFS: Max number of files allowed in a folder

As you noted, I don't see any mention of file count limits in the AWS EFS Limits doc https://docs.aws.amazon.com/efs/latest/ug/limits.html

These limits guides are a good way to know about hard service limits that would affect any implementation. If it's not mentioned here, there likely is no such limit.

Also note, EFS implements NFSv4, which is a network file sharing protocol. There is some good discussion of why NFS itself would have no such inherent limit. See https://superuser.com/questions/700102/file-count-limit-for-nfs-mounted-drive and See also https://aws.amazon.com/efs/

Finally, here is a good discussion on file system limits How many files can I put in a directory?

I would assume that the file system back end on EFS has to be ext4. As noted above, an ext4 filesystem has no limit to the number of files in a directory.

Based on the above, I believe is there is no such limit in EFS. It should handle your file requirements.

Maximum number of files/directories on Linux?

ext[234] filesystems have a fixed maximum number of inodes; every file or directory requires one inode. You can see the current count and limits with df -i. For example, on a 15GB ext3 filesystem, created with the default settings:

Filesystem           Inodes  IUsed   IFree IUse% Mounted on
/dev/xvda           1933312 134815 1798497    7% /

There's no limit on directories in particular beyond this; keep in mind that every file or directory requires at least one filesystem block (typically 4KB), though, even if it's a directory with only a single item in it.

As you can see, though, 80,000 inodes is unlikely to be a problem. And with the dir_index option (enablable with tune2fs), lookups in large directories aren't too much of a big deal. However, note that many administrative tools (such as ls or rm) can have a hard time dealing with directories with too many files in them. As such, it's recommended to split your files up so that you don't have more than a few hundred to a thousand items in any given directory. An easy way to do this is to hash whatever ID you're using, and use the first few hex digits as intermediate directories.

For example, say you have item ID 12345, and it hashes to 'DEADBEEF02842.......'. You might store your files under /storage/root/d/e/12345. You've now cut the number of files in each directory by 1/256th.

Optimal number of files per directory vs number of directories for EXT4

10k files inside a single folder is not a problem on Ext4. It should have the dir_index option enabled by default, which indexes directories content using a btree-like structure to prevent performance issues.

To sum up, unless you create millions of files or use ext2/ext3, you shouldn't have to worry about system or FS performance issues.

That being said, shell tools and commands don't like to be called with a lot of files as parameter ( rm * for example) and may return you an error message saying something like 'too many arguments'. Look at this answer for what happens then.

---

Related Topics