: What is a recommended maximum number of files in a directory on your webserver? I have directory that contains 400,000+ files. Potentially the directory could easily contain ~30,000,000 files.

@Megan663

I've tried to have directories with tens of thousands of files on ext3 and ext4 Linux. It eventually gets to the point where listing those files is prohibitively expensive. It can take a few seconds for ls to complete.

I would try do design for a max of 10,000 files per directory.

One way of doing so is to use the first couple characters of the file name as a directory. The git source control system takes this approach. See Why does git store objects in directories with the first two characters of the hash?. Git limits the number of hash revisions per directory to 6700.

Vote Up Vote Down

@Bryan171

I am going to give this a shot.

From a filesystem point of view:


FAT32:


Maximum number of files: 268,173,300
Maximum number of files per directory: 216 - 1 (65,535)
Maximum file size: 2 GiB - 1 without LFS, 4 GiB - 1 with


NTFS:


Maximum number of files: 232 - 1 (4,294,967,295)
Maximum file size


Implementation: 244 - 26 bytes (16 TiB - 64 KiB)
Theoretical: 264 - 26 bytes (16 EiB - 64 KiB)

Maximum volume size


Implementation: 232 - 1 clusters (256 TiB - 64 KiB)
Theoretical: 264 - 1 clusters



ext2:


Maximum number of files: 1018
Maximum number of files per directory: ~1.3 × 1020 (performance issues past
10,000)
Maximum file size


16 GiB (block size of 1 KiB)
256 GiB (block size of 2 KiB)
2 TiB (block size of 4 KiB)
2 TiB (block size of 8 KiB)

Maximum volume size


4 TiB (block size of 1 KiB)
8 TiB (block size of 2 KiB)
16 TiB (block size of 4 KiB)
32 TiB (block size of 8 KiB)



ext3:


Maximum number of files: min(volumeSize / 213, numberOfBlocks)
Maximum file size: same as ext2
Maximum volume size: same as ext2


ext4:


Maximum number of files: 232 - 1 (4,294,967,295)
Maximum number of files per directory: unlimited
Maximum file size: 244 - 1 bytes (16 TiB - 1)
Maximum volume size: 248 - 1 bytes (256 TiB - 1)



Reference: stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory
From a functionality point of view:


Keep in mind that on Linux if you have a directory with too many
files, the shell may not be able to expand wildcards. I have this
issue with a photo album hosted on Linux. It stores all the resized
images in a single directory. While the file system can handle many
files, the shell can't.


From a server speed point of view:

Too many files in one directory can cause load times to increase by seconds. Having too many directories can also increase load times. (Your server specs play a role in this)

From an SEO point of view.

I can understand not having proper image names for security reasons (assuming a user uploads photos and you have a rewrite in place) But you really should give a content relevant (sub)directory names for better search ranking and the ability to partially remove URL's to get to another spot on your server. (ie photos/outdoor/landscape/mountains)

In the end there is no one fits all but you can make a better informed decision based on the aforementioned.

Vote Up Vote Down