How Do Linux File Descriptor Limits Work

How do linux file descriptor limits work?

You want to look at /proc/sys/fs/file-max instead

From recent linux/Documentation/sysctl/fs.txt:

file-max & file-nr:

The kernel allocates file handles dynamically, but as yet it
doesn't free them again.

The value in file-max denotes the maximum number of file-
handles that the Linux kernel will allocate. When you get lots
of error messages about running out of file handles, you might
want to increase this limit.

Historically, the three values in file-nr denoted the number of
allocated file handles, the number of allocated but unused file
handles, and the maximum number of file handles. Linux 2.6 always
reports 0 as the number of free file handles -- this is not an
error, it just means that the number of allocated file handles
exactly matches the number of used file handles.

Attempts to allocate more file descriptors than file-max are
reported with printk, look for "VFS: file-max limit
reached".

EDIT: the underlying error is probably not the system running out of global filedescriptors, but just your process. It seems likely that the problem is the maximum size limit of select.

About limiting the number of file descriptors

/proc/sys/fs/file-max takes precedence over any ulimit settings in the shell.

More over /proc/sys/fs/file-max is the total number of FD open for ALL processes on given machine.

ulimit settings are per process, so any new process started will have given limit (unless the total form file-max is exceeded in the system).

How can we limit file descriptor to always be less than 1024?

You don't want to do this. Linux always uses the lowest possible file descriptor. So if you get file descriptor 1024 then it means file descriptors 0 up to 1023 were all used already.

If you make Linux only use file descriptors 0-1023, then your program still won't work, because instead of getting file descriptor 1024, you'll get an error saying there aren't any more file descriptors it can use.

You should:

1. Make sure your program closes file descriptors when it's done with them.
   
   Maybe the reason descriptors 0-1023 were all used is because you forgot to close them. So many sure your program closes them.
   
   If they were real file descriptors that your program was actually using, not just ones you forgot to close, then continue to step 2...
2. Use poll instead of select, like the document says. It is not a difficult change.
3. Consider using epoll, which is more efficient than select when you are polling a lot of file descriptors at the same time. However, it is much more complicated.

What's the impact of large number of file descriptors on Java applications

Operating systems have a limit on the number of files that can be concurrently open by any one process. The default for most distributions is only 1024 files. Each open file also has an associated file-descriptor. Socket connections are treated like files and they use file descriptor, and are therefore subject to the same resource limits.

1. You can verify or change the maximum limit by the command ulimit.

2. You can also view value of the MBean Attibutes -
   MaxFileDescriptorCount & OpenFileDescriptorCount by running the JMX tool- JConsole.

3. When OpenFileDescriptorCount is less than MaxFileDescriptorCount
   your application works fine, otherwise you would get java.io.IOException: Too many open files which causes malfunctioning of your application .

4. Normally for an application the count of the FD(File Descriptor)
   goes up/down to certain level. But it should be within
   MaxFileDescriptorCount.

Is the value of a Linux file descriptor always smaller than the open file limits?

Yes, the values are limited to the range from 0 to one less than the current limit as returned by getrlimit().

From the getrlimit() man page:

   RLIMIT_NOFILE
          Specifies a value one greater than the maximum file descriptor number
          that can be opened by this process.  Attempts (open(2), pipe(2),
          dup(2), etc.)  to exceed this limit yield the error EMFILE.
          (Historically, this limit was named RLIMIT_OFILE on BSD.)

From the Open Group Base Specification:

RLIMIT_NOFILE

This is a number one greater than the maximum value that the system may assign to a newly-created descriptor. If this limit is
exceeded, functions that allocate a file descriptor shall fail with
errno set to [EMFILE]. This limit constrains the number of file
descriptors that a process may allocate.

---

Related Topics