Upper Limit of File Descriptor in Linux

How can we limit file descriptor to always be less than 1024?

You don't want to do this. Linux always uses the lowest possible file descriptor. So if you get file descriptor 1024 then it means file descriptors 0 up to 1023 were all used already.

If you make Linux only use file descriptors 0-1023, then your program still won't work, because instead of getting file descriptor 1024, you'll get an error saying there aren't any more file descriptors it can use.

You should:

1. Make sure your program closes file descriptors when it's done with them.
   
   Maybe the reason descriptors 0-1023 were all used is because you forgot to close them. So many sure your program closes them.
   
   If they were real file descriptors that your program was actually using, not just ones you forgot to close, then continue to step 2...
2. Use poll instead of select, like the document says. It is not a difficult change.
3. Consider using epoll, which is more efficient than select when you are polling a lot of file descriptors at the same time. However, it is much more complicated.

Can I make the file descriptor limit a per-thread limit?

On Linux,

unshare(CLONE_FILES);

(if successful) will give the current thread its own filedescriptor table.

It should be usable from an already- spawned kernel-backed (all Linux pthreads implementations I know of) pthread.

There doesn't appear to be even a nonportable pthread attribute that you could set this up with, but you can use the above approach to wrap pthread_create, adding this capability.

If you're doing your own threading on Linux, you can pass the CLONE_FILES flag directly to clone. Otherwise you'll need the to make your wrapped_pthread_create wait until the child has made the unshare call (and cancel and reap the thread if the call failed).

About limiting the number of file descriptors

/proc/sys/fs/file-max takes precedence over any ulimit settings in the shell.

More over /proc/sys/fs/file-max is the total number of FD open for ALL processes on given machine.

ulimit settings are per process, so any new process started will have given limit (unless the total form file-max is exceeded in the system).

What's the range of file descriptors on 64-bit Linux?

The epoll_ctl(2) interface to add new filedescriptors takes an int fd argument, so you're already limited to 32-bit range (at least on the Linux platforms I'm familiar with).

You're further limited by /proc/sys/fs/file-max system-wide limit on the number of open files for all processes; /proc/sys/fs/file-max is currently 595956 on my system.

Each process is further limited via the setrlimit(2) RLIMIT_NOFILE per-process limit on the number of open files. 1024 is a common RLIMIT_NOFILE limit. (It's very easy to change this limit via /etc/security/limits.conf.)

It's a rare application that needs more than 1024. The full 32 bits seems unlikely as well, since each open file will take some kernel memory to represent -- four billion ~280 byte struct inode structures (at the minimum) is a lot of pinned memory.

Is the value of a Linux file descriptor always smaller than the open file limits?

Yes, the values are limited to the range from 0 to one less than the current limit as returned by getrlimit().

From the getrlimit() man page:

   RLIMIT_NOFILE
          Specifies a value one greater than the maximum file descriptor number
          that can be opened by this process.  Attempts (open(2), pipe(2),
          dup(2), etc.)  to exceed this limit yield the error EMFILE.
          (Historically, this limit was named RLIMIT_OFILE on BSD.)

From the Open Group Base Specification:

RLIMIT_NOFILE

This is a number one greater than the maximum value that the system may assign to a newly-created descriptor. If this limit is
exceeded, functions that allocate a file descriptor shall fail with
errno set to [EMFILE]. This limit constrains the number of file
descriptors that a process may allocate.

How do linux file descriptor limits work?

You want to look at /proc/sys/fs/file-max instead

From recent linux/Documentation/sysctl/fs.txt:

file-max & file-nr:

The kernel allocates file handles dynamically, but as yet it
doesn't free them again.

The value in file-max denotes the maximum number of file-
handles that the Linux kernel will allocate. When you get lots
of error messages about running out of file handles, you might
want to increase this limit.

Historically, the three values in file-nr denoted the number of
allocated file handles, the number of allocated but unused file
handles, and the maximum number of file handles. Linux 2.6 always
reports 0 as the number of free file handles -- this is not an
error, it just means that the number of allocated file handles
exactly matches the number of used file handles.

Attempts to allocate more file descriptors than file-max are
reported with printk, look for "VFS: file-max limit
reached".

EDIT: the underlying error is probably not the system running out of global filedescriptors, but just your process. It seems likely that the problem is the maximum size limit of select.

Is there any way to increase the maximum amount of file descriptors that select() can handle on Windows? _setmaxstdio doesn't affect it

A few days later, my best understanding of the situation is this: the per-process limit to number of file descriptors IS controlled by _setmaxstdio(), and I was using it correctly, BUT that upper limit set by _setmaxstdio() does not apply if you use the select() function, because select() has a hardcoded limit. In order for the limit you set with _setmaxstdio() to have an actual effect, you must use poll(), epoll(), etc., instead of select(). And if there is a way to increase the limits of the select() function, it seems like you need to recompile part of Windows' C runtime, which is not a good idea. Since I have the option to drop support for Windows and only support Unix instead, I'd much rather just do it that way.

If you're reading this answer because you want to increase the limits of Windows' select() function, and like me, you are new to the concept of file descriptors/completion ports, and haven't extensively worked with sockets before, and don't want to/don't know how to screw around with the C runtime (CRT), I suggest you first consider the following before continuing trying to alter the select() limit:

• Can you use poll() or epoll() instead of select()? In my case, the only reason I was using select() is because I was locked into because of the libraries I'm using - asyncio and psycopg. At the time of writing, psycopg does not support asyncio's ProactorEventLoop in Windows, so I was forced to instead use the SelectorEventLoop (which uses select()) instead of the default ProactorEventLoop, which doesn't use select(). However, psycopg has no such limitation in Unix - it will work with any asyncio event loop there. So if you can use poll(), epoll(), etc instead of select(), then that's probably a lot easier than trying to actually increase the select() limit since then you'll be able to just use _setmaxstdio() to set the file descriptor limit, and it'll actually let you have more file descriptors.

Instead of continuing to try to make this work with Windows, I'm just going to use Unix instead - I probably should have done that from the beginning since it's a backend kind of process, but the convenience of being able to run/test/debug the code directly on my development machine was tempting, and I thought that the added flexibility of supporting both Windows and Unix would be worth the (what I thought would be minimal) overhead of adding if os is Windows use SelectorEventLoop to the start of my application.

---

Related Topics