What Is a Reasonable Amount of Inotify Watches with Linux

What is a reasonable amount of inotify watches with Linux?

/proc/sys/fs/inotify/max_user_watches is the current max number of watches per user.

Historically, the kernel has defaulted this to 8192, but given that many Linux distros customize their kernel builds quite a bit, this may not be true on every Linux system. A recent kernel change [1] dynamically selects a default max_user_watches value in the range [8192, 1048576] based on how much RAM the system has. (5.11 is the first kernel release containing this change.)

AFAICT, root can change max_user_watches to any value that's 2147483647 (2³¹-1) or under, as long as you're confident you have enough RAM to support that number of watches.

[1] https://github.com/torvalds/linux/commit/92890123749bafc317bbfacbe0a62ce08d78efb7

Best way to watch a large number of directories?

Your approach is the correct one, and inotify is the most performant way to do this. If you run into this issue, you could just prompt the user to increase limit on their behalf to provide a better user experience (you have to perform a privilege escalation by asking for password for this).

Error User limit of inotify watches reached. ExtReact build

Why?

Programs that sync files such as dropbox, git etc use inotify to notice changes to the file system. The limit can be see by -

cat /proc/sys/fs/inotify/max_user_watches

For me, it shows 100000. When this limit is not enough to monitor all files inside a directory it throws this error.

---

Increasing the amount of inotify watchers(Short version):

If you are running Debian, RedHat, or another similar Linux distribution, run the following in a terminal:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

If you are running ArchLinux, run the following command instead (see here for why):

echo fs.inotify.max_user_watches=524288 | sudo tee /etc/sysctl.d/40-max-user-watches.conf && sudo sysctl --system

Then paste it in your terminal and press on enter to run it.

---

Technical details:

Listen uses inotify by default on Linux to monitor directories for changes. It's not uncommon to encounter a system limit on the number of files you can monitor. For example, Ubuntu Lucid's (64bit) inotify limit is set to 8192.

You can get your current inotify file watch limit by executing:

$ cat /proc/sys/fs/inotify/max_user_watches

When this limit is not enough to monitor all files inside a directory, the limit must be increased for Listen to work properly.

You can set a new limit temporary with:

$ sudo sysctl fs.inotify.max_user_watches=524288
$ sudo sysctl -p

If you like to make your limit permanent, use:

$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p

You may also need to pay attention to the values of max_queued_events and max_user_instances if Listen keeps on complaining.

Source: https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers

can multiple scripts use inotifywait to watch the same file simultaneously?

Yes, it is safe. inotify is a Linux kernel API which allows concurency.

correct use of linux inotify - reopen every time?

Usually the pseudo code inotify loop would look like this:

initialize inotify
watch a directory | file for events

while(receive event) {
  process event
}

[ remove watch ]
close inotify fd

There is no need to remove the watch and reinitialize inotify on every loop.

Why my Inotify script wont go further after watches established?

You have your while loop backwards, you just want to trigger rsync when inotify fires, in response to a file change in your directory -- not repeatedly call inotifywait. You should also add the -me (-m monitor) option for inotifywait, e.g.

inotifywait -me modify,create,delete /home/Mansoor/ITB/uploads | while read; do
    rsync -avz /home/My_home/ITB/uploads /home/My_home/destination
done

That way, each time inotifywait fires (you don't capture the output), you simply call rsync to sync the directories.

Here is an example:

Example Script

#!/bin/bash

inotifywait -me modify,create,delete ~/dev/src-c/tmp/debug/dir1 | while read; do
    rsync -avz ~/dev/src-c/tmp/debug/dir1/ ~/dev/src-c/tmp/debug/dir2
done

Create 2 Empty Dirs

$ ls -al dir1
total 52
drwxr-xr-x  2 david david  4096 May  4 01:42 .
drwxr-xr-x 14 david david 49152 May  4 01:38 ..

$ ls -al dir2
total 52
drwxr-xr-x  2 david david  4096 May  4 01:42 .
drwxr-xr-x 14 david david 49152 May  4 01:38 ..

Start inotifywait script

$ bash inw.sh &
Setting up watches.
Watches established.

Make a change

$ touch dir1/foo

(additional terminal output resulting from rsync triggered by inotifywait)

sending incremental file list
./
foo

sent 115 bytes  received 38 bytes  306.00 bytes/sec
total size is 0  speedup is 0.00

Verify it Works

$ ls -al dir2
total 52
drwxr-xr-x  2 david david  4096 May  4 01:44 .
drwxr-xr-x 14 david david 49152 May  4 01:38 ..
-rw-r--r--  1 david david     0 May  4 01:44 foo

Yes, working as advertised...

Do a couple more

$ touch dir1/bar
$ touch dir1/baz

and check

$ ls -al dir2
drwxr-xr-x  2 david david  4096 May  4 01:54 .
drwxr-xr-x 14 david david 49152 May  4 01:38 ..
-rw-r--r--  1 david david     0 May  4 01:53 bar
-rw-r--r--  1 david david     0 May  4 01:54 baz
-rw-r--r--  1 david david     0 May  4 01:44 foo

---

Related Topics