Suppress or Prevent Duplicate Inotifywait Events

How to stop the repetition of inotify result?

Some remarks:

• You should not "break" as you go out of the inside "while" loop and call read() again
• The IN_MODIFY event does not fill the event->name field (i.e. event->len = 0)
• The number of IN_MODIFY events depends on how you modify the files ("echo xxx >> file" = write only = 1 IN_MODIFY; "echo xxx > file" = truncate + write = 2 IN_MODIFY)

Here is a proposition for your program (with additional prints):

#include <stdio.h>
#include <sys/inotify.h>
#include <unistd.h>
#include <stdlib.h>

#define EVENT_BUF_LEN 4096
#define EVENT_SIZE sizeof(struct inotify_event)

typedef struct{
  int length, fd, wd1, wd2;
  char buffer[EVENT_BUF_LEN] __attribute__ ((aligned(__alignof__(struct inotify_event))));
} notification;

notification inotify;

int getNotified(char *pathname1, char *pathname2){

  inotify.fd = inotify_init();
  inotify.wd1 = inotify_add_watch(inotify.fd, pathname1, IN_MODIFY);
  printf("wd1 = %d\n", inotify.wd1);
  inotify.wd2 = inotify_add_watch(inotify.fd, pathname2, IN_MODIFY);
  printf("wd2 = %d\n", inotify.wd2);

  while(1){
    inotify.length = read(inotify.fd, inotify.buffer, EVENT_BUF_LEN); 
    int i = 0;
    printf("read() = %d\n", inotify.length);
    while(i < inotify.length){     
      struct inotify_event *event = (struct inotify_event *)&inotify.buffer[i];
      printf("event->len = %u\n", event->len);
      if(event->mask & IN_MODIFY){
        if(event->wd == inotify.wd1){
          printf("Pathname1 is modified\n");
        } else if (event->wd == inotify.wd2){
          printf("Pathname2 is modified\n");
        }
      }
      i += (EVENT_SIZE + event->len);
      printf("i=%d\n", i);
    }
  }
  inotify_rm_watch(inotify.fd, inotify.wd1);
  inotify_rm_watch(inotify.fd, inotify.wd2);

  close(inotify.fd);
  exit(0);
}

int main(void)
{
  getNotified("/tmp/foo", "/tmp/bar");

  return 0;

} // main

Here is an example of execution:

$ gcc notif.c -o notif
$ > /tmp/foo
$ > /tmp/bar
$ ./notif 
wd1 = 1
wd2 = 2

====== Upon "> /tmp/foo": 1 event (truncate operation)
read() = 16
event->len = 0
Pathname1 is modified
i=16

====== Upon "echo qwerty > /tmp/foo": 2 events (write operation, one event for truncate operation and one for the write of "qwerty" at the beginning of the file)
read() = 16
event->len = 0
Pathname1 is modified
i=16
read() = 16
event->len = 0
Pathname1 is modified
i=16

====== Upon "echo qwerty >> /tmp/foo": 1 event (write of "qwerty" at the end of the file)
read() = 16
event->len = 0
Pathname1 is modified
i=16

Are delete events considered close_write in inotifywait?

Cleaned-up your script and added some safety with quotes and check for already processed file in case the filesystem triggers duplicate events for same file.

#!/usr/bin/env bash

# Prevents expanding pattern without matches
shopt -s nullglob

# Expands pattern into an array
target=(/home/*/ftp/files/)

# Creates temporary directory and cleanup trap
declare -- tmpdir=
if tmpdir=$(mktemp -d); then
  trap 'rm -fr -- "$tmpdir"' EXIT INT
else
  # or exit error if it fails
  exit 1
fi

# In case no target matches, exit error
[ "${#target[@]}" -gt 0 ] || exit 1

s3move() {
  local -- p=$1
  local -- tmp="$tmpdir/$p"
  printf 'Copy path is: %s\n' "$p"
  # Moves the file to temporary dir
  # so it is away from inotify watch dir ASAP
  mv -- "$p" "$tmp"

  # Then perform the slow remote copy to s3 bucket
  # Remove the echo onces it is ok
  echo aws s3 mv "$p" s3://bucket
  
  # File has been copied to s3, tmp file no longer needed
  rm -f -- "$tmp"
}

while read -r -d '' p; do
  # Skip if file does not exist, as it has already been moved away
  # case of a duplicate event for already processed file
  [ -e "$p" ] || continue
  s3move "$p"
done < <(
  # Good practice to spell long option names in a script
  # --format will print null-delimited full file path
  inotifywait \
    --monitor \
    --recursive \
    --event close_write \
    --includei '.*\.mp4$' \
    --format '%w%f%0' \
    "${target[@]}" 2>/dev/null
)

Queuing events with inotifywait

You can use Watchman instead; it runs as a persistent service and will remember the events that you missed.

There are a number of different ways that you might achieve your use-case, but probably the easiest is to set up a trigger and have watchman directly run your script as files change, or use the pywatchman client bindings to have a persistent python script run and subscribe to the events and take whatever action you'd like.

inotifywait triggering event twice while converting docx to PDF

It's triggered twice because this is how soffice appears to behave internally.
One day it may start writing it 10 times and doing sleep 2 between such writes during a single run, our program can't and I believe shouldn't anticipate it and depend on it.

So I'd try solving the problem from a different angle - lets just put the converted file into a temporary directory and then move it to the target dir, like this:

soffice --headless --convert-to pdf:writer_pdf_Export <path>/upload/somedoc.docx --outdir <path>/tempdir/ && mv <path>/tempdir/somedoc.pdf <path>/upload/

and use inotifywait in the following way:

inotifywait -r -e moved_to  -m "<path>/upload" --format '%f##@@##%e##@@##%w'

The advantage is that you no longer depend on soffice's internal logic.
If you can't adjust behavior of the script producing the pdf files then indeed you'll need to resort to a workaround like @Tarun suggested.

inotify event IN_MODIFY occurring twice for tftp put

try using IN_CLOSE_WRITE instead

Q: What is the difference between IN_MODIFY and IN_CLOSE_WRITE?

The
IN_MODIFY event is emitted on a file content change (e.g. via the
write() syscall) while IN_CLOSE_WRITE occurs on closing the changed
file. It means each change operation causes one IN_MODIFY event (it
may occur many times during manipulations with an open file) whereas
IN_CLOSE_WRITE is emitted only once (on closing the file).

Q: Is it better to use IN_MODIFY or IN_CLOSE_WRITE?

It varies from
case to case. Usually it is more suitable to use IN_CLOSE_WRITE
because if emitted the all changes on the appropriate file are safely
written inside the file. The IN_MODIFY event needn't mean that a file
change is finished (data may remain in memory buffers in the
application). On the other hand, many logs and similar files must be
monitored using IN_MODIFY - in such cases where these files are
permanently open and thus no IN_CLOSE_WRITE can be emitted.

source

---

Related Topics