Possible I/O Sync Issue with Ruby Script Under Nohup

Possible I/O sync issue with Ruby script under nohup?

This due to the fact when your STDOUT is not connected to tty, but to a file, block-sized buffering is used, if it is connected to tty, line-based buffering is used.

If you ran your command without NOHUP, the output (both stdout and stderr) would still be connected to the same tty, and it would be line-buffered. The default behavior of NOHUP is to write to nohup.out file. Since buffers are generally a lot bigger than lines, it will take much longer to "display" the results.

Odd Ruby output lag under nohup?

I can't say for sure, but it sounds like an IO sync issue. I/O is generally buffered for speed. RAM is a lot faster than disks, so reads and writes are temporarily stored until the data is requested by the code, or the disk is able to receive it. Ruby's IO class has the sync= method that lets you tell Ruby to immediately flush the buffer on a write.

This should work:

logger_file = open('/mnt/dbsdata/output.log', File::WRONLY | File::APPEND | File::CREAT)
logger_file.sync = true
LOGGER = Logger.new(logger_file)
LOGGER.level = Logger::INFO

For simplicity, you can create the correct mode you want for your output using:

    logger_file = File.open('/mnt/dbsdata/output.log', 'a')

Run Ruby script in the background

Have a look at screen which is a command-line utility. Start it with

screen

You will get a new shell which is detached. Start your script there with

ruby whatever.rb

And watch it run. Then hit Ctrl-A Ctrl-D and you should be back at your original shell. You can leave the ssh session now, and the script will continue running. At a later time, login to your box and type

screen -r

and you should be back to the detached shell.

If you use screen more than once, you will have to select the screen session by pid which is not so comfortable. To simplify, you can do

screen -S worker

to start the session and

screen -r worker

to resume it.

When running a ruby script via nohup, why doesn't puts write to nohup.out

I'm not familiar with running commands through nohup, but approaching this from a "I'm outputting content to a file and it's only being written after the script exits" type of problem, those are caused by the output being buffered.

So it's very likely that being run through nohup (and thus redirecting the puts output to nohup.out) you lost synchronization. You might need to flush occasionally or enable sync. Since puts is "equivalent to $stdout.puts":

$stdout.flush # run this, occasionally
# or just
$stdout.sync = true

FIP Error during file installation

The '500' is an HTTP status code. Looking at wikipedia...

500 Internal Server Error A generic error message, given when no more
specific message is suitable.[2]

FIP doesn't return any error information that might help someone hack or crack it's operation, so you'll need to look at the output of fipserver on the destination machine.

If you started fipserver using nohup so it wouldn't hang up when you log off, using

nohup ./fipserver 39393 &

then the output will be in a file named nohup.out, in the same directory.

Running a cron every 30 seconds

You have */30 in the minutes specifier - that means every minute but with a step of 30 (in other words, every half hour). Since cron does not go down to sub-minute resolutions, you will need to find another way.

One possibility, though it's a bit of a kludge^(a), is to have two jobs, one offset by 30 seconds:

# Need these to run on 30-sec boundaries, keep commands in sync.
* * * * *              /path/to/executable param1 param2
* * * * * ( sleep 30 ; /path/to/executable param1 param2 )

You'll see I've added comments and formatted to ensure it's easy to keep them synchronised.

Both cron jobs actually run every minute but the latter one will wait half a minute before executing the "meat" of the job, /path/to/executable.

For other (non-cron-based) options, see the other answers here, particularly the ones mentioning fcron and systemd. These are probably preferable assuming your system has the ability to use them (such as installing fcron or having a distro with systemd in it).

---

If you don't want to use the kludgy solution, you can use a loop-based solution with a small modification. You'll still have to manage keeping your process running in some form but, once that's sorted, the following script should work:

#!/bin/env bash

# Debug code to start on minute boundary and to
# gradually increase maximum payload duration to
# see what happens when the payload exceeds 30 seconds.

((maxtime = 20))
while [[ "$(date +%S)" != "00" ]]; do true; done

while true; do
    # Start a background timer BEFORE the payload runs.

    sleep 30 &

    # Execute the payload, some random duration up to the limit.
    # Extra blank line if excess payload.

    ((delay = RANDOM % maxtime + 1))
    ((maxtime += 1))
    echo "$(date) Sleeping for ${delay} seconds (max ${maxtime})."
    [[ ${delay} -gt 30 ]] && echo
    sleep ${delay}

    # Wait for timer to finish before next cycle.

    wait
done

The trick is to use a sleep 30 but to start it in the background before your payload runs. Then, after the payload is finished, just wait for the background sleep to finish.

If the payload takes n seconds (where n <= 30), the wait after the payload will then be 30 - n seconds. If it takes more than 30 seconds, then the next cycle will be delayed until the payload is finished, but no longer.

You'll see that I have debug code in there to start on a one-minute boundary to make the output initially easier to follow. I also gradually increase the maximum payload time so you'll eventually see the payload exceed the 30-second cycle time (an extra blank line is output so the effect is obvious).

A sample run follows (where cycles normally start 30 seconds after the previous cycle):

Tue May 26 20:56:00 AWST 2020 Sleeping for 9 seconds (max 21).
Tue May 26 20:56:30 AWST 2020 Sleeping for 19 seconds (max 22).
Tue May 26 20:57:00 AWST 2020 Sleeping for 9 seconds (max 23).
Tue May 26 20:57:30 AWST 2020 Sleeping for 7 seconds (max 24).
Tue May 26 20:58:00 AWST 2020 Sleeping for 2 seconds (max 25).
Tue May 26 20:58:30 AWST 2020 Sleeping for 8 seconds (max 26).
Tue May 26 20:59:00 AWST 2020 Sleeping for 20 seconds (max 27).
Tue May 26 20:59:30 AWST 2020 Sleeping for 25 seconds (max 28).
Tue May 26 21:00:00 AWST 2020 Sleeping for 5 seconds (max 29).
Tue May 26 21:00:30 AWST 2020 Sleeping for 6 seconds (max 30).
Tue May 26 21:01:00 AWST 2020 Sleeping for 27 seconds (max 31).
Tue May 26 21:01:30 AWST 2020 Sleeping for 25 seconds (max 32).
Tue May 26 21:02:00 AWST 2020 Sleeping for 15 seconds (max 33).
Tue May 26 21:02:30 AWST 2020 Sleeping for 10 seconds (max 34).
Tue May 26 21:03:00 AWST 2020 Sleeping for 5 seconds (max 35).
Tue May 26 21:03:30 AWST 2020 Sleeping for 35 seconds (max 36).

Tue May 26 21:04:05 AWST 2020 Sleeping for 2 seconds (max 37).
Tue May 26 21:04:35 AWST 2020 Sleeping for 20 seconds (max 38).
Tue May 26 21:05:05 AWST 2020 Sleeping for 22 seconds (max 39).
Tue May 26 21:05:35 AWST 2020 Sleeping for 18 seconds (max 40).
Tue May 26 21:06:05 AWST 2020 Sleeping for 33 seconds (max 41).

Tue May 26 21:06:38 AWST 2020 Sleeping for 31 seconds (max 42).

Tue May 26 21:07:09 AWST 2020 Sleeping for 6 seconds (max 43).

If you want to avoid the kludgy solution, this is probably better. You'll still need a cron job (or equivalent) to periodically detect if this script is running and, if not, start it. But the script itself then handles the timing.

---

^(a) Some of my workmates would say that kludges are my specialty :-)

---

Related Topics