Gnu Time and Formatting Output

Gnu time and formatting output

Bash for one has a shell builtin named time. One way to get past it is to type command time - command will ignore the builtins and run the time program from your $PATH. Another way is alias time=/usr/bin/time. On the other hand the bash builtin respects environment variable TIMEFORMAT.

Storing output of GNU time command and script

The time command writes its output to stderr, so as to not pollute the stdout of the process being timed.

Notice that in your second example, you're redirecting both stdout and stderr with &>.

If you want to also pipe stderr to tee, use |&:

(/usr/bin/time -f '%P %M %E %S %U' python Script.py arg1 arg2) |& tee Script.output

How do I format output of time in ms for more precision?

Bash has a built-in time command which is used in your case.

The time described in man time (usually installed under /usr/bin/time) can be called as follows

command time -f "%S" ./main

it is too fast, so I need more decimal precision.
I would rather see: 0.000015

Neither bash's time nor GNU time seem to be able to output times with such high precision. And if you really think about it there is no need to. There is always background noise and starting a process takes time too. If you measure such short time intervals you mostly measure anything but the time your program actually takes.

If you really have to, you could measure the wall clock time in bash 5 ✱:

t=$EPOCHREALTIME
./main
t="$EPOCHREALTIME - $t" # measure time before starting bc
bc <<< "$t"

However, it would be better to do this directly inside your program. If you could repeat your actual program 1000 times there and measure the complete time you would get a somewhat usable measurement.

✱ On my system measuring with bash added around 0.000008 s of overhead. The overhead was measured with

$ for i in {1..10000}; do t=$EPOCHREALTIME; t="$EPOCHREALTIME - $t"; bc <<< "$t"; done | sponge | datamash min 1 median 1 max 1
4e-06   8e-06   8.1e-05

Custom format for time command

You could use the date command to get the current time before and after performing the work to be timed and calculate the difference like this:

#!/bin/bash

# Get time as a UNIX timestamp (seconds elapsed since Jan 1, 1970 0:00 UTC)
T="$(date +%s)"

# Do some work here
sleep 2

T="$(($(date +%s)-T))"
echo "Time in seconds: ${T}"

printf "Pretty format: %02d:%02d:%02d:%02d\n" "$((T/86400))" "$((T/3600%24))" "$((T/60%60))" "$((T%60))""

Notes:
$((...)) can be used for basic arithmetic in bash – caution: do not put spaces before a minus - as this might be interpreted as a command-line option.

See also: http://tldp.org/LDP/abs/html/arithexp.html

EDIT:
Additionally, you may want to take a look at sed to search and extract substrings from the output generated by time.

EDIT:

Example for timing with milliseconds (actually nanoseconds but truncated to milliseconds here). Your version of date has to support the %N format and bash should support large numbers.

# UNIX timestamp concatenated with nanoseconds
T="$(date +%s%N)"

# Do some work here
sleep 2

# Time interval in nanoseconds
T="$(($(date +%s%N)-T))"
# Seconds
S="$((T/1000000000))"
# Milliseconds
M="$((T/1000000))"

echo "Time in nanoseconds: ${T}"
printf "Pretty format: %02d:%02d:%02d:%02d.%03d\n" "$((S/86400))" "$((S/3600%24))" "$((S/60%60))" "$((S%60))" "${M}"

DISCLAIMER:
My original version said

M="$((T%1000000000/1000000))"

but this was edited out because it apparently did not work for some people whereas the new version reportedly did. I did not approve of this because I think that you have to use the remainder only but was outvoted.

Choose whatever fits you.

/usr/bin/time --format output elapsed time in milliseconds

One possibility is to use the date command:

ts=$(date +%s%N) ; my_command ; tt=$((($(date +%s%N) - $ts)/1000000)) ; echo "Time taken: $tt milliseconds"

%N should return nanoseconds, and 1 millisecond is 1000000 nanosecond, hence by division would return the time taken to execute my_command in milliseconds.

NOTE that the %N is not supported on all systems, but most of them.

GNU time returns different signal than it prints out

GNU time's %x only makes sense when the process exits normally, not killed by signals.

[STEP 101] $ /usr/bin/time -f "%%x = %x" bash -c 'exit 2'; echo '$? = '$?
Command exited with non-zero status 2
%x = 2
$? = 2
[STEP 102] $ /usr/bin/time -f "%%x = %x" bash -c 'exit 137'; echo '$? = '$?
Command exited with non-zero status 137
%x = 137
$? = 137
[STEP 103] $ /usr/bin/time -f "%%x = %x" bash -c 'kill -KILL $$'; echo '$? = '$?
Command terminated by signal 9
%x = 0
$? = 137
[STEP 104] $

For time timeout -s SIGKILL 2s sleep 10, timeout exits normally with 137, it's not killed by SIGKILL, just like bash -c 'exit 137' in my example.

---

UPDATE:

Took a look at time's source code and found out %x is blindly calling WEXITSTATUS() no matter the process exits normally or not.

655             case 'x':           /* Exit status.  */
656               fprintf (fp, "%d", WEXITSTATUS (resp->waitstatus));
657               break;

In the Git master it added new %Tx:

549             case 'T':
550               switch (*++fmt)
551                 {
...
575                 case 'x': /* exit code IF terminated normally */
576                   if (WIFEXITED (resp->waitstatus))
577                     fprintf (fp, "%d", WEXITSTATUS (resp->waitstatus));
578                   break;

And from the Git master's time --help output:

  ...

  %Tt  exit type (normal/signalled)
  %Tx  numeric exit code IF exited normally
  %Tn  numeric signal code IF signalled
  %Ts  signal name IF signalled
  %To  'ok' IF exited normally with code zero

  ...

How can I get GNU Lib C TZ format output from NSTimeZone?

This was a fun exploration, largely because fitting the data into just the right format is pretty complex. Problem components:

• We need the "current" TZ database rule that applies for a given time zone. This is a bit of a loaded concept, because:

  1. Darwin platforms don't actually use the TZ database directly for most applications, but instead use ICU's time zone database, which comes in a different format and is more complex. Even if you produce a string in this format, it's not necessarily descriptive of the actual time behavior on device

  2. While it is possible to read and parse the TZ database on iOS dynamically, the TZ database itself is not guaranteed to store information in the format needed here. rfc8536, the RFC governing the Time Zone Information Format says the following about the format you want:

     The TZ string in a version 3 TZif file MAY use the following extensions to POSIX TZ strings. These extensions are described using the terminology of Section 8.3 of the "Base Definitions" volume of [POSIX].

     Example: <-03>3<-02>,M3.5.0/-2,M10.5.0/-1
     
     Example: EST5EDT,0/0,J365/25

     While spelunking through the iOS TZ database, I found some database entries that do offer a rule at the end of the file in this format, but they appear to be a minority. You could parse these dynamically, but it's likely not worth it

  So, we need to use APIs to produce a string in this format.

• In order to produce a "rule" that is at least approximately correct on a given date, you need to know information about DST transitions around that date. This is an extremely thorny topic, because DST rules change all the time, and don't always make as much sense as you'd hope. At the very least:

  • Many time zones in the Northern hemisphere observe DST beginning in the spring and ending in the fall
  • Many time zones in the Southern hemisphere observe DST beginning in the fall and ending in the spring
  • Some time zones don't observe DST (are in standard time year-round)
  • Some time zones don't observe DST and are in daylight time year-round

  Because the rules are so complex, the rest of this answer assumes you're okay with producing a "good enough" answer that represents a specific date in time, and is willing to send further strings to your clock some time in the future when corrections are needed. e.g., to describe "now", we will be assuming that producing a rule based off of the last DST transition (if any) and the next DST transition (if any) is "good enough", but this may not work for all situations in many time zones

• Foundation provides DST transition information on TimeZone in the form of TimeZone.nextDaylightSavingTimeTransition/TimeZone.nextDaylightSavingTimeTransition(after:). Frustratingly, however, there's no way to get information about previous DST transitions, so we'll need to rectify that:

  • Foundation's localization support (including calendars and time zones) is based directly on the ICU library, which ships internally on all Apple platforms. ICU does provide a way to get information about previous DST transitions, but Foundation just doesn't offer this as API, so we'll need to expose it ourselves

  • ICU is a semi-private library on Apple platforms. The library is guaranteed to be present, and Xcode will offer you libicucore.tbd to link against in <Project> > <Target> > Build Phases > Link Binary with Libraries, but the actual headers and symbols are not directly exposed to apps. You can successfully link against libicucore, but you'll need to forward-declare the functionality we need in an Obj-C header imported into Swift

  • Somewhere in the Swift project, we need to expose the following ICU functionality:

    #include <stdint.h>
    
    typedef void * _Nonnull UCalendar;
    typedef double UDate;
    typedef int8_t UBool;
    typedef uint16_t UChar;
    
    typedef enum UTimeZoneTransitionType {
        UCAL_TZ_TRANSITION_NEXT,
        UCAL_TZ_TRANSITION_NEXT_INCLUSIVE,
        UCAL_TZ_TRANSITION_PREVIOUS,
        UCAL_TZ_TRANSITION_PREVIOUS_INCLUSIVE,
    } UTimeZoneTransitionType;
    
    typedef enum UCalendarType {
        UCAL_TRADITIONAL,
        UCAL_DEFAULT,
        UCAL_GREGORIAN,
    } UCalendarType;
    
    typedef enum UErrorCode {
        U_ZERO_ERROR = 0,
    } UErrorCode;
    
    UCalendar * _Nullable ucal_open(const UChar *zoneID, int32_t len, const char *locale, UCalendarType type, UErrorCode *status);
    void ucal_setMillis(const UCalendar * _Nonnull cal, UDate date, UErrorCode * _Nonnull status);
    UBool ucal_getTimeZoneTransitionDate(const UCalendar * _Nonnull cal, UTimeZoneTransitionType type, UDate * _Nonnull transition, UErrorCode * _Nonnull status);
    

    These are all forward declarations / constants, so no need to worry about implementation (since we get that by linking against libicucore).

  • You can see the values in UTimeZoneTransitionType — TimeZone.nextDaylightSavingTimeTransition just calls ucal_getTimeZoneTransitionDate with a value of UCAL_TZ_TRANSITION_NEXT, so we can offer roughly the same functionality by calling the method with UCAL_TZ_TRANSITION_PREVIOUS:

    extension TimeZone {
        func previousDaylightSavingTimeTransition(before: Date) -> Date? {
            // We _must_ pass a status variable for `ucal_open` to write into, but the actual initial
            // value doesn't matter.
            var status = U_ZERO_ERROR
    
            // `ucal_open` requires the time zone identifier be passed in as UTF-16 code points.
            // `String.utf16` doesn't offer a contiguous buffer for us to pass directly into `ucal_open`
            // so we have to create our own by copying the values into an `Array`, then
            let timeZoneIdentifier = Array(identifier.utf16)
            guard let calendar = Locale.current.identifier.withCString({ localeIdentifier in
                ucal_open(timeZoneIdentifier, // implicit conversion of Array to a pointer, but convenient!
                          Int32(timeZoneIdentifier.count),
                          localeIdentifier,
                          UCAL_GREGORIAN,
                          &status)
            }) else {
                // Figure out some error handling here -- we failed to find a "calendar" for this time
                // zone; i.e., there's no time zone date for this time zone.
                //
                // With more enum cases copied from `UErrorCode` you may find a good way to report an
                // error here if needed. `u_errorName` turns a `UErrorCode` into a string.
                return nil
            }
    
            // `UCalendar` functions operate on the calendar's current timestamp, so we have to apply
            // `date` to it. `UDate`s are the number of milliseconds which have passed since January 1,
            // 1970, while `Date` offers its time interval in seconds.
            ucal_setMillis(calendar, before.timeIntervalSince1970 * 1000.0, &status)
    
            var result: UDate = 0
            guard ucal_getTimeZoneTransitionDate(calendar, UCAL_TZ_TRANSITION_PREVIOUS, &result, &status) != 0 else {
                // Figure out some error handling here -- same as above (check status).
                return nil
            }
    
            // Same transition but in reverse.
            return Date(timeIntervalSince1970: result / 1000.0)
        }
    }
    

So, with all of this in place, we can fill out a crude method to produce a string in the format you need:

extension TimeZone {
    struct Transition {
        let abbreviation: String
        let offsetFromGMT: Int
        let date: Date
        let components: DateComponents

        init(for timeZone: TimeZone, on date: Date, using referenceCalendar: Calendar) {
            abbreviation = timeZone.abbreviation(for: date) ?? ""
            offsetFromGMT = timeZone.secondsFromGMT(for: date)
            self.date = date
            components = referenceCalendar.dateComponents([.month, .weekOfMonth, .weekdayOrdinal, .hour, .minute, .second], from: date)
        }
    }

    func approximateTZEntryRule(on date: Date = Date(), using calendar: Calendar? = nil) -> String? {
        var referenceCalendar = calendar ?? Calendar(identifier: .gregorian)
        referenceCalendar.timeZone = self

        guard let year = referenceCalendar.dateInterval(of: .year, for: date) else {
            return nil
        }

        // If no prior DST transition has ever occurred, we're likely in a time zone which is either
        // standard or daylight year-round. We'll cap the definition here to the very start of the
        // year.
        let previousDSTTransition = Transition(for: self, on: previousDaylightSavingTimeTransition(before: date) ?? year.start, using: referenceCalendar)

        // Same with the following DST transition -- if no following DST transition will ever come,
        // we'll cap it to the end of the year.
        let nextDSTTransition = Transition(for: self, on: nextDaylightSavingTimeTransition(after: date) ?? year.end, using: referenceCalendar)

        let standardToDaylightTransition: Transition
        let daylightToStandardTransition: Transition
        if isDaylightSavingTime(for: date) {
            standardToDaylightTransition = previousDSTTransition
            daylightToStandardTransition = nextDSTTransition
        } else {
            standardToDaylightTransition = nextDSTTransition
            daylightToStandardTransition = previousDSTTransition
        }

        let standardAbbreviation = daylightToStandardTransition.abbreviation
        let standardOffset = formatOffset(daylightToStandardTransition.offsetFromGMT)
        let daylightAbbreviation = standardToDaylightTransition.abbreviation
        let startDate = formatDate(components: standardToDaylightTransition.components)
        let endDate = formatDate(components: daylightToStandardTransition.components)
        return "\(standardAbbreviation)\(standardOffset)\(daylightAbbreviation),\(startDate),\(endDate)"
    }

    /* These formatting functions can be way better. You'll also want to actually cache the
       DateComponentsFormatter somewhere.
     */

    func formatOffset(_ dateComponents: DateComponents) -> String {
        let formatter = DateComponentsFormatter()
        formatter.allowedUnits = [.hour, .minute, .second]
        formatter.zeroFormattingBehavior = .dropTrailing
        return formatter.string(from: dateComponents) ?? ""
    }

    func formatOffset(_ seconds: Int) -> String {
        return formatOffset(DateComponents(second: seconds))
    }

    func formatDate(components: DateComponents) -> String {
        let month = components.month ?? 0
        let week = components.weekOfMonth ?? 0
        let day = components.weekdayOrdinal ?? 0
        let offset = formatOffset(DateComponents(hour: components.hour, minute: components.minute, second: components.second))
        return "M\(month).\(week).\(day)/\(offset)"
    }
}

Note that there's lots to improve here, especially in clarity and performance. (Formatters are notoriously expensive, so you'll definitely want to cache them.) This also currently only produces dates in the expanded form "Mm.w.d" and not Julian days, but that can be bolted on. The code also assumes that it's "good enough" to restrict unbounded rules to the current calendar year, since this is what the GNU C library docs seem to imply about e.g. time zones which are always in standard/daylight time. (This also doesn't recognize well-known time zones like GMT/UTC, which might be sufficient to just write out as "GMT".)

I have not extensively tested this code for various time zones, and the above code should be considered a basis for additional iteration. For my time zone of America/New_York, this produces "EST-5EDT,M3.3.2/3,M11.2.1/1", which appears correct to me at first glance, but many other edge cases might be good to explore:

• Boundary conditions around the start/end of the year
• Giving a date which exactly matches a DST transition (consider TRANSITION_PREVIOUS vs. TRANSITION_PREVIOUS_INCLUSIVE)
• Time zones which are always standard/daylight
• Non-standard daylight/timezone offsets

There's a lot more to this, and in general, I'd recommend trying to find an alternative method of setting a time on this device (preferably using named time zones), but this might hopefully at least get you started.

Extracting output of Linux time command into a file

GNU time has -o FILE parameter that you can use to output the timing into a file. You can process the file, extract the timing from the file and put it somewhere in CSV or such in a loop.

It also has -f format option that lets you customize the output format of the time.

So, in your case it could go like this:

/usr/bin/time -o mytime.log -f "%e" scp -i k30.pem ubuntu@ec2-XX-YY-ZZ-WW.compute-1.amazonaws.com:/home/ubuntu/pics/* /home/pi/

In this command the "%e" format makes the GNU time print only elapsed real time in seconds.

GNU Smalltalk - Formatting output and disabling GC messages

Garbage collection messages are disabled automatically when you use your program as a script (i.e. with gst -f), or you can use --no-gc-message (short option -g).

printNl is a programmer-oriented printing message. To print without quotes use display/displayNl. Similarly, characters will be printed without dollar signs.

---

Related Topics