Swift: How to Read Standard Output in a Child Process Without Waiting for Process to Finish

Swift: How to read standard output in a child process without waiting for process to finish

You are reading synchronously on the main thread, therefore the UI is not updated until the function returns to the main loop.

There are (at least) two possible approaches to solve the problem:

Do the reading from the pipe on a background thread (e.g. by dispatching it to a background queue – but don't forget
to dispatch the UI updates to the main thread again).
Use notifications to read asynchronously from the pipe (see
Real time NSTask output to NSTextView with Swift for an example).

Collect python output in Swift

Solution

As explained in this page
https://www.hackingwithswift.com/example-code/system/how-to-run-an-external-program-using-process

and shown in the answer suggested by Willeke, you can use Pipe() to do that.

I changed the code as shown below.

override func viewDidLoad() {
    super.viewDidLoad()
    let task = Process()
    task.executableURL = URL(fileURLWithPath: "/usr/bin/python")
    let filename = Bundle.main.path(forResource: "eject", ofType: "py")
    task.arguments = [filename!]
    let outputPipe = Pipe()
    task.standardOutput = outputPipe
    
       do{
           try task.run()
       } catch {
           print("error")
       }
    
    let outputData = outputPipe.fileHandleForReading.readDataToEndOfFile()
    let output = String(decoding: outputData, as: UTF8.self)
    print(output)
    
   }

Be sure to put this

let outputPipe = Pipe()
task.standardOutput = outputPipe

before the task.run() command and this

let outputData = outputPipe.fileHandleForReading.readDataToEndOfFile()
let output = String(decoding: outputData, as: UTF8.self)
print(output)

after it.

Real time NSTask output to NSTextView with Swift

(See Patrick F.'s answer for an update to Swift 3/4.)

You can read asynchronously from a pipe, using notifications.
Here is a simple example demonstrating how it works, hopefully that
helps you to get started:

let task = NSTask()
task.launchPath = "/bin/sh"
task.arguments = ["-c", "echo 1 ; sleep 1 ; echo 2 ; sleep 1 ; echo 3 ; sleep 1 ; echo 4"]

let pipe = NSPipe()
task.standardOutput = pipe
let outHandle = pipe.fileHandleForReading
outHandle.waitForDataInBackgroundAndNotify()

var obs1 : NSObjectProtocol!
obs1 = NSNotificationCenter.defaultCenter().addObserverForName(NSFileHandleDataAvailableNotification,
    object: outHandle, queue: nil) {  notification -> Void in
        let data = outHandle.availableData
        if data.length > 0 {
            if let str = NSString(data: data, encoding: NSUTF8StringEncoding) {
                print("got output: \(str)")
            }
            outHandle.waitForDataInBackgroundAndNotify()
        } else {
            print("EOF on stdout from process")
            NSNotificationCenter.defaultCenter().removeObserver(obs1)
        }
}

var obs2 : NSObjectProtocol!
obs2 = NSNotificationCenter.defaultCenter().addObserverForName(NSTaskDidTerminateNotification,
    object: task, queue: nil) { notification -> Void in
        print("terminated")
        NSNotificationCenter.defaultCenter().removeObserver(obs2)
}

task.launch()

Instead of print("got output: \(str)") you can append the received
string to your text view.

The above code assumes that a runloop is active (which is the case
in a default Cocoa application).

How to wait for exit of non-children processes

Nothing equivalent to wait(). The usual practice is to poll using kill(pid, 0) and looking for return value -1 and errno of ESRCH to indicate that the process is gone.

Update: Since linux kernel 5.3 there is a pidfd_open syscall, which creates an fd for a given pid, which can be polled to get notification when pid has exited.

How do I wrap up a series of related NSTask operations into discrete functions

waitUntilExit returns when the SIGCHILD signal has been received
to indicate that the child process has terminated. The notification
block is executed when EOF is read from the pipe to the child process.
It is not specified which of these events occurs first.

Therefore you have to wait for both. There are several possible solutions,
here is one using a "signalling semaphore", you could also use
a "dispatch group".

Another error in your code is that the observer is never removed.

func runCommit() -> Bool {

    let commitTask = NSTask()
    commitTask.standardOutput = NSPipe()
    commitTask.launchPath = gitPath
    commitTask.arguments = ["commit", "-m", "Initial Commit"]
    commitTask.currentDirectoryPath = demoProjectURL.path!

    commitTask.standardOutput!.fileHandleForReading.readToEndOfFileInBackgroundAndNotify()

    let sema = dispatch_semaphore_create(0)
    var obs : NSObjectProtocol!
    obs = nc.addObserverForName(NSFileHandleReadToEndOfFileCompletionNotification,
        object: commitTask.standardOutput!.fileHandleForReading, queue: nil) {
            (note) -> Void in
            // Get data and log it.
            if let data = note.userInfo?[NSFileHandleNotificationDataItem] as? NSData,
                let string = String(data: data, encoding: NSUTF8StringEncoding) {
                    print(string)
            }
            // Signal semaphore.
            dispatch_semaphore_signal(sema)
            nc.removeObserver(obs)
    }

    commitTask.launch()
    // Wait for process to terminate.
    commitTask.waitUntilExit()
    // Wait for semaphore to be signalled.
    dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER)
    let retval = commitTask.terminationStatus == EXIT_SUCCESS
    return retval
}

How to support an async callback with timeout in swift actor

Before I get to the timeout question, we should probably talk about how to wrap the Process within Swift concurrency.

One pattern would be to use AsyncSequence (i.e., an AsyncStream) for stdout:

actor ProcessWithStream {
    private let process = Process()
    private let stdin = Pipe()
    private let stdout = Pipe()
    private let stderr = Pipe()
    private var buffer = Data()

    init(url: URL) {
        process.standardInput = stdin
        process.standardOutput = stdout
        process.standardError = stderr
        process.executableURL = url
    }

    func start() throws {
        try process.run()
    }

    func terminate() {
        process.terminate()
    }

    func send(_ string: String) {
        guard let data = "\(string)\n".data(using: .utf8) else { return }
        stdin.fileHandleForWriting.write(data)
    }

    func stream() -> AsyncStream<Data> {
        AsyncStream(Data.self) { continuation in
            stdout.fileHandleForReading.readabilityHandler = { handler in
                continuation.yield(handler.availableData)
            }
            process.terminationHandler = { handler in
                continuation.finish()
            }
        }
    }
}

Then you can for await that stream:

let process = ProcessWithStream(url: url)

override func viewDidLoad() {
    super.viewDidLoad()

    Task {
        try await startStream()
        print("done")
    }
}

@IBAction func didTapSend(_ sender: Any) {
    let string = textField.stringValue
    Task {
        await process.send(string)
    }
    textField.stringValue = ""
}

func startStream() async throws {
    try await process.start()
    let stream = await process.stream()

    for await data in stream {
        if let string = String(data: data, encoding: .utf8) {
            print(string, terminator: "")
        }
    }
}

This is a simple approach. And it looks fine (because I am printing responses without a terminator).

But one needs to be careful because readabilityHandler will not always be called with the full Data of some particular output. It might be broken up or split across separate calls to the reachabilityhandler.

Another pattern would be to use lines, which avoids the problem of reachabilityHandler possibly being called multiple times for a given output:

actor ProcessWithLines {
    private let process = Process()
    private let stdin = Pipe()
    private let stdout = Pipe()
    private let stderr = Pipe()
    private var buffer = Data()
    private(set) var lines: AsyncLineSequence<FileHandle.AsyncBytes>?

    init(url: URL) {
        process.standardInput = stdin
        process.standardOutput = stdout
        process.standardError = stderr
        process.executableURL = url
    }

    func start() throws {
        lines = stdout.fileHandleForReading.bytes.lines
        try process.run()
    }

    func terminate() {
        process.terminate()
    }

    func send(_ string: String) {
        guard let data = "\(string)\n".data(using: .utf8) else { return }
        stdin.fileHandleForWriting.write(data)
    }
}

Then you can do:

let process = ProcessWithLines(url: url)

override func viewDidLoad() {
    super.viewDidLoad()

    Task {
        try await startStream()
        print("done")
    }
}

@IBAction func didTapSend(_ sender: Any) {
    let string = textField.stringValue
    Task {
        await process.send(string)
    }
    textField.stringValue = ""
}

func startStream() async throws {
    try await process.start()
    guard let lines = await process.lines else { return }

    for try await line in lines {
        print(line)
    }
}

This avoids the breaking of responses mid-line.

You asked:

How to support ... timeout in swift actor

The pattern is to wrap the request in a Task, and then start a separate task that will cancel that prior task after a Task.sleep interval.

But this is going to be surprisingly complicated in this case, because you have to coordinate that with the separate Process which will otherwise still proceed, unaware that the Task has been canceled. That can theoretically lead to problems (e.g., the process gets backlogged, etc.).

I would advise integrating the timeout logic in the app invoked by the Process, rather than trying to have the caller handle that. It can be done (e.g. maybe write the process app to capture and handle SIGINT and then the caller can call interrupt on the Process). But it is going to be complicated and, most likely, brittle.

Why does read() block and wait forever in parent process despite the writing end of pipe being closed?

Since the child demonstrably reaches ...

                printf("client here\n");

... but seems not to reach ...

                printf("client just read from pipe\n");

... we can suppose that it blocks indefinitely on one of the two read() calls between. With the right timing, that explains why the parent blocks on its own read() from the pipe. But how and why does that blocking occur?

There are at least three significant semantic errors in your program:

pipes do not work well for bidirectional communication. It is possible, for example, for a process to read back the bytes that it wrote itself and intended for a different process. If you want bidirectional communication then use two pipes. In your case, I think that would have avoided the apparent deadlock, though it would not, by itself, have made the program work correctly.
write and read do not necessarily transfer the full number of bytes requested, and short reads and writes are not considered erroneous. On success, these functions return the number of bytes transferred, and if you want to be sure to transfer a specific number of bytes then you need to run the read or write in a loop, using the return values to track progress through the buffer being transferred. Or use fread() and fwrite() instead.
Pipes convey undifferentiated streams of bytes. That is, they are not message oriented. It is not safe to assume that reads from a pipe will be paired with writes to the pipe, so that each read receives exactly the bytes written by one write. Yet your code depends on that to happen.

Here's a plausible failure scenario that could explain your observations:

The parent:

fork()s the child.
after some time performs two writes to the pipe, one from variable directory and the other from variable octal. At least the first of those is a short write.
closes its copy of the write end of the pipe.
blocks attempting to read from the pipe.

The child:

reads all the bytes written via its first read (into its copy of directory).
blocks on its second read(). It can do this despite the parent closing its copy of the write end, because the write end of the pipe is still open in the child.

You then have a deadlock. Both ends of the pipe are open in at least one process, the pipe is empty, and both processes are blocked trying to read bytes that can never arrive.

There are other possibilities that arrive at substantially the same place, too, some of them not relying on a short write.

Swift: How to Read Standard Output in a Child Process Without Waiting for Process to Finish