Execute Command Line and Return Command Output

Running shell command and capturing the output

In all officially maintained versions of Python, the simplest approach is to use the subprocess.check_output function:

>>> subprocess.check_output(['ls', '-l'])
b'total 0\n-rw-r--r--  1 memyself  staff  0 Mar 14 11:04 files\n'

check_output runs a single program that takes only arguments as input.¹ It returns the result exactly as printed to stdout. If you need to write input to stdin, skip ahead to the run or Popen sections. If you want to execute complex shell commands, see the note on shell=True at the end of this answer.

The check_output function works in all officially maintained versions of Python. But for more recent versions, a more flexible approach is available.

Modern versions of Python (3.5 or higher): run

If you're using Python 3.5+, and do not need backwards compatibility, the new run function is recommended by the official documentation for most tasks. It provides a very general, high-level API for the subprocess module. To capture the output of a program, pass the subprocess.PIPE flag to the stdout keyword argument. Then access the stdout attribute of the returned CompletedProcess object:

>>> import subprocess
>>> result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE)
>>> result.stdout
b'total 0\n-rw-r--r--  1 memyself  staff  0 Mar 14 11:04 files\n'

The return value is a bytes object, so if you want a proper string, you'll need to decode it. Assuming the called process returns a UTF-8-encoded string:

>>> result.stdout.decode('utf-8')
'total 0\n-rw-r--r--  1 memyself  staff  0 Mar 14 11:04 files\n'

This can all be compressed to a one-liner if desired:

>>> subprocess.run(['ls', '-l'], stdout=subprocess.PIPE).stdout.decode('utf-8')
'total 0\n-rw-r--r--  1 memyself  staff  0 Mar 14 11:04 files\n'

If you want to pass input to the process's stdin, you can pass a bytes object to the input keyword argument:

>>> cmd = ['awk', 'length($0) > 5']
>>> ip = 'foo\nfoofoo\n'.encode('utf-8')
>>> result = subprocess.run(cmd, stdout=subprocess.PIPE, input=ip)
>>> result.stdout.decode('utf-8')
'foofoo\n'

You can capture errors by passing stderr=subprocess.PIPE (capture to result.stderr) or stderr=subprocess.STDOUT (capture to result.stdout along with regular output). If you want run to throw an exception when the process returns a nonzero exit code, you can pass check=True. (Or you can check the returncode attribute of result above.) When security is not a concern, you can also run more complex shell commands by passing shell=True as described at the end of this answer.

Later versions of Python streamline the above further. In Python 3.7+, the above one-liner can be spelled like this:

>>> subprocess.run(['ls', '-l'], capture_output=True, text=True).stdout
'total 0\n-rw-r--r--  1 memyself  staff  0 Mar 14 11:04 files\n'

Using run this way adds just a bit of complexity, compared to the old way of doing things. But now you can do almost anything you need to do with the run function alone.

Older versions of Python (3-3.4): more about check_output

If you are using an older version of Python, or need modest backwards compatibility, you can use the check_output function as briefly described above. It has been available since Python 2.7.

subprocess.check_output(*popenargs, **kwargs)  

It takes takes the same arguments as Popen (see below), and returns a string containing the program's output. The beginning of this answer has a more detailed usage example. In Python 3.5+, check_output is equivalent to executing run with check=True and stdout=PIPE, and returning just the stdout attribute.

You can pass stderr=subprocess.STDOUT to ensure that error messages are included in the returned output. When security is not a concern, you can also run more complex shell commands by passing shell=True as described at the end of this answer.

If you need to pipe from stderr or pass input to the process, check_output won't be up to the task. See the Popen examples below in that case.

Complex applications and legacy versions of Python (2.6 and below): Popen

If you need deep backwards compatibility, or if you need more sophisticated functionality than check_output or run provide, you'll have to work directly with Popen objects, which encapsulate the low-level API for subprocesses.

The Popen constructor accepts either a single command without arguments, or a list containing a command as its first item, followed by any number of arguments, each as a separate item in the list. shlex.split can help parse strings into appropriately formatted lists. Popen objects also accept a host of different arguments for process IO management and low-level configuration.

To send input and capture output, communicate is almost always the preferred method. As in:

output = subprocess.Popen(["mycmd", "myarg"], 
                          stdout=subprocess.PIPE).communicate()[0]

Or

>>> import subprocess
>>> p = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, 
...                                    stderr=subprocess.PIPE)
>>> out, err = p.communicate()
>>> print out
.
..
foo

If you set stdin=PIPE, communicate also allows you to pass data to the process via stdin:

>>> cmd = ['awk', 'length($0) > 5']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
...                           stderr=subprocess.PIPE,
...                           stdin=subprocess.PIPE)
>>> out, err = p.communicate('foo\nfoofoo\n')
>>> print out
foofoo

Note Aaron Hall's answer, which indicates that on some systems, you may need to set stdout, stderr, and stdin all to PIPE (or DEVNULL) to get communicate to work at all.

In some rare cases, you may need complex, real-time output capturing. Vartec's answer suggests a way forward, but methods other than communicate are prone to deadlocks if not used carefully.

As with all the above functions, when security is not a concern, you can run more complex shell commands by passing shell=True.

Notes

1. Running shell commands: the shell=True argument

Normally, each call to run, check_output, or the Popen constructor executes a single program. That means no fancy bash-style pipes. If you want to run complex shell commands, you can pass shell=True, which all three functions support. For example:

>>> subprocess.check_output('cat books/* | wc', shell=True, text=True)
' 1299377 17005208 101299376\n'

However, doing this raises security concerns. If you're doing anything more than light scripting, you might be better off calling each process separately, and passing the output from each as an input to the next, via

run(cmd, [stdout=etc...], input=other_output)

Or

Popen(cmd, [stdout=etc...]).communicate(other_output)

The temptation to directly connect pipes is strong; resist it. Otherwise, you'll likely see deadlocks or have to do hacky things like this.

How to execute a command and get return code stdout and stderr of command in C++

From the man-page of popen:

The pclose() function waits for the associated process to terminate  and returns the exit status of the command as returned by wait4(2).

So, calling pclose() yourself (instead of using std::shared_ptr<>'s destructor-magic) will give you the return code of your process (or block if the process has not terminated).

std::string exec(const char* cmd) {
    std::array<char, 128> buffer;
    std::string result;

    auto pipe = popen(cmd, "r"); // get rid of shared_ptr

    if (!pipe) throw std::runtime_error("popen() failed!");

    while (!feof(pipe)) {
        if (fgets(buffer.data(), 128, pipe) != nullptr)
            result += buffer.data();
    }

    auto rc = pclose(pipe);

    if (rc == EXIT_SUCCESS) { // == 0

    } else if (rc == EXIT_FAILURE) {  // EXIT_FAILURE is not used by all programs, maybe needs some adaptation.

    }
    return result;
}

Getting stderr and stdout with popen(), I'm afraid you'd need to redirect the output of stderr to stdout from the command-line you're passing to popen() by adding 2>&1. This has the inconvinience that both streams are unpredictably mixed.

If you really want to have two distinguished file-descriptors for stderr and stdout, one way to do it is to do the forking yourself and to duplicate the new processes stdout/stderr to two pipes which are accessible from the parent process. (see dup2() and pipe()). I could go into more detail here, but this is quite a tedious way of doing things and much care must be taken. And the internet is full of examples.

C# execute, wait, read command's output

The following snippet will work

public static string GetSystemInfo()
{
    var command = "/c systeminfo";
    var cmdsi = new ProcessStartInfo("cmd.exe");
    cmdsi.Arguments = command;
    cmdsi.RedirectStandardOutput = true;
    cmdsi.UseShellExecute = false;
    var cmd = Process.Start(cmdsi);
    var output = cmd.StandardOutput.ReadToEnd();

    cmd.WaitForExit();

    return output;
}

You should set RedirectStandardOutput to true and read output before calling WaitForExit, otherwise you can get a deadlock, per MSDN

The example avoids a deadlock condition by calling
p.StandardOutput.ReadToEnd before p.WaitForExit. A deadlock condition
can result if the parent process calls p.WaitForExit before
p.StandardOutput.ReadToEnd and the child process writes enough text to
fill the redirected stream. The parent process would wait indefinitely
for the child process to exit.

/c means terminating command line after execution

How do I get output from a command to appear in a control on a Form in real-time?

A brief description of what the code performs in this example:

The shell command (cmd.exe) is run first, using start /WAIT as parameter. More or less the same functionality as /k: the console is started without any specific task, waiting to process a command when one is sent.

StandardOutput, StandardError and StandardInput are all redirected, setting RedirectStandardOutput, RedirectStandardError and RedirectStandardInput properties of the ProcessStartInfo to true.

The console Output stream, when written to, will raise the OutputDataReceived event; it's content can be read from the e.Data member of the DataReceivedEventArgs.

StandardError will use its ErrorDataReceived event for the same purpose.

You could use a single event handler for both the events, but, after some testing, you might realize that is probably not a good idea. Having them separated avoids some weird overlapping and allows to easily tell apart errors from normal output (as a note, you can find programs that write to the error Stream instead of the output Stream).

StandardInput can be redirected assigning it to a StreamWriter stream.

Each time a string is written to the stream, the console will interpret that input as a command to be executed.

Also, the Process is instructed to rise it's Exited event upon termination, setting its EnableRaisingEvents property to true.

The Exited event is raised when the Process is closed because an Exit command is processed or calling the .Close() method (or, eventually, the .Kill() method, which should only be used when a Process is not responding anymore, for some reason).

Since we need to pass the console Output to some UI controls (RichTextBoxes in this example) and the Process events are raised in ThreadPool Threads, we must synchronize this context with the UI's.

This can be done using the Process SynchronizingObject property, setting it to the Parent Form or using the Control.BeginInvoke method, that will execute a delegate function on the thread where the control's handle belongs.

Here, a MethodInvoker representing the delegate is used for this purpose.

---

The core function used to instantiate the Process and set its properties and event handlers:

using System;
using System.Diagnostics;
using System.IO;
using System.Windows.Forms;

public partial class frmCmdInOut : Form
{
    Process cmdProcess = null;
    StreamWriter stdin = null;

    public frmCmdInOut() => InitializeComponent();

    private void MainForm_Load(object sender, EventArgs e)
    {
        rtbStdIn.Multiline = false;
        rtbStdIn.SelectionIndent = 20;
    }

    private void btnStartProcess_Click(object sender, EventArgs e)
    {
        btnStartProcess.Enabled = false;
        StartCmdProcess();
        btnEndProcess.Enabled = true;
    }

    private void btnEndProcess_Click(object sender, EventArgs e)
    {
        if (stdin.BaseStream.CanWrite) {
            stdin.WriteLine("exit");
        }
        btnEndProcess.Enabled = false;
        btnStartProcess.Enabled = true;
        cmdProcess?.Close();
    }

    private void rtbStdIn_KeyPress(object sender, KeyPressEventArgs e)
    {
        if (e.KeyChar == (char)Keys.Enter) {
            if (stdin == null) {
                rtbStdErr.AppendText("Process not started" + Environment.NewLine);
                return;
            }

            e.Handled = true;
            if (stdin.BaseStream.CanWrite) {
                stdin.Write(rtbStdIn.Text + Environment.NewLine);
                stdin.WriteLine();
                // To write to a Console app, just 
                // stdin.WriteLine(rtbStdIn.Text); 
            }
            rtbStdIn.Clear();
        }
    }

    private void StartCmdProcess()
    {
        var pStartInfo = new ProcessStartInfo {
             FileName = "cmd.exe",
            // Batch File Arguments = "/C START /b /WAIT somebatch.bat",
            // Test: Arguments = "START /WAIT /K ipconfig /all",
            Arguments = "START /WAIT",
            WorkingDirectory = Environment.SystemDirectory,
            // WorkingDirectory = Application.StartupPath,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            RedirectStandardInput = true,
            UseShellExecute = false,
            CreateNoWindow = true,
            WindowStyle = ProcessWindowStyle.Hidden,
        };

        cmdProcess = new Process {
            StartInfo = pStartInfo,
            EnableRaisingEvents = true,
            // Test without and with this
            // When SynchronizingObject is set, no need to BeginInvoke()
            //SynchronizingObject = this
        };

        cmdProcess.Start();
        cmdProcess.BeginErrorReadLine();
        cmdProcess.BeginOutputReadLine();
        stdin = cmdProcess.StandardInput;
        // stdin.AutoFlush = true;  <- already true

        cmdProcess.OutputDataReceived += (s, evt) => {
            if (evt.Data != null)
            {
                BeginInvoke(new MethodInvoker(() => {
                    rtbStdOut.AppendText(evt.Data + Environment.NewLine);
                    rtbStdOut.ScrollToCaret();
                }));
            }
        };

        cmdProcess.ErrorDataReceived += (s, evt) => {
            if (evt.Data != null) {
                BeginInvoke(new Action(() => {
                    rtbStdErr.AppendText(evt.Data + Environment.NewLine);
                    rtbStdErr.ScrollToCaret();
                }));
            }
        };

        cmdProcess.Exited += (s, evt) => {
            stdin?.Dispose();
            cmdProcess?.Dispose();
        };
    }
}

Since the StandardInput has been redirected to a StreamWriter:

stdin = cmdProcess.StandardInput;

we just write to the Stream to execute a command:

stdin.WriteLine(["Command Text"]);

---

The sample Form can be downloaded from PasteBin.

---

VB.Net version

Controls' names:

rtbStdOut -> RichTextBox (blue background), receives StdOut

rtbStdErr -> RichTextBox (in the middle), receives StdErr

rtbStdIn -> RichTextBox (at the bottom), writes to StdIn

btnStartProcess -> Button (on the right), starts the Process

btnEndProcess -> Button (on the left), stops te Process

Download this Form from Google Drive

Imports System.Diagnostics
Imports System.IO

Public Class frmCmdInOut

    Private cmdProcess As Process = Nothing
    Private stdin As StreamWriter = Nothing

    Protected Overrides Sub OnLoad(e As EventArgs)
        MyBase.OnLoad(e)
        rtbStdIn.Multiline = False
        rtbStdIn.SelectionIndent = 20
    End Sub

    Private Sub btnStartProcess_Click(sender As Object, e As EventArgs) Handles btnStartProcess.Click
        btnStartProcess.Enabled = False
        StartCmdProcess(Me)
        btnEndProcess.Enabled = True

    End Sub

    Private Sub btnEndProcess_Click(sender As Object, e As EventArgs) Handles btnEndProcess.Click
        If stdin.BaseStream IsNot Nothing AndAlso stdin.BaseStream.CanWrite Then stdin.WriteLine("exit")
        btnEndProcess.Enabled = False
        btnStartProcess.Enabled = True
        cmdProcess?.Close()
    End Sub

    Private Sub rtbStdIn_KeyPress(sender As Object, e As KeyPressEventArgs) Handles rtbStdIn.KeyPress
        If e.KeyChar = ChrW(Keys.Enter) Then
            If stdin Is Nothing Then
                rtbStdErr.AppendText("Process not started" + Environment.NewLine)
                Return
            End If

            e.Handled = True
            If stdin.BaseStream.CanWrite Then
                stdin.Write(rtbStdIn.Text + Environment.NewLine)
                stdin.WriteLine() ' To write to a Console app, just stdin.WriteLine(rtbStdIn.Text); 
            End If
            rtbStdIn.Clear()
        End If
    End Sub

    Private Sub StartCmdProcess(synchObj As Control)

        ' Arguments = $"start /WAIT cscript.exe script.vbs /xpr",
        ' Batch File Arguments = "/C START /b /WAIT batchfile.bat",
        ' Test: Arguments = "START /WAIT /K ipconfig /all",

        ' start with /U
        ' StandardErrorEncoding = Encoding.Unicode,
        ' StandardOutputEncoding = Encoding.Unicode,

        Dim pStartInfo = New ProcessStartInfo() With {
            .FileName = "cmd.exe",
            .Arguments = "START /WAIT",
            .CreateNoWindow = True,
            .RedirectStandardError = True,
            .RedirectStandardInput = True,
            .RedirectStandardOutput = True,
            .UseShellExecute = False,
            .WindowStyle = ProcessWindowStyle.Hidden,
            .WorkingDirectory = Application.StartupPath
        }

        cmdProcess = New Process() With {
            .EnableRaisingEvents = True,
            .StartInfo = pStartInfo,
            .SynchronizingObject = synchObj
        }

        cmdProcess.Start()
        cmdProcess.BeginErrorReadLine()
        cmdProcess.BeginOutputReadLine()
        stdin = cmdProcess.StandardInput

        AddHandler cmdProcess.OutputDataReceived,
            Sub(s, evt)
                If evt.Data IsNot Nothing Then
                    rtbStdOut.AppendText(evt.Data + Environment.NewLine)
                    rtbStdOut.ScrollToCaret()
                End If
            End Sub
        AddHandler cmdProcess.ErrorDataReceived,
            Sub(s, evt)
                If evt.Data IsNot Nothing Then
                    rtbStdErr.AppendText(evt.Data + Environment.NewLine)
                    rtbStdErr.ScrollToCaret()
                End If
            End Sub

        AddHandler cmdProcess.Exited,
            Sub(s, evt)
                stdin?.Dispose()
                cmdProcess?.Dispose()
            End Sub
    End Sub
End Class

How do I execute a command and get the output of the command within C++ using POSIX?

#include <cstdio>
#include <iostream>
#include <memory>
#include <stdexcept>
#include <string>
#include <array>

std::string exec(const char* cmd) {
    std::array<char, 128> buffer;
    std::string result;
    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);
    if (!pipe) {
        throw std::runtime_error("popen() failed!");
    }
    while (fgets(buffer.data(), buffer.size(), pipe.get()) != nullptr) {
        result += buffer.data();
    }
    return result;
}

Pre-C++11 version:

#include <iostream>
#include <stdexcept>
#include <stdio.h>
#include <string>

std::string exec(const char* cmd) {
    char buffer[128];
    std::string result = "";
    FILE* pipe = popen(cmd, "r");
    if (!pipe) throw std::runtime_error("popen() failed!");
    try {
        while (fgets(buffer, sizeof buffer, pipe) != NULL) {
            result += buffer;
        }
    } catch (...) {
        pclose(pipe);
        throw;
    }
    pclose(pipe);
    return result;
}

Replace popen and pclose with _popen and _pclose for Windows.

Run a command in a new cmd and save the return into a file

Like @Compo and @aschipfl said, the > in your script right now is piping the output of the whole start command (which is nothing) and piping your current script. However, if you either 1. break the > out with ^, the > will pass to the next command prompt, or if you 2. put " around the part of your start command that passes stuff to do to the next command prompt so it passes the things you want to pass including the > pipe.

---

Overall you could either use this:

start "NewConsole" cmd.exe /c my_batch.bat ^> file.log

or this:

start "NewConsole" cmd.exe /c "my_batch.bat > file.log"

Save command line output to variable in Fortran

Since you're using BASH, lets assume you're working on some kind of unix-like system. So you could use a FIFO. Something like

program readfifo
  implicit none
  integer :: u, i
  logical :: ex
  inquire(exist=ex, file='foo')
  if (.not. ex) then
     call execute_command_line ("mkfifo foo")
  end if
  call execute_command_line ("echo 5 > foo&")
  open(newunit=u, file='foo', action='read')
  read(u, *) i
  write(*, *) 'Managed to read the value ', i
end program readfifo

Note that the semantics of FIFO's wrt blocking can be a bit tricky (that's why there is the '&' after the echo command, you might want to read up on it a bit and experiment (particularly make sure you haven't got a zillion bash processes hanging around when you do this multiple times).

Run Commands continuously and step by step based on previous command output in JSch

after 1 month struggling with code and documentation i've found the answer.

i wanted to implement the ssh in enterprise application.first of all i have to say if you want interactive behavior you should use "shell" because it can give your command and return answer and stay connect with server but "exec" after execute command close channel.then i defined output and input as class variables.this is my class:

@Service
public class SshService {

private Channel channel;
private Session session;
ByteArrayOutputStream responseStream = new ByteArrayOutputStream();

PipedInputStream pip = new PipedInputStream(40);
PipedOutputStream pop = new PipedOutputStream(pip);
PrintStream print = new PrintStream(pop);

public SshService() throws IOException {
}

public String sshConnectToOlt(String username, String password, String host, int port) throws Exception {

    session = new JSch().getSession(username, host, port);
    session.setPassword(password);
    session.setConfig("StrictHostKeyChecking", "no");
    session.connect(30000);

    channel = session.openChannel("shell");

    channel.setInputStream(pip);

    responseStream= new ByteArrayOutputStream();
    channel.setOutputStream(responseStream);

    channel.connect(3 * 1000);

    Thread.sleep(2000);

    return responseStream.toString();

}

public String sshToOltCommands(String command) throws Exception {
    print.println(command);

    Thread.sleep(2000);

    String response = responseStream.toString();
    return response.replaceAll("\u001B\\[[\\d;]*[^\\d;]","");
}

public void disconnectFromOlt() {
    this.channel.disconnect();
    this.session.disconnect();
}

}

i write 3 methods.
sshConnectToOlt: for connecting to server
sshToOltCommands : executes command on server
disconnectFromOlt : disconnect from server

NOTICE:

i used

response.replaceAll("\u001B\\[[\\d;]*[^\\d;]","");

to remove ANSI control chars (VT100) from response

---

Related Topics