Python: Capture standard output, standard error, and the exit code of a subprocess

I might be missing the obvious, but I don't think the subprocess module has a method that will capture the standard output, standard error, and the exit code of a subprocess in a single call. It is not complex to write one and can be useful.

The code that captures theresults

examples/python/capture.py

import subprocess
import sys


def run(cmd):
    proc = subprocess.Popen(cmd,
        stdout = subprocess.PIPE,
        stderr = subprocess.PIPE,
    )
    stdout, stderr = proc.communicate()

    return proc.returncode, stdout, stderr

code, out, err = run([sys.executable, 'examples/python/run.py'])

print("out: '{}'".format(out))
print("err: '{}'".format(err))
print("exit: {}".format(code))

A sample external program

examples/python/run.py

import sys

sys.stdout.write("STDOUT: Hello World!\n")
sys.stderr.write("STDERR: Welcome to the dark side!\n")
sys.stdout.write("STDOUT: Second line\n")
sys.stderr.write("STDERR: Warning\n")
exit(42)

Running the external program directly

$ python examples/python/run.py

STDOUT: Hello World!
STDERR: Welcome to the dark side!
STDOUT: Second line
STDERR: Warning

echo $?
42

The results

out: 'STDOUT: Hello World!
STDOUT: Second line
'
err: 'STDERR: Welcome to the dark side!
STDERR: Warning
'
exit: 42

As you can see in this case the standard output and standard error are separated. You can't tell the exact order.

Capture STDOUT and STDERR together

Sometimes it is better to be able to capture the two together:

examples/python/capture_together.py

import subprocess
import sys
import os


def run(cmd):
    os.environ['PYTHONUNBUFFERED'] = "1"
    proc = subprocess.Popen(cmd,
        stdout = subprocess.PIPE,
        stderr = subprocess.STDOUT,
    )
    stdout, stderr = proc.communicate()

    return proc.returncode, stdout, stderr

code, out, err = run([sys.executable, 'examples/python/run.py'])

print("out: '{}'".format(out))
print("err: '{}'".format(err))
print("exit: {}".format(code))

Here we had stderr = subprocess.STDOUT instead of stderr = subprocess.PIPE.

out: 'STDOUT: Hello World!
STDERR: Welcome to the dark side!
STDOUT: Second line
STDERR: Warning
'
err: 'None'
exit: 42

In order for this to work properly on a Python script we'll need to turn off output buffering for the child process. This can be done by setting the PYTHONUNBUFFERED environment variable.

Tee: Capture and also print

Finally in this example we both collect the out and at the same time keep printing to the screen. Just to show off we captur STDOUT and STDERR both individually and mixed together.

You'll probably use some subset of these features.

examples/python/capture_tee.py

from __future__ import print_function
import os
import subprocess
import sys

def run(cmd):
    os.environ['PYTHONUNBUFFERED'] = "1"
    proc = subprocess.Popen(cmd,
                            stdout = subprocess.PIPE,
                            stderr = subprocess.PIPE,
                            universal_newlines = True,
                            )
    stdout = []
    stderr = []
    mix = []
    while proc.poll() is None:
        line = proc.stdout.readline()
        if line != "":
            stdout.append(line)
            mix.append(line)
            print(line, end='')

        line = proc.stderr.readline()
        if line != "":
            stderr.append(line)
            mix.append(line)
            print(line, end='')

    return proc.returncode, stdout, stderr, mix

code, out, err, mix = run([sys.executable, 'examples/python/run.py'])

print("out: '{}'".format(out))
print("err: '{}'".format(err))
print("err: '{}'".format(mix))
print("exit: {}".format(code))

python examples/python/capture_tee.py 

STDOUT: Hello World!
STDERR: Welcome to the dark side!
STDOUT: Second line
STDERR: Warning
out: '['STDOUT: Hello World!\n', 'STDOUT: Second line\n']'
err: '['STDERR: Welcome to the dark side!\n', 'STDERR: Warning\n']'
err: '['STDOUT: Hello World!\n', 'STDERR: Welcome to the dark side!\n', 'STDOUT: Second line\n', 'STDERR: Warning\n']'
exit: 42