Streaming output from other programs (pwkit.slurp)

The pwkit.slurp module makes it convenient to read output generated by other programs. This is accomplished with a context-manager class known as Slurper, which is built on top of the standard subprocess.Popen module.

The chief advantage of Slurper above subprocess.Popen is that it provides convenient, streaming access to subprogram output, maintaining the distinction between “stdout” (standard output, written to file descriptor #1) and “stderr” (standard error, written to file descriptor #2). It can also forward signals to the child program.

Standard usage might look like:

from pwkit import slurp
argv = ['cat', '/etc/passwd']

with slurp.Slurper (argv, linebreak=True) as slurper:
    for etype, data in slurper:
        if etype == 'stdout':
            print ('got line:', data)

print ('exit code was:', slurper.proc.returncode)

Slurper is a context manager to ensure that the child process is always cleaned up. Within the context manager body, you should iterate over the Slurper instance to get a series of “event” 2-tuples consisting of a Unicode string giving the event type, and the event data. Most, but not all, events have to do with receiving data over the stdout or stderr pipes. The events are:

Event type	Event data	Description
"stdout"	The output	Data were received from the subprogram’s standard output.
"stderr"	The output	Data were received from the subprogram’s standard error.
"forwarded-signal"	The signal number	This process received a signal and forwarded it to the child.
"timeout"	None	No data were received from the child within a fixed timeout.

The data provided on the "stdout" and "stderr" events follow the usual Python patterns for EOF. Namely, when either of those pipes is closed by the subprocess, a final event is sent in which the data payload has zero length. (It may be either a bytes object or a Unicode string depending on whether decoding is enabled; see below.)

Warning

It is important to realize that programs that use the standard C I/O routines, such as Python programs, buffer their output by default. The pwkit.slurp module may appear to be having problems while really the child program is batching up its output and writing it all at once. This can be surprising because the default behavior is line-buffered when stdout is connected to a TTY (as when you run programs in your terminal), but buffered in large blocks when connected to a pipe (as when using this module). On systems built on glibc, you can control this by using the stdbuf program to launch your subprogram with different buffering options. To run the command foo bar with both stdout and stderr buffered at the line level, run stdbuf -oL -eL foo bar. To disable buffering on both streams, run stdbuf -o0 -e0 foo bar.

class pwkit.slurp.Slurper(argv=None, env=None, cwd=None, propagate_signals=True, timeout=10, linebreak=False, encoding=None, stdin=slurp.Redirection.DevNull, stdout=slurp.Redirection.Pipe, stderr=slurp.Redirection.Pipe, executable=None)

Construct a context manager used to read output from a subprogram. argv is used to launch the subprogram using subprocess.Popen with the shell keyword set to False. env, cwd, executable, stdin, stdout, and stderr are forwarded to the subprocess.Popen constructor as well.

Regarding the redirection parameters stdin, stdout, and stderr, the constants in the Redirection object gives more user-friendly names to the analogues provided by the subprocess module, with the addition of a Redirection.DevNull option emulating behavior added in Python 3. Otherwise these values are passed to subprocess.Popen verbatim, so you can use anyting that subprocess.Popen would accept. Keep in mind that you can only fetch the subprogram’s output if one or both of the output paramers are set to Redirection.Pipe!

If propagate_signals is true, signals received by the parent process will be forwarded to the child process. This can be valuable to obtain correct behavior on SIGINT, for instance. Forwarded signals are SIGHUP, SIGINT, SIGQUIT, SIGTERM, SIGUSR1, and SIGUSR2. This is done by overwriting the calling process’ Python signal handlers; the original handlers are restored upon exit from the with-statement block.

If linebreak is true, output from the child process will be gathered into whole lines (split by "\n") before being sent to the caller. The newline characters will be discarded, making it impossible to tell whether the final line of output ended with a newline or not.

If encoding is not None, a decoder will be created with codecs.getincrementaldecoder() and the subprocess output will be converted from bytes to Unicode before being returned to the calling process.

timeout sets the timeout for the internal select.select() call used to check for output from the subprogram. It is measured in seconds.

Slurper instances have attributes argv, env, cwd, executable, propagate_signals, :timeout, linebreak, attr:encoding, stdin, :stdout, and stderr recording the construction parameters.

pwkit.slurp.Redirection

An enum-like object defining ways to redirect the I/O streams of the subprogram. These values are identical to those used in subprocess but with nicer names.

Constant	Meaning
Redirection.Pipe	Pipe output to the calling program.
Redirection.Stdout	Only valid for stderr; merge it with stdout
Redirection.DevNull	Direct input from /dev/null, or output thereto.

The whole raison d’être of pwkit.slurp is to make it easy to communicate output between programs, so you probably will probably want to use Redirection.Pipe for stdout and stderr most of the time.

Slurper reference

Slurper.proc

The subprocess.Popen instance of the child program. After the program has exited, you can access its exit code as Slurper.proc.returncode.

Slurper.argv

The argv of the program to be launched.

Slurper.env

Environment dictionary for the program to be launched.

Slurper.cwd

The working directory for the program to be launched.

Slurper.executable

The name of the executable to launch (argv[0] is allowed to differ from this).

Slurper.propagate_signals

Whether to forward the subprogram any signals that are received by the calling process.

Slurper.timeout

The timeout (in seconds) for waiting for output from the child program. If nothing is received, a "timeout" event is generated.

Slurper.linebreak

Whether to gather the subprogram output into textual lines.

Slurper.encoding

The encoding to be used to decode the subprogram output from bytes to Unicode, or None if no such decoding is to be done.

Slurper.stdin

How to redirect the standard input of the subprogram, if at all.

Slurper.stdout

How to redirect the standard output of the subprogram, if at all. If not Pipe, no "stdout" events will be received.

Slurper.stderr

How to redirect the standard error of the subprogram, if at all. If not Pipe, no "stderr" events will be received. If Stdout, events that would have had a type of "stderr" will have a type of "stdout" instead.