Streaming output from other programs (pwkit.slurp
)¶
The pwkit.slurp
module makes it convenient to read output generated by
other programs. This is accomplished with a context-manager class known as
Slurper
, which is built on top of the standard
subprocess.Popen
module.
The chief advantage of Slurper
above subprocess.Popen
is
that it provides convenient, streaming access to subprogram output,
maintaining the distinction between “stdout” (standard output, written to file
descriptor #1) and “stderr” (standard error, written to file descriptor #2).
It can also forward signals to the child program.
Standard usage might look like:
from pwkit import slurp
argv = ['cat', '/etc/passwd']
with slurp.Slurper (argv, linebreak=True) as slurper:
for etype, data in slurper:
if etype == 'stdout':
print ('got line:', data)
print ('exit code was:', slurper.proc.returncode)
Slurper
is a context manager to ensure that the child process is
always cleaned up. Within the context manager body, you should iterate over
the Slurper
instance to get a series of “event” 2-tuples consisting
of a Unicode string giving the event type, and the event data. Most, but not
all, events have to do with receiving data over the stdout or stderr pipes.
The events are:
Event type | Event data | Description |
---|---|---|
"stdout" |
The output | Data were received from the subprogram’s standard output. |
"stderr" |
The output | Data were received from the subprogram’s standard error. |
"forwarded-signal" |
The signal number | This process received a signal and forwarded it to the child. |
"timeout" |
None |
No data were received from the child within a fixed timeout. |
The data provided on the "stdout"
and "stderr"
events follow the usual
Python patterns for EOF. Namely, when either of those pipes is closed by the
subprocess, a final event is sent in which the data payload has zero length.
(It may be either a bytes object or a Unicode string depending on whether
decoding is enabled; see below.)
Warning
It is important to realize that programs that use the standard C
I/O routines, such as Python programs, buffer their output by default. The
pwkit.slurp
module may appear to be having problems while really the
child program is batching up its output and writing it all at once. This
can be surprising because the default behavior is line-buffered when
stdout
is connected to a TTY (as when you run programs in your
terminal), but buffered in large blocks when connected to a pipe (as when
using this module). On systems built on glibc
, you can control this by
using the stdbuf
program to launch your subprogram with different
buffering options. To run the command foo bar
with both stdout and
stderr buffered at the line level, run stdbuf -oL -eL foo bar
. To
disable buffering on both streams, run stdbuf -o0 -e0 foo bar
.
-
class
pwkit.slurp.
Slurper
(argv=None, env=None, cwd=None, propagate_signals=True, timeout=10, linebreak=False, encoding=None, stdin=slurp.Redirection.DevNull, stdout=slurp.Redirection.Pipe, stderr=slurp.Redirection.Pipe, executable=None)[source]¶ Construct a context manager used to read output from a subprogram. argv is used to launch the subprogram using
subprocess.Popen
with the shell keyword set to False. env, cwd, executable, stdin, stdout, and stderr are forwarded to thesubprocess.Popen
constructor as well.Regarding the redirection parameters stdin, stdout, and stderr, the constants in the
Redirection
object gives more user-friendly names to the analogues provided by thesubprocess
module, with the addition of aRedirection.DevNull
option emulating behavior added in Python 3. Otherwise these values are passed tosubprocess.Popen
verbatim, so you can use anyting thatsubprocess.Popen
would accept. Keep in mind that you can only fetch the subprogram’s output if one or both of the output paramers are set toRedirection.Pipe
!If propagate_signals is true, signals received by the parent process will be forwarded to the child process. This can be valuable to obtain correct behavior on SIGINT, for instance. Forwarded signals are SIGHUP, SIGINT, SIGQUIT, SIGTERM, SIGUSR1, and SIGUSR2. This is done by overwriting the calling process’ Python signal handlers; the original handlers are restored upon exit from the with-statement block.
If linebreak is true, output from the child process will be gathered into whole lines (split by
"\n"
) before being sent to the caller. The newline characters will be discarded, making it impossible to tell whether the final line of output ended with a newline or not.If encoding is not
None
, a decoder will be created withcodecs.getincrementaldecoder()
and the subprocess output will be converted from bytes to Unicode before being returned to the calling process.timeout sets the timeout for the internal
select.select()
call used to check for output from the subprogram. It is measured in seconds.Slurper
instances have attributesargv
,env
,cwd
,executable
,propagate_signals
, :timeout
,linebreak
, attr:encoding,stdin
, :stdout
, andstderr
recording the construction parameters.
-
pwkit.slurp.
Redirection
¶ An enum-like object defining ways to redirect the I/O streams of the subprogram. These values are identical to those used in
subprocess
but with nicer names.Constant Meaning Redirection.Pipe
Pipe output to the calling program. Redirection.Stdout
Only valid for stderr
; merge it withstdout
Redirection.DevNull
Direct input from /dev/null
, or output thereto.The whole raison d’être of
pwkit.slurp
is to make it easy to communicate output between programs, so you probably will probably want to useRedirection.Pipe
forstdout
andstderr
most of the time.
Slurper
reference¶
-
Slurper.
proc
¶ The
subprocess.Popen
instance of the child program. After the program has exited, you can access its exit code asSlurper.proc.returncode
.
-
Slurper.
argv
¶ The
argv
of the program to be launched.
-
Slurper.
env
¶ Environment dictionary for the program to be launched.
-
Slurper.
cwd
¶ The working directory for the program to be launched.
-
Slurper.
executable
¶ The name of the executable to launch (
argv[0]
is allowed to differ from this).
-
Slurper.
propagate_signals
¶ Whether to forward the subprogram any signals that are received by the calling process.
-
Slurper.
timeout
¶ The timeout (in seconds) for waiting for output from the child program. If nothing is received, a
"timeout"
event is generated.
-
Slurper.
linebreak
¶ Whether to gather the subprogram output into textual lines.
-
Slurper.
encoding
¶ The encoding to be used to decode the subprogram output from bytes to Unicode, or
None
if no such decoding is to be done.
-
Slurper.
stdin
¶ How to redirect the standard input of the subprogram, if at all.
-
Slurper.
stdout
¶ How to redirect the standard output of the subprogram, if at all. If not
Pipe
, no"stdout"
events will be received.
-
Slurper.
stderr
¶ How to redirect the standard error of the subprogram, if at all. If not
Pipe
, no"stderr"
events will be received. IfStdout
, events that would have had a type of"stderr"
will have a type of"stdout"
instead.