.. Copyright 2015 Peter K. G. Williams and collaborators. This file licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License (CC-BY-SA). Streaming output from other programs (:mod:`pwkit.slurp`) ============================================================================== .. module:: pwkit.slurp :synopsis: streaming output from other programs The :mod:`pwkit.slurp` module makes it convenient to read output generated by other programs. This is accomplished with a context-manager class known as :class:`Slurper`, which is built on top of the standard :class:`subprocess.Popen` module. The chief advantage of :class:`Slurper` above :class:`subprocess.Popen` is that it provides convenient, *streaming* access to subprogram output, maintaining the distinction between “stdout” (standard output, written to file descriptor #1) and “stderr” (standard error, written to file descriptor #2). It can also forward signals to the child program. Standard usage might look like:: from pwkit import slurp argv = ['cat', '/etc/passwd'] with slurp.Slurper (argv, linebreak=True) as slurper: for etype, data in slurper: if etype == 'stdout': print ('got line:', data) print ('exit code was:', slurper.proc.returncode) :class:`Slurper` is a context manager to ensure that the child process is always cleaned up. Within the context manager body, you should iterate over the :class:`Slurper` instance to get a series of “event” 2-tuples consisting of a Unicode string giving the event type, and the event data. Most, but not all, events have to do with receiving data over the stdout or stderr pipes. The events are: ====================== ================= ============================================================= Event type Event data Description ====================== ================= ============================================================= ``"stdout"`` The output Data were received from the subprogram’s standard output. ``"stderr"`` The output Data were received from the subprogram’s standard error. ``"forwarded-signal"`` The signal number This process received a signal and forwarded it to the child. ``"timeout"`` ``None`` No data were received from the child within a fixed timeout. ====================== ================= ============================================================= The data provided on the ``"stdout"`` and ``"stderr"`` events follow the usual Python patterns for EOF. Namely, when either of those pipes is closed by the subprocess, a final event is sent in which the data payload has zero length. (It may be either a bytes object or a Unicode string depending on whether decoding is enabled; see below.) .. warning:: It is important to realize that programs that use the standard C I/O routines, such as Python programs, buffer their output by default. The :mod:`pwkit.slurp` module may appear to be having problems while really the child program is batching up its output and writing it all at once. This can be surprising because the default behavior is line-buffered when ``stdout`` is connected to a TTY (as when you run programs in your terminal), but buffered in large blocks when connected to a pipe (as when using this module). On systems built on ``glibc``, you can control this by using the ``stdbuf`` program to launch your subprogram with different buffering options. To run the command ``foo bar`` with both stdout and stderr buffered at the line level, run ``stdbuf -oL -eL foo bar``. To disable buffering on both streams, run ``stdbuf -o0 -e0 foo bar``. .. class:: Slurper(argv=None, env=None, cwd=None, propagate_signals=True, timeout=10, linebreak=False, encoding=None, stdin=slurp.Redirection.DevNull, stdout=slurp.Redirection.Pipe, stderr=slurp.Redirection.Pipe, executable=None) Construct a context manager used to read output from a subprogram. *argv* is used to launch the subprogram using :class:`subprocess.Popen` with the *shell* keyword set to False. *env*, *cwd*, *executable*, *stdin*, *stdout*, and *stderr* are forwarded to the :class:`subprocess.Popen` constructor as well. Regarding the redirection parameters *stdin*, *stdout*, and *stderr*, the constants in the :data:`Redirection` object gives more user-friendly names to the analogues provided by the :mod:`subprocess` module, with the addition of a :data:`Redirection.DevNull` option emulating behavior added in Python 3. Otherwise these values are passed to :class:`subprocess.Popen` verbatim, so you can use anyting that :class:`subprocess.Popen` would accept. Keep in mind that you can only fetch the subprogram’s output if one or both of the output paramers are set to :data:`Redirection.Pipe`! If *propagate_signals* is true, signals received by the parent process will be forwarded to the child process. This can be valuable to obtain correct behavior on SIGINT, for instance. Forwarded signals are SIGHUP, SIGINT, SIGQUIT, SIGTERM, SIGUSR1, and SIGUSR2. This is done by overwriting the calling process’ Python signal handlers; the original handlers are restored upon exit from the with-statement block. If *linebreak* is true, output from the child process will be gathered into whole lines (split by ``"\n"``) before being sent to the caller. *The newline characters will be discarded*, making it impossible to tell whether the final line of output ended with a newline or not. If *encoding* is not ``None``, a decoder will be created with :func:`codecs.getincrementaldecoder` and the subprocess output will be converted from bytes to Unicode before being returned to the calling process. *timeout* sets the timeout for the internal :func:`select.select` call used to check for output from the subprogram. It is measured in seconds. :class:`Slurper` instances have attributes :attr:`argv`, :attr:`env`, :attr:`cwd`, :attr:`executable`, :attr:`propagate_signals`, ::attr:`timeout`, :attr:`linebreak`, attr:`encoding`, :attr:`stdin`, ::attr:`stdout`, and :attr:`stderr` recording the construction parameters. .. data:: Redirection An enum-like object defining ways to redirect the I/O streams of the subprogram. These values are identical to those used in :mod:`subprocess` but with nicer names. ======================= =================================================== Constant Meaning ======================= =================================================== ``Redirection.Pipe`` Pipe output to the calling program. ``Redirection.Stdout`` Only valid for ``stderr``; merge it with ``stdout`` ``Redirection.DevNull`` Direct input from ``/dev/null``, or output thereto. ======================= =================================================== The whole *raison d’être* of :mod:`pwkit.slurp` is to make it easy to communicate output between programs, so you probably will probably want to use ``Redirection.Pipe`` for ``stdout`` and ``stderr`` most of the time. :class:`Slurper` reference ------------------------------------------------------------------------ .. attribute:: Slurper.proc The :class:`subprocess.Popen` instance of the child program. After the program has exited, you can access its exit code as ``Slurper.proc.returncode``. .. attribute:: Slurper.argv The ``argv`` of the program to be launched. .. attribute:: Slurper.env Environment dictionary for the program to be launched. .. attribute:: Slurper.cwd The working directory for the program to be launched. .. attribute:: Slurper.executable The name of the executable to launch (``argv[0]`` is allowed to differ from this). .. attribute:: Slurper.propagate_signals Whether to forward the subprogram any signals that are received by the calling process. .. attribute:: Slurper.timeout The timeout (in seconds) for waiting for output from the child program. If nothing is received, a ``"timeout"`` event is generated. .. attribute:: Slurper.linebreak Whether to gather the subprogram output into textual lines. .. attribute:: Slurper.encoding The encoding to be used to decode the subprogram output from bytes to Unicode, or ``None`` if no such decoding is to be done. .. attribute:: Slurper.stdin How to redirect the standard input of the subprogram, if at all. .. attribute:: Slurper.stdout How to redirect the standard output of the subprogram, if at all. If not ``Pipe``, no ``"stdout"`` events will be received. .. attribute:: Slurper.stderr How to redirect the standard error of the subprogram, if at all. If not ``Pipe``, no ``"stderr"`` events will be received. If ``Stdout``, events that would have had a type of ``"stderr"`` will have a type of ``"stdout"`` instead.