sclapp

an easy-to-use framework for python command-line applications

Forest Bond

Copyright © 2005-2006 Forest Bond

Table of Contents

Overview
Quick Start
Core Functionality
Signal Handling
Protected Output
Prioritized Error Output
Miscellaneous Optional Functionality
Daemonization
Transparent Help & Version Options
User Friendly Bug Handling
Module Reference
Exceptions
Functions
Data
Modules

Overview

sclapp is a Python module providing functionality intended for use by command line applications. It provides some functions and classes that are intended to reduce the boiler-plate required when writing small command-line applications in Python. Currently, this includes:

  • Augmented signal handling on POSIX systems. When signals are caught, they are converted to exceptions, which are trivial to handle properly in Python (as opposed to raw signals, the proper handling of which can be both tedious and fragile).
  • Protected output that permits clean program termination even under exceptional stdio failure conditions.
  • Prioritized error output (using the standard logging module).
  • Simplified GNU-style command line option parsing for standard options such as -h/--help and -v/--version.
  • User-friendly handling of uncaught exceptions (a message reporting that a bug has occured is displayed that encourages the user to file a bug report).

While some of this functionality can be used through a bits-and-pieces approach, much of it is best integrated into your program's main() function with a particular order of calls. This is achieved through the use of a wrapper function that contains and calls the existing main() function. An existing program can usually be modified very easily to take advantage of this.

sclapp was written primarily because I found myself including most of the same code in every command line application I was writing. Putting this code in a separate module makes my life easier, and makes my programs more consistent. Others may find these benefits to their liking as well, which is why I've made it available as a separate module.

Quick Start

Using sclapp should be fairly straight-forward. Here's the simplest scenario: 

import sclapp

def main(argv):
  do_something()

main = sclapp.mainWrapper(main)

if __name__ == '__main__':
  sys.exit(main())

The immediate benefits of making use of sclapp's main() wrapper are:

  • Better handling of signals on POSIX systems. Perhaps most importantly, SIGPIPE and SIGINT will not cause your program to produce a traceback.
  • User-friendly handling of uncaught exceptions. A brief error message notifying the user of what appears to be a bug. By default, a traceback is included along with the message so the user can include it with a bug report, which he is asked to submit.
  • A number of other small enhancements that I typically incorporate into my programs. Most of these can be explicitly disabled. See the documentation for the mainWrapper() function for more information.

Let's look at a slightly more complicated example. Suppose your program needs to perform some cleanup action before terminating. Python's default signal handling configuration results in the receipt of SIGPIPE to cause a traceback due to an IOError. Even if the IOError is handled properly using a try...finally block, a traceback is still likely to occur (unless the IOError is explicitly caught, something most programmers do not implement in practice).

Even worse, if stderr is being redirected to stdout (not an entirely uncommon situation), and output causes SIGPIPE (and, consequently, an IOError), any output generated during the shutdown sequence will re-trigger the error. In some cases, your program can spiral out of control in a chain reaction of exceptions that prevent your cleanup code from ever running!

Consider the following example: 

import sclapp, signal

def main(argv):
  try:
    sclapp.setErrorOutputLevel(sclapp.INFO)
    sclapp.printDebug('calling do_something()...')
    do_something()
  finally:
    sclapp.printWarning('cleaning up...')
    cleanup()

main = sclapp.mainWrapper(
  main,
  exit_signals = ( signal.SIGPIPE, )
)

if __name__ == '__main__':
  sys.exit(main())

If this program receives a SIGPIPE, the following events will occur:

  • SIGPIPE will now cause an ExitSignalError. An IOError may also be raised, however sclapp's protected output handles that situation by disabling the file object that caused the exception, thus preventing further exceptions for the same reason.
  • The ExitSignalError will be handled properly by the try...finally block, and your cleanup code will execute as expected.
  • sclapp's mainWrapper() will catch the ExitSignalError and return 0 to trigger normal program termination.

This example also illustrates usage of sclapp's prioritized error output functions. The first message printed will not typically be visiable, as it has priority "DEBUG," which is lower than the specified output level, "INFO." For more information on the prioritized error output functions and constants, see the documentation specific to that functionality below.

Core Functionality

Signal Handling

Signal handling is something that can be tedious to implement properly due to the asynchronous nature of signals. The typical approach is to set global flags when signals are received, and poll those flags periodically. This is tedious for programmers, and frequently results in poor responsiveness.

sclapp's signal handling strategy is to map signals to exceptions, which are also asynchronous by nature. Exceptions, however, are trivially dealt with in Python, which has language constructs to make them easy to work with. Python's default signal configuration leads to less-than-satisfactory results in most situations, and even the most trivial command-line applications should improve upon this to function as expected. For instance, no command-line application should print a traceback when SIGPIPE is received.

Signal Handling Defaults

sclapp's default signal handling configuration is intended to be sane, without providing unnecessary behavior in simple applications. Signals that are commonly expected to result in normal program termination are set to the default operating system behavior (SIG_DFL). Currently, this includes any of the following signals supported by the underlying system:

  • SIGHUP
  • SIGINT
  • SIGQUIT
  • SIGILL
  • SIGABRT
  • SIGFPE
  • SIGPIPE
  • SIGTERM
  • SIGBUS
  • SIGPROF
  • SIGSYS
  • SIGTRAP
  • SIGXCPU
  • SIGXFSZ

Additionally, signals that are usually explicitly dealt with (one way or another) are ignored by default:

  • SIGUSR1
  • SIGUSR2
  • SIGALRM

The consequences of these defaults are that simple programs (that do not need to perform cleanup actions prior to exit) will behave roughly as expected. Programs that do require cleanup should not use the default signal handling configuration. This is a considerable improvement over Python's default behavior, which results in unnecessary tracebacks in normal use of command-line applications.

Custom Signal Handling Configurations

sclapp categorizes signals into one of four groups of signals:

  • Exit signals: those that will cause the program to exit, possibly after performing some cleanup actions.
  • Notify signals: those that the program would like to know about, but may not necessarily lead to termination.
  • Default signals: those that are set to SIG_DFL, which result in behavior determined by the operating system (silent and immediate termination for many signals on most systems).
  • Ignore signals: those that are set to SIG_IGN, which are completely ignored (with certain exceptions imposed by the operating system i.e. SIGKILL on most systems).

Exit and notify signals trigger ExitSignalErrors and SignalErrors, respectively. Both ignore and default signals are handled by the operating system without any notification to the caller. This is typical behavior for UNIX-like systems. Callers set their signal-handling configurations by mapping signals to these categories, and handling exceptions appropriately.

It is important to note that, apart from generating different types of exceptions, sclapp handles exit signals and notify signals slightly differently. At times, a program may receive a signal while a previously caught signal is being handled. Also, due to the way Python handles output, certain conditions result in both receipt of SIGPIPE as well as one or more IOErrors being raised. In either of these scenarios, it is possible for more than one signal to have been caught before sclapp has an opportunity to raise an appropriate exception. If this is the case, sclapp will always attempt to give the exit priority, and the exception resulting from the caught notify signal will be dropped in favor of the ExitSignalError.

It should be understood that exit and notify signals are to be considered fundamentally different in a semantic sense. sclapp makes the assumption that signals categorized as exit signals are truly intended to cause program exit. Consequently, while it is possible to catch ExitSignalErrors and handle them explicitly, doing so is highly discouraged, as it would not be consistent with the approach that sclapp takes. In summary, ExitSignalErrors should only be handled using try...finally blocks, and not explicitly caught with try...except statements.

Finally, do note that, since default signals do not trigger exceptions to be raised, callers will have no opportunity to perform cleanup tasks if such signals are caught. Thus, those signals that are normally considered default signals should be re-mapped as exit signals by programs requiring cleanup.

Specifying Signal-Handling Configurations

XXX

Protected Output

Situations may arise when stdout and/or stderr become unusable, and writing to them raises an IOError. sclapp's output protection catches this exception, and handles it appropriately. Namely, if sclapp is handling SIGPIPE, the error is dropped, and, if necessary, the exception resulting from SIGPIPE is forcibly raised, if it was interrupted by the IOError. If sclapp is not handling SIGPIPE or the program is running on a non-POSIX system, the IOError is converted to a CriticalError, which is caught in the main function (after any cleanup has occured), and reported to the user. Consequently, these situations should never result in a traceback.

sclapp achieves this output protection by replacing sys.stdout and sys.stderr with alternative, limited, file-like objects. Be aware of this if you are also attempting to modify these objects (it is likely this will not work very well).

Be advised that sclapp will not treat all IOErrors as described above, only those that are either likely to occur as a result of permanent problems with the respective file object, or those that are not likely to have been caused by a bug in the calling program. (Currently, sclapp only deals with EPIPE errors, and passes all others along to the caller).

Prioritized Error Output

It is common for command-line applications to have variable levels of verbosity with regard to program output. The logging from Python's standard library is ideal for this sort of task, however, the module is quite flexible, and configuring it for typical usage by simple command-line programs can be tedious. For this reason, sclapp provides access to the functionality offered by the logging module pre-configured to be appropriate for the kind of prioritized error output simple command-line applications sometimes require.

sclapp's prioritized error output functionality is accessed via several functions, each of which prints a message of a given priority to stderr. These functions are listed below:

  • printDebug: prints a message with priority DEBUG.
  • printInfo: prints a message with priority INFO.
  • printWarning: prints a message with priority WARNING.
  • printError: prints a message with priority ERROR.
  • printCritical: prints a message with priority CRITICAL.

ALL, DEBUG, INFO, WARNING, ERROR, CRITICAL are constants defined by sclapp (actually, they are defined in the logging module, but sclapp makes them available for import as well). Just as with the logging module, these constants correspond with integer values the indicate a priority level; these values are 10, 20, 30, 40, and 50, respectively.

The signficance of the integer priority levels is simple. When your program is run, it will set the desired level of error output, which is itself an integer (although it is normally set using one of the pre-defined constants listed above). Any messages printed with an associated priority that is greater than or equal to this error output level will appear on stderr; messages whose priorities are less than the error output level do not actually appear on the terminal.

The error output level can be set using the sclapp function setErrorOutputLevel(). This is typically done as soon as possible in the program's main function. If you are using sclapp's built-in main() wrapper, mainWrapper(), the desired output level can be specified using a keyword argument, error_output_level. For a more thorough explanation of the concepts surrounding prioritized output, see the documentation for the logging module.

Miscellaneous Optional Functionality

sclapp provides some "bonus" convenience functionality if your program needs it.

Daemonization

sclapp will daemonize your program if given the appropriate argument to the main() wrapper.

Transparent Help & Version Options

User Friendly Bug Handling

sclapp will, by default, handle uncaught exceptions (except for CriticalErrors, ExitSignalErrors, and some IOErrors, as described above) by telling the user that a bug has been encountered, and asking him to file a bug report. The exact message can be changed (see the documentation for the main() wrapper).

Module Reference

Note: this section is automatically generated from source code.

Exceptions

CriticalError (sclapp.exceptions.Error)

indicates a fatal error (can and should be raised by sclapp caller)

ExitSignalError (sclapp.exceptions.CriticalError)

indicates that an exit signal has been caught (raised by sclapp)

SignalError (sclapp.exceptions.Error)

indicates that a notify signal has been caught (raised by sclapp)

UsageError (sclapp.exceptions.CriticalError)

indicates a usage error (can and should be raised by sclapp caller)

Functions

disableSignalHandling()

disableSignalHandling() -> None

Disables sclapp signal handling.

enableSignalHandling(exit_signals = None, notify_signals = None, default_signals = None, ignore_signals = None)

enableSignalHandling() -> None

Enables sclapp signal handling.

getCaughtSignals()

getCaughtSignals() -> list

Returns a list containing all signal numbers that have been caught by sclapp signal handlers.

getDefaultSignals()

getDefaultSignals() -> list

Returns a list containing all signal numbers that are currently being handled by sclapp as default signals (SIG_DFL).

getExitSignals()

getExitSignals() -> list

Returns a list containing all signal numbers that are currently being handled by sclapp as exit signals.

getIgnoreSignals()

getIgnoreSignals() -> list

Returns a list containing all signal numbers that are currently being handled by sclapp as ignore signals (SIG_IGN).

getNotifySignals()

getNotifySignals() -> list

Returns a list containing all signal numbers that are currently being handled by sclapp as notify signals.

mainWrapper(
realmain,
name = None,
author = 'the author',
version = None,
doc = None,
handle_signals = True,
exit_signals = None,
notify_signals = None,
default_signals = None,
ignore_signals = None,
protect_output = True,
daemonize = False,
bug_message = '${traceback}\nSomething bad happened, and is most likely a bug.\nPlease file a bug report to ${author}.\nInclude the error message(s) printed here.',
version_message = '${name} version ${version}',
ignore_unrecognized_options = True,
error_output_level = None
)

Some of the optional parameters mentioned above cause sclapp to parse argv and respond appropriately:

* If version_message is not None, sclapp will respond to the -v command line switch by printing the version_message to stdout. * If doc is not None, sclapp will respond to the -h command line switch by printing doc to stdout.

Both bug_message and version_message are parsed for substitution strings prior to printing. Specifically, sclapp uses the Template class of the standard library's string module to make the following substitutions:

${name} is replaced by the name parameter ${author} is replaced by the author parameter ${version} is replaced by the version parameter ${doc} is replaced by the doc parameters

The doc parameter is parsed for all other substitutions before it is itself substituted. Thus, callers should feel free to add ${name}, ${author}, and ${version} to the doc parameter, and they will be replaced appropriately before printing.

Be default, sclapp installs signal handlers that are easily configurable using module functions. Setting handle_signals to False will disable this behavior.

sclapp can also be used to write simple daemons. To cause the program to daemonize before launching the wrapped main() function, set daemonize to True.

bug_message, if not None, will be printed in the event that an unhandled exception is caught by the main wrapper. The default message simply informs the user that a likely bug has been encountered, and asks them to file a bug report to the author.

If ignore_unrecognized_options is False, sclapp will complain about options it isn't expecting. If your program does not explicitly handle command line options, you probably want this. Otherwise, you definately don't want this, as any of your options will cause a UsageError to be reported before you have a chance to handle them.

If sclapp finds that it can't do anything useful with -h/--help or -v/--version (if version, doc are None, or contain substitutions sclapp can't fulfill), sclapp does nothing with command line options.

The new main() function will return the wrapped function's return value, unless an exception is raised (including exceptions resulting from an exit signal being received), in which case the return value is an integer intended to act as the program's termination status.

printCritical(message)

printCritical(message) -> None

Prints a message to stderr with priority CRITICAL.

printDebug(message)

printDebug(message) -> None

Prints a message to stderr with priority DEBUG.

printError(message)

printError(message) -> None

Prints a message to stderr with priority ERROR.

printInfo(message)

printInfo(message) -> None

Prints a message to stderr with priority INFO.

printWarning(message)

printWarning(message) -> None

Prints a message to stderr with priority WARNING.

protectOutput()
setErrorOutputLevel(level)

setErrorOutputLevel(level) -> None

level: int minimum priority of error messages that are displayed

Sets the minimum priority required for an error output message to actually be seen. sclapp defines the following constants for convenience:

ALL = 0 SCLAPP_DEBUG = 5 DEBUG = 10 INFO = 20 WARNING = 30 ERROR = 40 CRITICAL = 50

Observant programmers will recognize these as the same constants defined by the logging module, which provides the underlying functionality for sclapp's prioritized error output.

unprotectOutput()

Data

ALL = 0

ALL_SIGNALS = (6, 14, 7, 17, 17, 18, 8, 1, 4, 2, 29, 6, 9, 13, 29, 27, 30, 3, 64, 34, 11, 19, 31, 15, 5, 20, 21, 22, 23, 10, 12, 26, 28, 24, 25)

CRITICAL = 50

DEBUG = 10

ERROR = 40

INFO = 20

SCLAPP_DEBUG = 5

STD_DEFAULT_SIGNALS = (1, 2, 3, 4, 6, 8, 13, 15, 7, 27, 31, 5, 24, 25)

STD_EXIT_SIGNALS = ()

STD_IGNORE_SIGNALS = (10, 12, 14)

STD_NOTIFY_SIGNALS = ()

WARNING = 30

Modules

Module sclapp.daemonize

Functions
daemonize()

Performs traditional *nix daemonization. For more information, see http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16.

getRootDirectory()

Tries to find the root directory by moving up the directory tree starting from the current working directory.

Module sclapp.processes

Classes
Class BackgroundCommand
BackgroundCommand (sclapp.processes._BackgroundProcess)

Runs an external command in a forked process. The status of the forked process can be monitored by the caller.

Functions
__init__(
self,
command,
args,
stdin = None,
stdout = None,
stderr = None
)

Overrides sclapp.processes._BackgroundProcess.__init__

command will be run with arguments args. If stdin, stdout, or stderr are specified, the standard I/O file descriptors for the sub-process will be redirected. stdin, stdout, and stderr should be specified as for redirectFds(), or left unspecified if no I/O redirection is desired.

cont(self)

Inherited from sclapp.processes._BackgroundProcess

Re-starts a stopped (paused) process (usually by sending SIGCONT).

getExitSignal(self)

Inherited from sclapp.processes._BackgroundProcess

Returns the signal that caused the process to terminate.

getExitStatus(self)

Inherited from sclapp.processes._BackgroundProcess

Returns the exit status of the process.

getPid(self)

Inherited from sclapp.processes._BackgroundProcess

Returns the PID (process ID) of the process.

isRunning(self)

Inherited from sclapp.processes._BackgroundProcess

Returns True if process is running, False otherwise.

isStopped(self)

Inherited from sclapp.processes._BackgroundProcess

Returns True if process is stopped, False otherwise.

kill(self, signum = 2)

Inherited from sclapp.processes._BackgroundProcess

Kills the process, or sends an arbitrary signal (specified by the signum argument) to the process. Returns False if the process doesn't appear to be running in the first place, True otherwise.

reap(self)

Inherited from sclapp.processes._BackgroundProcess

Tries to reap the process. Returns True if successful, False otherwise.

run(self)

Overrides sclapp.processes._BackgroundProcess.run

Runs the command.

stop(self)

Inherited from sclapp.processes._BackgroundProcess

Stops (pauses) the process (usually by sending SIGSTOP).

wait(self, block = True)

Inherited from sclapp.processes._BackgroundProcess

Blocks until the process has terminated. If the block argument is False, process will simply be reaped (if it has terminated), and the function will return. Returns True if process is reaped by this call, False otherwise.

Class BackgroundFunction
BackgroundFunction (sclapp.processes._BackgroundProcess)

Runs a function in a forked background process. The status of the forked process can be monitored by the caller during function execution.

Functions
__init__(
self,
function,
args,
kwargs,
stdin = None,
stdout = None,
stderr = None
)

Overrides sclapp.processes._BackgroundProcess.__init__

cont(self)

Inherited from sclapp.processes._BackgroundProcess

Re-starts a stopped (paused) process (usually by sending SIGCONT).

getExitSignal(self)

Inherited from sclapp.processes._BackgroundProcess

Returns the signal that caused the process to terminate.

getExitStatus(self)

Inherited from sclapp.processes._BackgroundProcess

Returns the exit status of the process.

getPid(self)

Inherited from sclapp.processes._BackgroundProcess

Returns the PID (process ID) of the process.

isRunning(self)

Inherited from sclapp.processes._BackgroundProcess

Returns True if process is running, False otherwise.

isStopped(self)

Inherited from sclapp.processes._BackgroundProcess

Returns True if process is stopped, False otherwise.

kill(self, signum = 2)

Inherited from sclapp.processes._BackgroundProcess

Kills the process, or sends an arbitrary signal (specified by the signum argument) to the process. Returns False if the process doesn't appear to be running in the first place, True otherwise.

reap(self)

Inherited from sclapp.processes._BackgroundProcess

Tries to reap the process. Returns True if successful, False otherwise.

run(self)

Overrides sclapp.processes._BackgroundProcess.run

stop(self)

Inherited from sclapp.processes._BackgroundProcess

Stops (pauses) the process (usually by sending SIGSTOP).

wait(self, block = True)

Inherited from sclapp.processes._BackgroundProcess

Blocks until the process has terminated. If the block argument is False, process will simply be reaped (if it has terminated), and the function will return. Returns True if process is reaped by this call, False otherwise.

Functions
redirectFds(stdin = None, stdout = None, stderr = None)
waitPid(pid, block = True)