Using and writing Ghostscript testing scripts

General overview
Running tests

Individual tests
Regression testing

Writing new tests

This document describes how to use the scripts located in the toolbin/tests directory, and conventions for writing new testing scripts.

For other information, see the Ghostscript overview and the documentation related to maintaining Ghostscript.

General overview

The test scripts discussed here are written in Python, a language whose implementation is freely available from http://www.python.org. The scripts require Python version 2.1 or later.

Running tests

On Unix and Linux systems, test scripts written in Python can be executed directly simply by typing their name into the shell, e.g.,

toolbin/tests/check_source.py

On other systems, it may be necessary to invoke Python explicitly, e.g.,

python toolbin/tests/check_source.py

The test scripts will print information about any failures that occur.

Individual tests

The individual test scripts are named toolbin/tests/check_xxx.py (if they do not run Ghostscript) or toolbin/tests/gscheck_xxx.py (if they do run Ghostscript). Any script with such a name can be run individually, and is also normally run as part of regression testing (described in the next section).

We don't list the individual test scripts here, because any such documentation would inevitably be out of date most of the time. Each of these scripts contains documentation about what it tests: we suggest you read the documentation in the scripts.

Regression testing

We run a nightly regression test to discover any obvious problems caused by code checked in the previous day. Here is a list of the scripts and supporting files that make up the regression test.

Top-level scripts

dump_testdb: A debugging script that will print the contents of the database defined by gsconf.testdatadb.

make_testdb: This script creates an initial test database. It uses gsconf.baselinegs to create raster data from the test files and computes their MD5 sums and stores them in the gsconf.testdatadb database.

make_two_versions: A helper script to make two versions of a particular file for visual diffing or manual analysis. When a test fails nightly regression, this is generally the first investigative step.

make_two_pdfversions: Same as above, except for pdfwrite regressions.

testdiff: this script provides the difference between two sets of regression results. if end date is omitted, the current date will be used.

revert_baseline: In cases where a baseline has been accidentally updated, this script will revert the database entry to the MD5 sum computed with gsconf.baselinegs.

revert_pdfbaseline: Same as above, except for pdfwrite baselines.

run_nightly: This is the control script (usually invoked by cron) that controls the nightly test run. It's responsible for checking the latest code out of CVS, building a new Ghostscript to compare with and launching the test suite via run_regression.

run_regression: This script runs the full gamut of regression tests using files from gsconf.comparedir. It differentiates files by extension and controls what tests get run and with what options.

update_baseline: This script is invoked to update the MD5 sum in the test database when a nightly regression is really a progression. Generally after noticing that the output from make_two_versions is acceptable or better, this script is run to log the changes to the database.

update_pdfbaseline: Same as above, except for pdfwrite baselines.

get_baselines: This script returns the change log for the test database starting at a given date.

Support files

fuzzy.c: A fuzzy comparison tool appropriate for tests where exact binary matches aren't appropriate.

gsconf.py: This is the configuration framework for the scripts above. It reads configuration files and makes available those configuration options to the rest of the testing framework.

testing.cfg.example: This is an example configuration file for the scripts above. It controls where files are found, where Ghostscript executables are and the location of the test database. Most test configuration will be in this file. It must be copied to testing.cfg in order for the tests to find it.

testing.cfg: This is the configuration file for the scripts above. It controls where files are found, where Ghostscript executables are and the location of the test database. Most test configuration will be in this file. The file testing.cfg.example above can be used as a template.

gstestgs.py: This provides classes for running tests that actually execute Ghostcript.

gsparamsets.py: What parameters tests get run with by default are stored in this file. Between this configuration information and the information in gsconf.py all configurable testing parameters should be covered.

gssum.py: Helper functions that compute, compare and store MD5 sums.

rasterdb.py: Hepler functions for accessing the raster database.

Writing new tests

Some of Ghostscript's test scripts follow a set of conventions that allow them to be run either stand-alone or as part of a suite; in particular, they can be run as part of the nightly regression test suite. In this section, we provide pointers to documentation on how to write new tests that follow these conventions, since that will make them the most useful.

The test scripts are based on Python's unittest module. We suggest that if you are not familiar with this module, you read the documentation, which is available at http://www.python.org/doc/current/lib/module-unittest.html.

Ghostscript specializes the unittest module by defining subclasses, which all individual tests use in place of those in unittest. These subclasses are defined in toolbin/tests/gstestutils.py.

Since code documentation separate from the code itself is always out of date, we have decided to maintain the primary documentation for writing new tests in gstestutils.py rather than here in a separate document. Please read that file for more information.

This file is part of AFPL Ghostscript. See the Aladdin Free Public License (the "License") for full details of the terms of using, copying, modifying, and redistributing AFPL Ghostscript.

Ghostscript version 8.11, 16 August 2003