Plum Hall Test Suites
The Plum Hall Test Suites test for conformance of C and C++ compilers and libraries
to ISO and ANSI standards as defined by the respective organizations. The test case suites
are released every year in August, tracking conformance to the (developing) standards
which are released on a 3 year basis. The Plum Hall Suites test against standards
as they evolve for each standards release from C and C++-11, through C++-14, C++-17, C++20,
C++23, and C++26 (proposed)*. The test suites evolve, tests
being added, disabled, and modified as the standards evolve.
The Plum Hall Test Suites have value in testing along a number of dimensions:
- First and foremost is testing against the current C and C++ standards, with tests
specifically associated down to the level of a single phrase in the standard.
- Plum Hall has a long history of standards conformance testing. The tests have developed
over time since the inception of C/C++. The test case suites have grown as the language
standards have evolved.
- There is one directory tree of original test case sources (the "source tree"),
but each installation may have any number of "destination trees", each of which is for a
different compiler, a different target machine, a different release version, etc.
- Embedded-systems (or "Freestanding") support: The test suites were designed to
test on embedded systems. Only the lowest level output function needs to be implemented
and the tests do not presume a local implementation of the Standard C++ Library.
- The test suite infrastructure is configured that the tests can be selectively tested against
different standards years, new and old. So, for example, the tests can be configured to test against an older
standard year, such as C/C++-11 or C/C++-17. Tests against later or earlier language features will be skipped.
- Certain language or library features may be turned on/off at will with build test flags. For example,
complex number arithmetic tests may be turned off if not relevant to a particular environment.
- Newer tests are associated with specific standard year proposals, such as c2x or c2y. Features
of the languages come and go. A specific feature may only be present in one particular standard
year and then be deprecated, for example. In this way compilers that are currently being written
for the newest language proposals can check their new implementations.
- flags.h has been split into flags.h, containing release year-specific flag settings and
compiler-flags.h, containing compile flags specific to a particular compiler. The specific flags
are enabled through ifdefs.
Availability of the 24a Release
The 24a release of the Plum Hall Test Suites is available now. This release contains infrastructure
improvements, bug fixes and preliminary tests for the proposed C++26 / C26 standards.
Please see details here: News
Reporting has also been improved. HTML Reports, which may be viewed in a web browser,
are produced with each test run.
*Tests actually reach as far back as C/C++-90, though formal support starts from C/C++-11.