Plum Hall, Inc.

Plum Hall is the world leader in making compiler validation suites and software for the programmer and software development community.

Suite++®:
The Plum Hall Validation Suite for C++

Suite++® has become the industry's preferred test method for testing C++ compilers against ISO/IEC 14882:1998 (which is the ANSI/ISO C++ Standard). The C++ Standard was published by ISO and ANSI in September 1998. Suite++ was produced by industry-leading authors recognized for making the most well-informed judgments about the cases they write. The authoring team includes Thomas Plum (Convener of WG21, the ISO C++ standards committee) and Daniel Saks (for six years the Secretary of X3J16, the ANSI C++ standards committee).

Plum Hall defines a "test fact" as one sentence from the C++ Standard, or one alternative rule. Those sentences which imply both positive tests and negative tests are counted as two cases. The total of "test facts" in the C++ Standard is more than 2600 positive test facts and more than 2000 negative test facts, for a total of at least 4600 test facts. (These totals cover the language definition of C++, and do not include any cases for the library.) Each positive fact may embody several executable tests within its source code. Suite++ includes tests for the more difficult features, like templates and exceptions.

Plum Hall is committed to encouraging the maximum feasible compatibility between C and C++, a market requirement of the majority of C and C++ users. To promote this compatibility, The Plum Hall Validation Suite for C™ has been enhanced (in the CONFORM section) with specific switches to exclude all test cases whose behavior is meant to change in C++. That is, The Plum Hall C Validation Suite for C is configurable to test for the "C-like" requirements of C++. All those new aspects of C++, the changes to C and the completely new features, are tested in Suite++. Because of this structure, Suite++ must be used in conjunction with The Plum Hall Validation Suite for C.

The license fee for the full Suite++ provides a single site, "two-mile radius" source code license for use, with no limit on the number of destination machines or users within the radius. Suite++ is only licensed in conjunction with a license for The Plum Hall Validation Suite for C. Any future modifications to the "language" component in the C++ Standard will be provided as part of optional ongoing Maintenance. The "library" section is available under a separate license for LibSuite++®: The Plum Hall Validation Suite for the C++ Library

TEST CASE DESIGN PRINCIPLES

Each "test fact", as defined above, is tested by hand-crafted, individually-designed C++ executable code. A few representative examples are attached below; they will serve to define the intended level of detail. We provide a level of coverage comparable to that in The Plum Hall Validation Suite for C, carefully hand-crafted examples, sometimes the simplest possible example, sometimes more complicated. We don't "bulk up" Suite++ with redundant or unverified tests. The tests are all meant to be intuitively graspable pieces of code. We avoid enormous constructs whose correctness is beyond human parsing.

For C++, of course, the problem of interaction between features is much more severe than it is in C. We have identified the following list, which is illustrative but not exhaustive:

Attempting to cover all possible interactions of features in every "test fact" would be infeasible, and each such case would contribute little marginal utility. Our approach has been to select combinations of features to be exemplified in different facts. That is, we will attempt to provide examples of all important interactions somewhere, in one or more facts.

STRUCTURE OF THE DELIVERY HARNESS

We have developed a major upgrade in the area of multiple-target and freestanding support. Here are the major features of the harness:

Multiple trees: There is one directory tree of original sources (the "source tree"), but each installation may have any number of "destination trees", each of which is for a different compiler, a different target machine, a different release version, etc. The source tree can reside on a LAN server, with destination trees in various workstations. Or, the source tree can be on a read-only medium, to minimize version-control and backup concerns.

Distinction between host and target: The tool programs (txtchk, unarc, score, etc.) may need to be compiled by a (host) compiler different from the compiler being tested. This entails a distinction between the tested-compiler's headers (defs.h, compil.h, and machin.h) and host-compiler headers (hodefs.h, hocompil.h, and homachin.h).

Uniform batch-script conventions: Some Plum Hall clients have always used MAKE, with the MAKEFILEs we supply. Other clients need the configurability provided by the SNAP tool. Now both groups can simplify their harnessing be configuring their own local versions of the following "batch files" ("shell scripts", "command files", etc):

Embedded-systems (or "Freestanding") support: Suite++ was designed to work testing embedded systems. Only the lowest level output function needs to be implemented and the tests do not presume a local implementation of the Standard C++ Library. The utility-support functions in util.c are written to use only a minimal set of system-support functions in sdutil.h . Putting #define FREESTANDING 1 into defs.h will turn off "hosted-only" code in util.c , etc.

The following, are some illustrative examples from Suite++:

PLUM HALL INC: SAMPLE ONE


    // Suite++: The Plum Hall Validation Suite for C++
    // Unpublished copyright (c) 1991, Plum Hall Inc (Editor)
    // VERSION: 1.05
    // DATE: 1992-06-30
    // SECTION AUTHOR: Daniel Saks
    // As per your license agreement, your distribution is not to be
    // moved or copied outside the Designated Site
    // without specific permission from Plum Hall Inc.

    --------- Excerpt from s04.c ---------------------------------------

    // 46p17c: ptr to class converted to ptr to accessible base class if unambiguous
            struct B { int bi; };
            class D : private B
                {
                int di;
            public:
                void foo()
                    {
                    chk(bar() == this);
                    aeq(&bar()->bi, &bi); // checks 46p18 too
                    };
                const B *bar() const { return this; }
                };
            D d;
            d.foo();

    // 47p12a: ref to class converted to ref to accessible (10) base class if unambiguous (10.1)
            struct B { int bi; };
            struct D : B { int di; };
            D d;
            D &dr = d;
            B &br = dr;
            chk(&br == &dr);
            aeq(&br.bi, &dr.bi); // checks 47p13 too

PLUM HALL INC: SAMPLE TWO


    // Suite++: The Plum Hall Validation Suite for C++
    // Unpublished copyright (c) 1991, Plum Hall Inc (Editor)
    // VERSION: 1.05
    // DATE: 1992-06-30
    // SECTION AUTHOR: Daniel Saks
    // As per your license agreement, your distribution is not to be
    // moved or copied outside the Designated Site
    // without specific permission from Plum Hall Inc.

    --------- Excerpt from m04.in ("Negative Tests" for diagnostics) ------------

    // #005 46p17a: ptr to class not converted to ptr to inaccessible base class.
    void ignore(void *);
    int main()
        {
        struct B { int bi; };
        struct D : private B { int di; };
        D d;
        d.di = 0;
        B *bp = &d; // conversion to inaccessible base
        ignore(bp);
        ignore(&d);
        return 0;
        }

    // #006 46p17b: ptr to class not conv to ptr to accessible base class if conversion is ambiguous.

    void ignore(void *);
    int main()
    {
        struct B { int bi; };
        struct D1 : B { int di1; };
        struct D2 : B { int d21; };
        struct D : D1, D2 { int di; };
        D d;
        d.di = 0;
        B *bp = &d; // ambiguous conversion
        ignore(bp);
        ignore(&d);
        return 0;
    }

PLUM HALL INC: SAMPLE THREE


    // Suite++: The Plum Hall Validation Suite for C++
    // Unpublished copyright (c) 1991, Plum Hall Inc (Editor)
    // VERSION: 1.05
    // DATE: 1992-06-30
    // SECTION AUTHOR: Thomas Plum
    // As per your license agreement, your distribution is not to be moved or copied outside the Designated Site
    // without specific permission from Plum Hall Inc.

    --------- Excerpt from s06.c ---------------------------------------
    // 67p32b valid to jump past init decl if from point where already init'ed
            int inits = ivalue(0);

            for (short n = ivalue(1); n <= 3; ) // two iterations
                {
            before_decl_p32b:
                int inner = inits++; // three traversals
                ignore(&inner);
                if (n++ == 1) goto before_decl_p32b;
                }
            ieq(inits, ivalue(3));

    // 67p41b    init of static is done the first time control passes thru its decl

            struct X
                {
                int i;
                X(int ii) { i = ii; }
                };
            static int inits = ivalue(0);

            for (short n = ivalue(1); n <= 2; ++n) // two iterations
                {
                static X inner (inits++); // two traversals, only one init
                ignore(&inner);
                }
            ieq(inits, ivalue(1));