doxytest.py

Introduction

doxytest.py is a Python script that generates standalone C++ test programs from code embedded in header file comments.

Its inspiration is a Rust feature called doctests.

A Rust doctest is a snippet of sample code in the documentation block above a function or type definition. The example becomes part of the documentation that you see on docs.rs. Moreover, examples are typically formulated using assertions and become part of the project’s test suite.

In a similar vein, doxytest.py looks for comment lines in C++ header files that start with /// and which contain a fenced code block — a doctest. Any leading whitespace before the /// is ignored.

The script extracts the doctests, wraps them in try blocks to catch any failures, and then embeds them in a standalone test program.

Of course, for this to be useful, doctest code must be formulated as a test. To that end, Doxytest also supplies assertions that you can use in your doctests. These are relatively simple macros that capture the values of the arguments passed to an assertion along with some other helpful information. They throw a particular exception if the assertion fails — the test program captures and processes that exception. The assertion macros are defined and included in every test program generated by doxytest.py.

Example

It is easiest to explain doxytest by building a simple example.

Suppose we have a Student.h header file that opens with the following comment block:

/// @brief This class represents a student with a name and a unique student ID.
///
/// Student IDs are incremented for each new Student.
///
/// # Examples
/// ```
/// Student student_1{"John Doe"};
/// Student student_2{"Mary Doe"};
/// assert_eq(student_1.name(), "John Doe");
/// assert(student_2.name() == "Mary Doe");
/// assert_eq(student_2.id(), student_1.id() + 1, "Oops, expected student-ids to be consecutive!");
/// ```
/// class Student {
///     ...
/// };

Here, for the sake of exposition, we have used both Doxytest’s macros: assert and assert_eq. In the third invocation, we demonstrate how to add a custom message to print if the assertion fails.

The doxytest.py script extracts the fenced code block and embeds it in a test program. By default, the source file for the test program is called doxy_Student.cpp and the script creates the file in the current directory.

The entire fenced code block, including all its assertions, constitutes a single doctest. The code within the block constitutes a single test, and any failures are caught and reported.

The generated test file will include a directive for the Student.h header. It will also define the assert and assert_eq macros, and of course, contain a main program with a little bit of scaffolding to run this doctest and any others it extracted from the header file.

The script will convert the comment block shown above like this:

 // ... boilerplate ...
1    std::print(stderr, "test {}/{} ({}:{}) ... ", ++test, test_count, header_file, header_line);
2    try {
        Student student_1{"John Doe"};
        Student student_2{"Mary Doe"};
3        assert_eq(student_1.name(), "John Doe");
        assert(student_2.name() == "Mary Doe");
        assert_eq(student_2.id(), student_1.id() + 1, "Oops, expected student-ids to be consecutive!");
4    } catch (const doxy::error& failure) { handle_failure(failure.what()); }
5    if (test_passed) std::println(stderr, "pass");

1

The arguments to the print method track the doctest number, the file name, and the line number of the doctest in the corresponding header file.

2

The test code is wrapped in a try block to catch any assertions that fail.

3

The assert_eq & assert macros throw a doxy::error exception if any assertion fails.

4

If there is any assertion failure in the block, we increment the total fail count, print an error message, and, if the number of failures exceeds the maximum allowed, the program exits with a non-zero status.

5

If there are no failures, you see a “pass” message for this doctest and the program continues to the next one.

Notes

• The generated test source files are designed to be compiled as standalone test programs. However, you often need to use the script’s --include option to include extra headers that are required to compile the test code. The doxytest.py script always includes the header file it is processing in the test program (so Student.h is always included in doxy_Student.cpp).
• Error messages you get from the test program show the location of the corresponding doctest in the associated header file. The line number in the test program itself is of little interest; the critical information is the line number in the source header file.
• The try block will exit on the first failed assertion it encounters. This is a feature, not a bug and is also the way Rust handles doctests like this.
• By default, test programs allow for up to 10 failures before they exit with a non-zero status. Experience shows that with that many failures, there is a fundamental flaw that is “obvious” from a small number of error messages.
• You can adjust that failure threshold with --max_fails <count> if you need a different limit.
• When the test program finishes, it prints a summary of the test results, including a summary of the messages from any failed tests.

There are several options that you can pass to the script, which we will cover in the next section. You can always see the complete list of options by running doxytest.py --help.

The Full Example

Here is the complete code for Student.h, which includes embedded doctests with deliberate failures.

#pragma once

#include <string>
#include <format>

/// @brief This class represents a student with a name and a unique student ID.
///
/// Student IDs are incremented for each new Student.
///
/// # Examples
/// ```
/// Student student_1{"John Doe"};
/// Student student_2{"Mary Doe"};
/// assert_eq(student_1.name(), "John Doe");
/// assert(student_2.name() == "Mary Doe");
/// assert_eq(student_2.id(), student_1.id() + 1, "Oops, expected student-ids to be consecutive!");
/// ```
class Student {
public:
 /// @brief Construct a new Student with the given `name`.
 ///
 /// # Examples
 /// ```
 /// Student student{"John Doe"};
 /// assert_eq(student.name(), "John Doe");
 /// ```
    Student(std::string_view name) : m_name(name), m_student_id(next_student_id++) {}

 /// @brief Provides read/write access to the Student's name.
 ///
 /// # Examples
 /// ```
 /// Student student{"John Doe"};
 /// assert_eq(student.name(), "John Doe");
 /// auto id = student.id();
 /// student.name() = "Jane Doe";
 /// assert_eq(student.name(), "Jane Doe");
 /// assert_eq(student.id(), id);
 /// ```
    constexpr auto& name() { return m_name; }

 /// @brief Provides read-only access to the Student's name.
 ///
 /// # Examples
 /// ```
 /// Student student{"John Doe"};
 /// assert_eq(student.name(), "John Doe");
 /// ```
    constexpr auto name() const { return m_name; }

 /// @brief Provides read-only access to the Student's ID.
 ///
 /// Student IDs start at 1000 and are incremented for each new Student.
 ///
 /// # Examples
 /// ```
 /// Student s1{"John Doe"};
 /// Student s2{"Jane Doe"};
 /// assert_eq(s2.id(), s1.id() + 1);
 /// ```
    constexpr auto id() const { return m_student_id; }

 /// @brief Compares two students for equality (only the ID matters).
 ///
 ///
 /// # Examples
 /// ```
 /// Student john{"John Doe"};
 /// Student jane{"Jane Doe"};
 /// assert_eq(john, john, "We're expecting this to pass!");
 /// assert_eq(john, jane, "As expected, this failed");
 /// ```
    constexpr bool operator==(const Student& other) const { return m_student_id == other.m_student_id; }

private:
    std::string m_name;
    std::size_t m_student_id;

    static inline std::size_t next_student_id = 1000;
};

/// @brief Specialise `std::formatter` for `Student`.
///
/// # Examples
/// ```
/// Student student{"John Doe"};
/// assert_eq(std::format("{}", student), "John Doe (1000)", "This is a deliberate error!");
/// ```
template<>
struct std::formatter<Student> {
    constexpr auto parse(std::format_parse_context& ctx) { return ctx.begin(); }

    template<typename FormatContext>
    auto format(const Student& student, FormatContext& ctx) const {
        return std::format_to(ctx.out(), "{} ({})", student.name(), student.id());
    }
};

Running the Script

Run the script on this header file:

doxytest.py Student.h

Which will print the message:

Generated test file: doxy_Student.cpp with 7 test cases.

The script creates the file doxy_Student.cpp in the current directory, which you can compile and run:

1g++ -std=c++23 -o doxy_Student doxy_Student.cpp
./doxy_Student

1

You need to use a compiler that has access to std::format and std::println, etc.

Because we deliberately introduced some failures, the output will look something like this:

1Running 7 tests extracted from: `Student.h`
2test 1/7 (Student.h:11) ... pass
test 2/7 (Student.h:23) ... pass
test 3/7 (Student.h:32) ... pass
test 4/7 (Student.h:45) ... pass
test 5/7 (Student.h:56) ... pass
3test 6/7 (Student.h:67) ... FAIL

4FAILED `assert_eq(john, jane)` [Student.h:67]
As expected, this failed
lhs = John Doe (1007)
rhs = Jane Doe (1008)

5test 7/7 (Student.h:85) ... FAIL

FAILED `assert_eq(std::format("{}", student), "John Doe (1000)")` [Student.h:85]
This is a deliberate error!
lhs = John Doe (1009)
rhs = John Doe (1000)


6Test FAIL summary for `Student.h`: Ran 7 of a possible 7 tests, PASSED: 5, FAILED: 2
--------------------------------------------------------------------------------------
FAILED `assert_eq(john, jane)` [Student.h:67]
As expected, this failed
lhs = John Doe (1007)
rhs = Jane Doe (1008)

FAILED `assert_eq(std::format("{}", student), "John Doe (1000)")` [Student.h:85]
This is a deliberate error!
lhs = John Doe (1009)
rhs = John Doe (1000)

1

The test program prints a summary of the number of doctests extracted from the header file.

2

You see the location of the first doctest in the header file (line 11 of Student.h) and that it passed.

3

The sixth doctest failed (line 67 of Student.h).

4

The error message shows the values of the arguments to the failed assertion and the custom message we provided.

5

Testing continues, and you see the location of the next doctest, which also fails (line 85 of Student.h).

6

Finally, you see a summary of the test results showing the number of tests run, the number of tests that passed, and the number of tests that failed. A summary of the failed test messages follows that.

Header files can contain dozens or more doctests. The summary of test failure messages at the end saves you from having to scroll back through a lot of irrelevant test program output.

See examples/Student/ErrorFreeStudent.h in the repository for the corrected version of the Student class.
The test program generated from this corrected version will pass with no errors.

doxytest.py ErrorFreeStudent.h
g++ -std=c++23 -o doxy_ErrorFreeStudent doxy_ErrorFreeStudent.cpp
./doxy_ErrorFreeStudent

Outputs:

Running 7 tests extracted from: `ErrorFreeStudent.h`
test 1/7 (ErrorFreeStudent.h:11) ... pass
test 2/7 (ErrorFreeStudent.h:23) ... pass
test 3/7 (ErrorFreeStudent.h:32) ... pass
test 4/7 (ErrorFreeStudent.h:45) ... pass
test 5/7 (ErrorFreeStudent.h:56) ... pass
test 6/7 (ErrorFreeStudent.h:67) ... pass
test 7/7 (ErrorFreeStudent.h:85) ... pass
[ErrorFreeStudent.h] All 7 tests PASSED

Next Steps

See the full reference for all the options understood by doxytest.py.

Get more information about the two provided assertion macros and examine some more advanced custom use scenarios.

If you use CMake, be sure also to check out doxytest.cmake.