Introduction

Doxytest is a tool for generating C++ test programs from documentation, for example, from code embedded in header file comments.

The inspiration is a Rust feature called doctests.

A Rust doctest is a snippet of sample code in the documentation block above a function or type definition. The example becomes part of the documentation you see on docs.rs. Moreover, examples are typically formulated using assertions and become part of the project’s test suite.

Here is a very high-tech Rust add function — PhD-level sort of stuff:

/// Adds two numbers together.
///
/// # Examples
/// ```
/// assert_eq!(add(1, 2), 3);
/// ```
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

The comment block above the function definition contains a typical working code example. The sample is wrapped in a triple backtick-delimited code block and formulated as a test using Rust’s assert_eq! macro. It is a doctest.

Rust’s cargo doc command picks out specially marked comments and turns them into your project’s “official” documentation. Doxygen can similarly generate documentation for C++ functions.

However, Rust doctests are not just part of the documentation; they are also used to generate test programs. The cargo test command collects all the project’s doctests by looking for comments containing triple backtick-delimited code blocks. The extracted doctests are then compiled and run as test programs.

Looking through the source code of Rust crates, you will see lots of embedded doctests. Example code is usually crafted as tests using Rust’s assert! and assert_eq! macros.

At first, I was sceptical about the value of this scheme. There are many comment lines in a typical Rust source file! That can make them daunting to navigate.

However, after embracing my editor’s ability to fold comments and using Rust’s doctest feature for a while, I began to wish to do the same in C++. Why not have simple code samples that document how to use a function and serve as basic tests?

Hence, the Python script doxytest.py which looks for fenced code blocks — doctests. The script extracts the doctests, wraps them in try blocks to catch any failures, and then embeds them in a standalone test program.

A companion CMake module doxytest.cmake integrates the script into a CMake project.

Scope

Doxytest is a simple tool for generating C++ test programs from code embedded in header file comments. It’s not a replacement for a full-blown testing framework like Catch2 or Google Test.

Doctests are typically just a few lines of code crafted as tests — they primarily illustrate how to use a function or class. You’re unlikely to write many complicated edge case codes as comments in a header file.

On the other hand, once you get used to the idea, you tend to write a doctest for almost every function or class you write. So, while the depth of test coverage may not be as high as that of a full-blown testing framework, the breadth of coverage is excellent.

That breadth of coverage is very valuable when adding new features or fixing bugs. Compiling all the doctests can serve as a quick sanity check to see if you have inadvertently broken something else. And, of course, running the tests will help you catch at least basic regression errors.

If you are using Rust in an IDE like VSCode, you can run the doctest for an individual method by clicking the “Run” button above the method. A “Run” button above a type definition runs all the doctests for that type.

We haven’t yet implemented a doxytest extension for VSCode. However, we have a CMake module, doxytest.cmake, that can automate extracting those tests and adding build targets for each resulting test program. This is already very useful, and if you are using the CMake Tools extension for VSCode, it lets you run doctests with a single click.

Summary

Doxytest has two main components:

• The main event is a Python script doxytest.py that generates standalone C++ test programs from code embedded in documentation. Typically, these are comments above types and methods in header files.
• There is also a companion CMake module called doxytest.cmake that can automate extracting those tests and adding build targets for each resulting test program.

If you don’t use CMake, you can use the Python script directly. The module is a convenience wrapper around the script. The script has no CMake dependencies.

Python Script Features

• The Python script generates standalone C++ test programs from code embedded in documentation.
• It can generate those programs by scanning multiple files or directories in a single run.
• It can generate a single test program containing all the combined test code from multiple sources.
• The script supports a variety of options for controlling the generation of test programs:
  • You can specify the output directory for the generated test programs.
  • You have control over the filename prefix for the generated test programs.
  • You can specify extra header files to include in the generated test programs.
  • You can control which files in a directory to scan for doctests.
• The script has rudimentary dependency tracking, which means that the generated test programs are only recreated if the corresponding header files change.

CMake Module Features

• The CMake module automates the process of extracting those tests, and it adds build targets for each resulting test program. It will also let you easily run individual tests in an IDE like VSCode with a single click.
• Like the script, the module can generate test programs for multiple sources and directories in a single run.
• Like the script, the module can generate a single test program containing all the combined test code from multiple sources.
• The module supports a variety of options for controlling the generation of test programs:
  • You can specify the output directory for the generated test programs.
  • You have control over the filename prefix for the generated test programs.
  • You can specify custom header files to include in the generated test programs.
  • You can control which files in a directory are scanned for doctests.
• The module uses CMake to handle the dependency tracking so that the generated test programs are only rebuilt if the corresponding header files change or the script is updated.

Basic Usage

Here is a C++ header file, add.h, equivalent to add.rs above. Like the Rust version, there is a comment block containing a sample code snippet:

/// @brief Adds two numbers together.
///
/// # Examples
/// ```
1/// assert_eq(add(1, 2), 3);
/// ```
constexpr int add(int a, int b) { return a + b; }

1

assert_eq is used to assert that two values are equal.
On failure, it prints the argument values and may terminate the program.

C++ has an assert macro, but it is not very useful for testing because it does not print the values of the expression that failed the assertion. For that reason, in the doxytest-generated code, we override the’ assert’ macro with our own assert and also provide the assert_eq macro, as referenced in the example. The two doxytest assertion macros are automatically defined and included in the test programs generated by doxytest.py. They only affect test code extracted from doctests.

You can run the doxytest.py script on this header file:

doxytest.py add.h

This command will print the message Generated test file: doxy_add.cpp with 1 test cases and then create the file doxy_add.cpp in the current directory.

If you look at doxy_add.cpp, you will see an include directive for add.h, the definition of our version of assert and assert_eq, and a main program that contains the tests (just one in this example) in their own try blocks.

You can compile the test program using your favourite C++ compiler; for example, using g++:

g++ -std=c++23 -o doxy_add doxy_add.cpp

The doxytest generated test source uses std::println and friends, so typically you need to invoke the compiler with an appropriate level of “modernity” to access those features.

Running ./doxy_add prints some output along the lines:

Running 1 tests extracted from: `add.h`
test 1/1 (add.h:4) ... pass
[add.h] All 1 tests PASSED

Next Steps

Check out the doxytest.py script story and its full reference page.

Get more information about the two provided assertion macros and examine some more advanced custom use scenarios.

If you use CMake, be sure also to check out doxytest.cmake.