diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 302c38a93f..88f39f18ea 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -26,7 +26,8 @@ To contribute patches: #. Fork the upstream `CMake Repository`_ into a personal account. #. Run `Utilities/SetupForDevelopment.sh`_ for local git configuration. #. See `Building CMake`_ for building CMake locally. -#. See the `CMake Source Code Guide`_ for coding guidelines. +#. See the `CMake Source Code Guide`_ for coding guidelines + and the `CMake Testing Guide`_ for testing instructions. #. Create a topic branch named suitably for your work. Base all new work on the upstream ``master`` branch. Base work on the upstream ``release`` branch only if it fixes a @@ -49,6 +50,7 @@ The merge request will enter the `CMake Review Process`_ for consideration. .. _`Utilities/SetupForDevelopment.sh`: Utilities/SetupForDevelopment.sh .. _`Building CMake`: README.rst#building-cmake .. _`CMake Source Code Guide`: Help/dev/source.rst +.. _`CMake Testing Guide`: Help/dev/testing.rst .. _`commit messages`: Help/dev/review.rst#commit-messages .. _`CMake Review Process`: Help/dev/review.rst diff --git a/Help/dev/README.rst b/Help/dev/README.rst index abb4ac9f1a..2a2d32d9d9 100644 --- a/Help/dev/README.rst +++ b/Help/dev/README.rst @@ -38,10 +38,12 @@ CMake developer documentation is provided by the following documents: * The `CMake Source Code Guide`_. * The `CMake Documentation Guide`_. +* The `CMake Testing Guide`_. * The `CMake Experimental Features Guide`_. .. _`CMake Source Code Guide`: source.rst .. _`CMake Documentation Guide`: documentation.rst +.. _`CMake Testing Guide`: testing.rst .. _`CMake Experimental Features Guide`: experimental.rst Maintainer Documentation diff --git a/Help/dev/integration-testing.rst b/Help/dev/integration-testing.rst index 1b666d6f78..434cf2301a 100644 --- a/Help/dev/integration-testing.rst +++ b/Help/dev/integration-testing.rst @@ -4,7 +4,10 @@ CMake Integration Testing The following documents how to run integration testing builds. See documentation on `CMake Development`_ for more information. +See the `CMake Testing Guide`_ for running the test suite locally. + .. _`CMake Development`: README.rst +.. _`CMake Testing Guide`: testing.rst CMake Dashboard Scripts ======================= diff --git a/Help/dev/source.rst b/Help/dev/source.rst index fa7d620d1d..c3d7ac3355 100644 --- a/Help/dev/source.rst +++ b/Help/dev/source.rst @@ -310,6 +310,7 @@ The CMake source tree is organized as follows. * ``Tests/``: The test suite. See `Tests/README.rst`_. + To run the tests, see the `CMake Testing Guide`_. * ``Utilities/``: Scripts, third-party source code. @@ -331,5 +332,6 @@ The CMake source tree is organized as follows. See `Utilities/Release/README.rst`_. .. _`CMake Documentation Guide`: documentation.rst +.. _`CMake Testing Guide`: testing.rst .. _`Tests/README.rst`: ../../Tests/README.rst .. _`Utilities/Release/README.rst`: ../../Utilities/Release/README.rst diff --git a/Help/dev/testing.rst b/Help/dev/testing.rst index 2179f60c06..f90cef7703 100644 --- a/Help/dev/testing.rst +++ b/Help/dev/testing.rst @@ -1,8 +1,111 @@ CMake Testing Guide ******************* -See `CMake Integration Testing`_ for running integration testing builds. +The following is a guide to the CMake test suite for developers. See documentation on `CMake Development`_ for more information. -.. _`CMake Integration Testing`: integration-testing.rst +See `CMake Integration Testing`_ for running integration testing builds. + +See `Tests/README.rst`_ for the test suite layout in the source tree. + .. _`CMake Development`: README.rst +.. _`CMake Integration Testing`: integration-testing.rst +.. _`Tests/README.rst`: ../../Tests/README.rst + +Running Tests in the Build Tree +=============================== + +After `Building CMake`_, one may run the test suite in the build tree +using `ctest(1)`_: + +* With a single-configuration CMake generator, such as ``Ninja`` + or ``Unix Makefiles``, one may simply run ``ctest``: + + .. code-block:: console + + $ ctest + +* With a multi-configuration CMake generator, such as + ``Ninja Multi-Config``, ``Visual Studio``, or ``Xcode``, + one must tell ``ctest`` which configuration to test + by passing the ``-C `` option: + + .. code-block:: console + + $ ctest -C Debug + +Some useful `ctest(1)`_ options include: + +``-N`` + List test names without running them. + +``-V`` + Show verbose output from each test. + +``-j `` + Run to run up to ``N`` tests concurrently. + +``-R `` + Select tests for which the regular expression matches a substring + of their name. + +Cleaning Test Build Trees +------------------------- + +Many CMake tests create their own test project build trees underneath +the ``Tests/`` directory at the top of the CMake build tree. These +build trees are left behind after testing completes in order to +facilitate manual investigation of results. Many of the tests do *not* +clean their build trees if they are run again, with the exception of +tests using the `RunCMake`_ infrastructure. + +In order to clear test build trees, drive the ``test_clean`` custom target +in the CMake build tree: + +.. code-block:: console + + $ cmake --build . --target test_clean + +This removes the ``Tests/`` subdirectories created by individual tests +so they will use a fresh directory next time they run. + +.. _`Building CMake`: ../../README.rst#building-cmake +.. _`ctest(1)`: https://cmake.org/cmake/help/latest/manual/ctest.1.html +.. _`RunCMake`: ../../Tests/RunCMake/README.rst + +Running Tests with a Different Generator +======================================== + +After `Building CMake`_ with one CMake generator, one may configure the +test suite using a different generator in a separate build tree, without +building CMake itself again, by defining ``CMake_TEST_EXTERNAL_CMAKE`` +to be the absolute path to the ``bin`` directory containing the ``cmake``, +``ctest``, and ``cpack`` executables. + +For example, after building CMake with the ``Ninja`` generator: + +.. code-block:: console + + $ cmake -B build-ninja -G Ninja -DCMAKE_BUILD_TYPE=Debug + $ cmake --build build-ninja + +one may configure a second build tree to drive tests with the +``Ninja Multi-Config`` generator: + +.. code-block:: console + + $ cmake -B build-nmc-tests -G "Ninja Multi-Config" \ + -DCMake_TEST_EXTERNAL_CMAKE="$PWD/build-ninja/bin" + $ cmake --build build-nmc-tests --config Release + +The second build tree does not build CMake itself, but does configure +the test suite and build test binaries. One may then run tests normally: + +.. code-block:: console + + $ cd build-nmc-tests + $ ctest -C Release + +Note that the configuration with which one drives tests in the second +build tree is independent of the configuration with which CMake was +built in the first. diff --git a/README.rst b/README.rst index 1ae77ef4f7..8a471db0e9 100644 --- a/README.rst +++ b/README.rst @@ -59,7 +59,11 @@ To build the documentation, install `Sphinx`_ and configure CMake with "man" builder. Add ``-DSPHINX_EXECUTABLE=/path/to/sphinx-build`` if the tool is not found automatically. +To run the test suite, run ``ctest`` in the CMake build directory after +building. See the `CMake Testing Guide`_ for details. + .. _`Sphinx`: https://sphinx-doc.org +.. _`CMake Testing Guide`: Help/dev/testing.rst Building CMake from Scratch --------------------------- diff --git a/Tests/README.rst b/Tests/README.rst index 2810fec4ef..4bbbce1951 100644 --- a/Tests/README.rst +++ b/Tests/README.rst @@ -2,8 +2,9 @@ CMake Tests Directory ********************* This directory contains the CMake test suite. -See also the `CMake Source Code Guide`_. +See also the `CMake Testing Guide`_ and the `CMake Source Code Guide`_. +.. _`CMake Testing Guide`: ../Help/dev/testing.rst .. _`CMake Source Code Guide`: ../Help/dev/source.rst Many tests exist as immediate subdirectories, but some tests diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index ea6db54d50..f4bcc572cd 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -6,9 +6,11 @@ precisely checking their return code and stdout/stderr content. The RunCMake infrastructure is useful for testing error cases and diagnostic output. -See also `../README.rst`_ and the `CMake Source Code Guide`_. +See also `../README.rst`_, the `CMake Testing Guide`_, +and the `CMake Source Code Guide`_. .. _`../README.rst`: ../README.rst +.. _`CMake Testing Guide`: ../../Help/dev/testing.rst .. _`CMake Source Code Guide`: ../../Help/dev/source.rst .. _`CMakeLists.txt`: CMakeLists.txt