Covers almost all blocks containing actual code, except:
* Parsed-literal blocks can't be highlighted, including many command
summaries and substitution-heavy docs like find_... commands.
This is a Sphinx limitation.
* Code with errors, like CMP0049, DEPLOYMENT_ADDITIONAL_FILES,
DEPLOYMENT_REMOTE_DIRECTORY, @PACKAGE_INIT@ substitution in the
tutorial, bracket arguments/comments in cmake-language.7 and
cmake-developer.7.
* FindQt4 module, which needs reformatting.
This will now preserve empty values in the TEST_LAUNCHER and
CROSSCOMPILING_EMULATOR target properties for tests added by:
- The add_test() command.
- The ExternalData_Add_Test() command from the ExternalData module.
- The gtest_add_tests() or gtest_discover_tests() commands from the
GoogleTest module.
For the gtest_add_tests() and gtest_discover_tests() commands,
empty elements in the values passed after the EXTRA_ARGS keyword
are also now preserved.
Policy CMP0178 is added to provide backward compatibility with the
old behavior where empty values were silently discarded from the
above cases.
Fixes: #26337
When an external data object was recently created on disk, perhaps
fetched by the current process or another process, it may be
inaccessible on Windows for a short time. Tell our `file(COPY_FILE)`
call to retry a few times to copy the object to the final file path.
Hopefully this will resolve our long-standing spurious failures of the
`Module.ExternalData` test on Windows.
If more than one content link references the same object, the build
system may launch multiple download processes for the same object
concurrently. Use whichever one finishes first, and discard the others.
Without this, we replace the objects and use the last finisher instead
of the first. This is okay on non-Windows platforms where `rename(2)`
gives reliable atomic replacement. However, on Windows platforms and
NTFS this is less reliable. I've observed `MoveFileEx` somehow cause
another process to get `ERROR_SHARING_VIOLATION` when attempting to read
the destination file. We may be able to improve the `file(RENAME)`
implementation on modern Windows 10 versions, but for ExternalData's use
case it is simpler to just not replace existing objects.
During CI builds (at least), download progress is just noise. Allow it
to be suppressed.
Default to `OFF` for Ninja due to the tool's behavior of not showing
output until a command is complete (which makes any progress reporting
of little use) and `ON` otherwise.
Add support for projects to have `Data.txt.md5` *and* `Data.txt.sha512`
where the content links hold hashes for the same file. Check all
`ExternalData_URL_TEMPLATES` entries in order for all available hashes.
The data acquisition is considered a failure if none of the available
URL resources has any of the given hashes. This makes it possible to
have multiple data server resources where all servers do not support all
hashing algorithms.
Per-source copyright/license notice headers that spell out copyright holder
names and years are hard to maintain and often out-of-date or plain wrong.
Precise contributor information is already maintained automatically by the
version control tool. Ultimately it is the receiver of a file who is
responsible for determining its licensing status, and per-source notices are
merely a convenience. Therefore it is simpler and more accurate for
each source to have a generic notice of the license name and references to
more detailed information on copyright holders and full license terms.
Our `Copyright.txt` file now contains a list of Contributors whose names
appeared source-level copyright notices. It also references version control
history for more precise information. Therefore we no longer need to spell
out the list of Contributors in each source file notice.
Replace CMake per-source copyright/license notice headers with a short
description of the license and links to `Copyright.txt` and online information
available from "https://cmake.org/licensing". The online URL also handles
cases of modules being copied out of our source into other projects, so we
can drop our notices about replacing links with full license text.
Run the `Utilities/Scripts/filter-notices.bash` script to perform the majority
of the replacements mechanically. Manually fix up shebang lines and trailing
newlines in a few files. Manually update the notices in a few files that the
script does not handle.
If multiple ExternalData_Target_Add calls generate the same output file
then we need to avoid calling add_custom_command multiple times with
that output. This was already done within a single target by setting a
variable in the local function scope. This will not be visible in other
calls though so we need to use a directory property instead to prevent
adding a custom command multiple times for one output in a directory.
Normally it is not safe to have multiple custom commands that produce
the same output file across multiple independent targets, but since we
use atomic replacement of outputs the resulting races should not be a
problem. For the convenience of projects, tolerate this instead of
diagnosing it. In particular, we previously allowed up to two copies
of the custom command in one directory because CMake has a fallback
from MAIN_DEPENDENCY to an `<output>.rule` file.
While at it, add a note to the documentation that typically only one
external data target should be needed for a project.
Reported-by: David Manthey <david.manthey@kitware.com>
Add an ExternalData_NO_SYMLINKS to enable use of copies instead of
symlinks to populate the real data files behind a DATA{} reference.
This will be useful on UNIX-like systems when the underlying filesystem
does not actually support symbolic links.
Suggested-by: Matt McCormick <matt.mccormick@kitware.com>
Extend the ``DATA{Dir/,...}`` syntax with a new ``RECURSE:`` option
to enable recursive matching of associated files. This will allow
an entire directory tree of data to be referenced at once.
Extend the _ExternalData_arg_find_files signature with an option to
specify the kind of file(GLOB) operation to be performed. Set
CMP0009 to NEW so that GLOB_RECURSE does not follow symlinks.
Allow URL templates to contain a %(algo:<key>) placeholder that is
replaced by mapping the canonical hash algorithm name through a map
defined by the <key>.
Extend the Module.ExternalData test to cover the behavior.
Extend the RunCMake.ExternalData test to cover error cases.
Add support for a special URL template to map the fetch operation
to a project-specified .cmake script insead of using file(DOWNLOAD).
Extend the Module.ExternalData test to cover the behavior.
Extend the RunCMake.ExternalData test to cover error cases.
Move the basic DATA{} description to a section just before the
file series description. Move all sections on referencing files
into subsections of a common "Referencing Files" section.
Subsume example usage into the introduction since it gives a
high-level starting point to understand the rest of the docs.
Let clients do their own research on the current strength of each
hash algorithm to choose what is best for their needs.
Suggested-by: Nico Schlömer <nico.schloemer@gmail.com>
When the primary source tree path named by a DATA{} reference does not
exist, produce an AUTHOR_WARNING instead of a FATAL_ERROR. This is
useful when writing a new DATA{} reference to a test reference output
that has not been created yet. This way the developer can run the test,
manually verify the output, and then copy it into place to provide the
reference and eliminate the warning.
If the named source tree path is expected to be a file but exists as a
directory, we still need to produce a FATAL_ERROR.
The matches have already been calculated and can simply be taken from
CMAKE_MATCH_n variables. This avoids multiple compilations of the same or very
similar regular expressions.
Allow ExternalData_URL_TEMPLATES to be empty if a value for
ExternalData_OBJECT_STORES is provided. Assume in this use case that
the object stores will already contain all needed objects. Extend the
Module.ExternalData test to cover this case (all objects in stores).
Extend the RunCMake.ExternalData test to cover the non-failure message
case when stores are provided without URL templates.
Manually revise the .rst format of the documentation. Use inline
literal quotes appropriately in paragraph text. Move the :: literal
block openers to the end of the preceding paragraphs. Convert the
command signature documentation and examples to cmake code-block
directives.
The ExternalData_LINK_CONTENT option tells ExternalData to convert real
data files it finds into content links and to "stage" the original
content in a ".ExternalData_<algo>_<hash>" file. However, after a data
object has been staged it is possible that a user-provided pattern in
the "REGEX:" option will later match the staged object file. We must
not process staged object files even when a user pattern matches them.
Fix the implementation to not match a staged object file as a normal
data file for conversion. Extend the RunCMake.ExternalData test to
cover this case.
The CMake language implicitly flattens lists so a ";" in a list element
must be escaped with a backslash. List expansion removes backslashes
escaping semicolons to leave raw semicolons in the values. Teach
ExternalData_Add_Test and ExternalData_Expand_Arguments to re-escape
semicolons found in list elements so the resulting argument lists work
as if constructed directly by the set() command.
For example:
ExternalData_Add_Test(Data NAME test1 COMMAND ... "a\\;b")
ExternalData_Expand_Arguments(Data args2 "c\\;d")
add_test(NAME test2 COMMAND ... ${args2})
should be equivalent to
set(args1 "a\\;b")
add_test(NAME test1 COMMAND ... ${args1})
set(args2 "c\\;d")
add_test(NAME test2 COMMAND ... ${args2})
which is equivalent to
add_test(NAME test1 COMMAND ... "a;b")
add_test(NAME test2 COMMAND ... "c;d")
Note that it is not possible to make ExternalData_Add_Test act exactly
like add_test when quoted arguments contain semicolons because the CMake
language flattens lists when constructing function ARGN values. This
re-escape approach at least allows test arguments to have semicolons.
While at it, teach ExternalData APIs to not transform "DATA{...;...}"
arguments because the contained semicolons are non-sensical.
Suggested-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>
Each data file to be created in the build tree corresponds one-to-one with
a raw file or content link in the source tree. Use the MAIN_DEPENDENCY of
add_custom_command to attach the build rule to the source tree file. This
looks much nicer in the IDE project file browser and avoids ".rule" files.
Use private global variables _ExternalData_REGEX_(ALGO|EXT) to match the
possible hash algorithm names and extensions in regular expressions.
Use "file(<algo>)" instead of "cmake -E md5sum" to compute hashes
without a child process and to support more hash algorithms.
Use a trailing slash to reference a directory. Require that a list
of associated files be specified to select from within the directory.
One may simply use DATA{Dir/,REGEX:.*} to reference all files but
get a directory passed on the command line.
Refactor use of the ExternalData_SERIES_MATCH value to avoid assuming
that it has no ()-groups that interfere with group indexing.
Extend the Module.ExternalData test to cover this case.
Automatic series recognition can generate false positives too easily
when the default series configuration is flexible enough to handle
common cases. Avoid false positives by requiring an explicit syntax to
activate series recognition. Choose the syntax DATA{<name>,:} to be
short, simple, and look like a vertical ellipsis.
This allows us to improve the default series match configuration. Allow
series references to contain one of the numbered file names. Allow '-'
as a separator in addition to '.' and '_'. Document what the default
configuration matches. Also provide more options to configure series
<name> parsing.
