[CMake] custom commands, targets and parallel builds

[Kris Thielemans]

Hi all



I'm trying to add doxygen-generated comments to my Python module build via
SWIG using https://github.com/m7thon/doxy2swig. This means I need to run
doxygen first, then run doxy2swig, and only then run swig. However, I'm
getting reports that parallel builds are failing (sometimes).



My strategy is based on CMake advice and some blogs (e.g. at
https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-ta
rgets-and-files-and-custom-commands/ ) and relies on creating custom
commands to generate the files and custom targets to be able to specify
dependencies on those generated files. Somewhat simplified code is below
(original is at https://github.com/UCL/STIR/pull/280)



# command that runs doxygen

add_custom_command(

OUTPUT doxygen.stamp

DEPENDS ${doxyfile}

COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}

COMMAND cmake -E touch doxygen.stamp

.)

# corresponding target

add_custom_target( doc ALL DEPENDS doxygen.stamp)



# command that runs doxy2swig

add_custom_command(

OUTPUT STIR_DOXY2SWIG.i

DEPENDS doc

COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml STIR_DOXY2SWIG.i

)

# corresponding target

add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i)



# now add that target to the SIWG module

add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig)



Any suggestions on what I'm doing wrong?



Many thanks!

Kris



PS: I note that FindDoxygen's doxygen_add_docs merges the add_custom_command
and target into one add_custom_target, see
https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb1573c876a
1a/Modules/FindDoxygen.cmake#L1104

I thought that couldn't be done but I guess I was wrong.

PS: I also note that doxygen_add_docs doesn't create a doxygen.stamp file to
prevent re-runs though, is that not needed then?

[Alan W. Irwin]

Post by Kris Thielemans
Hi all
I'm trying to add doxygen-generated comments to my Python module build via
SWIG using https://github.com/m7thon/doxy2swig. This means I need to run
doxygen first, then run doxy2swig, and only then run swig. However, I'm
getting reports that parallel builds are failing (sometimes).
My strategy is based on CMake advice and some blogs (e.g. at
https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-ta
rgets-and-files-and-custom-commands/ ) and relies on creating custom
commands to generate the files and custom targets to be able to specify
dependencies on those generated files. Somewhat simplified code is below
(original is at https://github.com/UCL/STIR/pull/280)
# command that runs doxygen
add_custom_command(
OUTPUT doxygen.stamp
DEPENDS ${doxyfile}
COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}
COMMAND cmake -E touch doxygen.stamp
.)
# corresponding target
add_custom_target( doc ALL DEPENDS doxygen.stamp)
# command that runs doxy2swig
add_custom_command(
OUTPUT STIR_DOXY2SWIG.i
DEPENDS doc
COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml STIR_DOXY2SWIG.i
)
# corresponding target
add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i)
# now add that target to the SIWG module
add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig)
Any suggestions on what I'm doing wrong?
Many thanks!
Kris
PS: I note that FindDoxygen's doxygen_add_docs merges the add_custom_command
and target into one add_custom_target, see
https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb1573c876a
1a/Modules/FindDoxygen.cmake#L1104
I thought that couldn't be done but I guess I was wrong.
PS: I also note that doxygen_add_docs doesn't create a doxygen.stamp file to
prevent re-runs though, is that not needed then?

Hi Kris:

To answer your last set of questions first, you can execute all
desired commands with add_custom_target, but the issue is that COMMAND
always reruns. So your way of doing it above (with paired
add_custom_command/add_custom_target) is the recommended procedure
which should only re-run if the OUTPUT file is non-existent or older
than a dependent file.

Anyhow, I think what you have outlined above is generally correct.
For example, your "DEPENDS doc" argument to add_custom_command where
doc is a custom target argument is correct according to the
documentation at
<https://cmake.org/cmake/help/latest/command/add_custom_command.html>.
However, your simplified code above has several deviations from the
pattern I always use. I know my pattern generally works, and I think
at least your first two deviations from it are important.

1. I always like to specify the full pathname for all files (e.g.,
${CMAKE_CURRENT_BINARY_DIR}/doxygen.stamp and
${CMAKE_CURRENT_BINARY_DIR}/STIR_DOXY2SWIG.i. This is true not only
for OUTPUT files and the corresponding DEPENDS but also for input
(which typically start with ${CMAKE_CURRENT_SOURCE_DIR} and output
files for COMMANDs.

2. At the same time I make sure I run all commands in
${CMAKE_CURRENT_BINARY_DIR} by specifying that as the working
directory. So that all unmentioned files on the command line are
generated in the build tree to keep the source tree as clean as
possible.

3. I don't think it matters above but I always use the VERBATIM
attribute for COMMANDS.

I have found one of my most difficult CMake tasks is to avoid build
race conditions that can occur for parallel builds. So your real code
is most important in that regard rather than a simplified example of
your CMake logic. Perhaps others here would like to comment on the
most reliable way to find such race conditions. But what I do is
process "make -j<jobs> <target_name>" output results to remove the
progress percentage marks, sort those results, and look for anything
that is repeated that shouldn't be. But even such care is not
sufficient, and I recently got help on this list from someone who
found a race condition (several different commands using the same
temporary filename to store intermediate results) in the build system
for PLplot. That issue had been around for years!

Finally, although I have found parallel builds to be reliable on
Linux, Arjen Markus, one of the PLplot developers, has discovered that
parallel builds are currently unreliable for all Unix-like Windows
platforms, e.g., classical MinGW/MSYS, its modern replacement
MinGW-w64/MSYS2, and Cygwin. So if some of your users are reporting
parallel build troubles on any of those platforms it is likely an
issue with the make command on those platforms rather than your build
system.

Alan
__________________________
Alan W. Irwin

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[Kris Thielemans]

From: Alan W. Irwin, Sent: 25 November 2018 01:48

Many thanks Alan, answers and questions below

Post by Kris Thielemans
I'm trying to add doxygen-generated comments to my Python module build
via SWIG using https://github.com/m7thon/doxy2swig. This means I need
to run doxygen first, then run doxy2swig, and only then run swig.
However, I'm getting reports that parallel builds are failing

(sometimes).

Post by Kris Thielemans
My strategy is based on CMake advice and some blogs (e.g. at

https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-ta
rgets-and-files-and-custom-commands/ ) and relies on creating

Post by Kris Thielemans
custom commands to generate the files and custom targets to be able to
specify dependencies on those generated files. Somewhat simplified
code is below (original is at https://github.com/UCL/STIR/pull/280)
# command that runs doxygen
add_custom_command(
OUTPUT doxygen.stamp
DEPENDS ${doxyfile}
COMMAND ${DOXYGEN_EXECUTABLE} ${doxyfile}
COMMAND cmake -E touch doxygen.stamp
.)
# corresponding target
add_custom_target( doc ALL DEPENDS doxygen.stamp)
# command that runs doxy2swig
add_custom_command(
OUTPUT STIR_DOXY2SWIG.i
DEPENDS doc
COMMAND ${PYTHON_EXECUTABLE} doxy2swig.py -c index.xml
STIR_DOXY2SWIG.i
)
# corresponding target
add_custom_target(doxy2swig DEPENDS STIR_DOXY2SWIG.i)
# now add that target to the SIWG module
add_dependencies(${SWIG_MODULE_stir_REAL_NAME} doxy2swig)
Any suggestions on what I'm doing wrong?
Many thanks!
Kris
PS: I note that FindDoxygen's doxygen_add_docs merges the
add_custom_command and target into one add_custom_target, see

https://github.com/Kitware/CMake/blob/a44191abc489373d295ecaeb5c2eb153c876a1
a/Modules/FindDoxygen.cmake#L1104

Post by Kris Thielemans
I thought that couldn't be done but I guess I was wrong.
PS: I also note that doxygen_add_docs doesn't create a doxygen.stamp
file to prevent re-runs though, is that not needed then?

To answer your last set of questions first, you can execute all desired
commands with add_custom_target, but the issue is that COMMAND always
reruns. So your way of doing it above (with paired
add_custom_command/add_custom_target) is the recommended procedure
which should only re-run if the OUTPUT file is non-existent or older than

a

dependent file.

Ok. That makes sense to me. thanks!

Anyhow, I think what you have outlined above is generally correct.
For example, your "DEPENDS doc" argument to add_custom_command where
doc is a custom target argument is correct according to the documentation

at

<https://cmake.org/cmake/help/latest/command/add_custom_command.html

Post by Kris Thielemans
.

However, your simplified code above has several deviations from the

pattern I

always use. I know my pattern generally works, and I think at least your

first

two deviations from it are important.
1. I always like to specify the full pathname for all files (e.g.,
${CMAKE_CURRENT_BINARY_DIR}/doxygen.stamp and
${CMAKE_CURRENT_BINARY_DIR}/STIR_DOXY2SWIG.i. This is true not only for
OUTPUT files and the corresponding DEPENDS but also for input (which
typically start with ${CMAKE_CURRENT_SOURCE_DIR} and output files for
COMMANDs.
2. At the same time I make sure I run all commands in
${CMAKE_CURRENT_BINARY_DIR} by specifying that as the working directory.
So that all unmentioned files on the command line are generated in the

build

tree to keep the source tree as clean as possible.
3. I don't think it matters above but I always use the VERBATIM attribute

for

COMMANDS.

Thanks for these suggestions. RE 2 and 3, yes, I did the same. For the sake
of brevity I cut some of the relevant bits. Sorry! (full code at
https://github.com/UCL/STIR/pull/280)

I'm indeed not using the full pathnames. However, given the
WORKING_DIRECTORY option, all files are "local" as far as CMake concerns, it
seems to me this is not relevant, although I should give it try.

I have found one of my most difficult CMake tasks is to avoid build race
conditions that can occur for parallel builds. So your real code is most
important in that regard rather than a simplified example of your CMake

logic.

Perhaps others here would like to comment on the most reliable way to find
such race conditions. But what I do is process "make -j<jobs>

<target_name>"

output results to remove the progress percentage marks, sort those

results,

and look for anything that is repeated that shouldn't be.

Sounds painful :-; The person who reported the problem can currently not
provide me with a log file. He's on OSX but I don't know yet what CMake/make
versions he's using.

So instead I did some more digging in the generated makefiles. From these,
it seems all is ok (see end of email). I'm now puzzled of course.

Finally, although I have found parallel builds to be reliable on Linux,

Arjen

Markus, one of the PLplot developers, has discovered that parallel builds

are

currently unreliable for all Unix-like Windows platforms, e.g., classical
MinGW/MSYS, its modern replacement MinGW-w64/MSYS2, and Cygwin. So if
some of your users are reporting parallel build troubles on any of those
platforms it is likely an issue with the make command on those platforms
rather than your build system.

That is pretty amazing. Who knows, maybe "make" on OSX should be added to
the list of problematic "make" implementations as well.

If I can get dig out anything else on reproducing the problem or debugging,
I'll let you know. Thanks again for your help!

Kris

PS: some checks on build files

== Visual Studio 2015 generated files (with CMake 3.11.4):

My swig module _stir has a reference "doxy2swig", which has a reference
"doc", so that seems fine. (I'm assuming that a reference means it'll check
dependencies).

== Unix Makefiles generated by CMake 3.7 and 3.13.0:
I first got confused by the fact that cross-directory dependencies in the
makefiles in each sub-dir are non-existent, but I realised that "make" is
called for each of these from the "master" makefile. Checking
CMakeFiles/Makefile2 in the root of my build, everything seems fine as this
contains dependencies:

src/swig/CMakeFiles/doxy2swig.dir/all: src/CMakeFiles/doc.dir/all
src/swig/CMakeFiles/_stir.dir/all: src/swig/CMakeFiles/doxy2swig.dir/all

(I've now learned that using "make" from in one of the subdirs is a
dangerous thing to do as it doesn't check dependencies properly, but that
makes some sense.)
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake