[CMake] Tips on reducing configure time

[Poughon Victor]

Hi all

I am working on a large C++ project using CMake (https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a lot of time. This can be annoying for developers working on features where the configure step needs to be re-run, or for CI in general.

We have looked into it a bit and it seems like one issue is that the configure/generate steps do a lot of small files I/O. Specifically, using:

inotofy-wait -mr <build_dir>

shows that "cmake ." causes about 150K file access/modify, etc. There are some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp' that appear thousands of times in most of the log file.

I have an SSD which somewhat helps, but for colleagues still on mechanical drives this is especially painful. Is there a way to disable writing all those files to the disk?

Any other tips on optimizing the performance of 'cmake .'?

Thanks,

Victor Poughon
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[Eric Noulard]

Hi Victor,

Post by Poughon Victor
Hi all
I am working on a large C++ project using CMake (
https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a
lot of time. This can be annoying for developers working on features where
the configure step needs to be re-run, or for CI in general.

Could you give us a clue on "takes quite a lot of time"?
Are we speaking of tens of seconds, minutes etc... ?

Post by Poughon Victor
We have looked into it a bit and it seems like one issue is that the
inotofy-wait -mr <build_dir>
shows that "cmake ." causes about 150K file access/modify, etc. There are
some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp'
that appear thousands of times in most of the log file.
I have an SSD which somewhat helps, but for colleagues still on mechanical
drives this is especially painful. Is there a way to disable writing all
those files to the disk?
Any other tips on optimizing the performance of 'cmake .'?

Did you try to test drive a switch to ninja generator ?
Ninja generator generates only two files (build.ninja and rules.ninja) you
won't have the "mny small files issue".
You can have long configuration times as well but not coming from file
access.

Post by Poughon Victor
Thanks,
Victor Poughon
--
Powered by www.kitware.com
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
https://cmake.org/mailman/listinfo/cmake

[Poughon Victor]

Thanks for the replies

You can see configure times on our dashboard (switch to advanced view): https://dash.orfeo-toolbox.org/index.php?project=OTB
It ranges from 1-3min. There's even one windows machine that's at 4m30s.

Thanks for the tip about Ninja generator, it's indeed much faster. But switching the entire dev team + CI servers from make to ninja is not really an option for us right now because of a bug in ninja that affects our SuperBuild configuration (see here if you're interested in the details: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625)


Victor Poughon

De : Eric Noulard <***@gmail.com>
Envoyé : mercredi 14 novembre 2018 11:12
À : Poughon Victor <***@cnes.fr>
Cc : CMake Mailinglist <***@cmake.org>
Objet : Re: [CMake] Tips on reducing configure time

Hi Victor,
Le mer. 14 nov. 2018 à 10:20, Poughon Victor <***@cnes.fr<mailto:***@cnes.fr>> a écrit :
Hi all

I am working on a large C++ project using CMake (https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a lot of time. This can be annoying for developers working on features where the configure step needs to be re-run, or for CI in general.

Could you give us a clue on "takes quite a lot of time"?
Are we speaking of tens of seconds, minutes etc... ?


We have looked into it a bit and it seems like one issue is that the configure/generate steps do a lot of small files I/O. Specifically, using:

inotofy-wait -mr <build_dir>

shows that "cmake ." causes about 150K file access/modify, etc. There are some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp' that appear thousands of times in most of the log file.

I have an SSD which somewhat helps, but for colleagues still on mechanical drives this is especially painful. Is there a way to disable writing all those files to the disk?

Any other tips on optimizing the performance of 'cmake .'?

Did you try to test drive a switch to ninja generator ?
Ninja generator generates only two files (build.ninja and rules.ninja) you won't have the "mny small files issue".
You can have long configuration times as well but not coming from file access.




Thanks,

Victor Poughon
--
Powered by www.kitware.com<http://www.kitware.com>

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[Eric Noulard]

Post by Poughon Victor
Thanks for the replies
https://dash.orfeo-toolbox.org/index.php?project=OTB
It ranges from 1-3min. There's even one windows machine that's at 4m30s.

Somes windows system are known to be slow on many files accesses.
However on this particular machine (
https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342008 I guess) you
already seem
to be using NInja + MSVC. so many small files access to ?makefiles? should
not be the culprit.

That said whatever the system your build and test time seems to dominate
the overall time (in the CI).

I have had great speed-up experience using CMake+ninja+ccache for our
build in CI. We did go from 1 hour down to less than 10min.
We only have various linux build (no Windows) it is really a big time
savior combo.

We even share the compil' cache between the CI and the developers. So
everybody is feeding compile cache and takes benefits from others including
CI.

You seem to be using that as well here:
https://dash.orfeo-toolbox.org/buildSummary.php?buildid=341993
and you get less than 5min build time which corroborate my suggestion.

The same machine build with clang:
https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342033 is lasting
30min+
and is not using ccache.

Post by Poughon Victor
Thanks for the tip about Ninja generator, it's indeed much faster. But
switching the entire dev team + CI servers from make to ninja is not really
an option for us right now because of a bug in ninja that affects our
https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625)

I see.
Not using ninja in the Superbuild case does not mean you cannot use it in
other cases, right?
May be the one-size fits all is a little too much to ask at this point? My
opinion though.
Did you report the issue upstream (ninja or cmake)?

I was impressed in the past by the amount of tests orfeo is doing, I'm
still are !!
Are you running those tests in parallel (with ctest --parallel) ? It can be
a time savior as well is the running machine has a fair amount of cores and
memory.

[Poughon Victor]

Yes you are correct on all your observations. We already use ninja and ccache wherever we can. In fact we have an issue about the whole end-to-end build performance where we track all effort on this throughout the project (if you're interested: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1649)

There are indeed quick and easy wins to be had with those tools (and we are working on it). But my original question is not about that. It's about speeding up the configure step with makefile generator. I still don't think it's normal that it does hundreds of thousands of I/O on files that are a few bytes or even empty. However it's possible that it's because we do something incorrect in our CMakeLists.txt and not CMake's fault.

It's true that a few minutes of configure is not much when doing a full build, but consider incremental builds where all you do is change one cxx file (and cmake is triggered because you changed git branch or something). Then the generate step is a significant part of the time you wait.

Victor Poughon

De : Eric Noulard <***@gmail.com>
Envoyé : mercredi 14 novembre 2018 12:34
À : Poughon Victor <***@cnes.fr>
Cc : CMake Mailinglist <***@cmake.org>
Objet : Re: [CMake] Tips on reducing configure time


Le mer. 14 nov. 2018 à 12:06, Poughon Victor <***@cnes.fr<mailto:***@cnes.fr>> a écrit :
Thanks for the replies

You can see configure times on our dashboard (switch to advanced view): https://dash.orfeo-toolbox.org/index.php?project=OTB
It ranges from 1-3min. There's even one windows machine that's at 4m30s.

Somes windows system are known to be slow on many files accesses.
However on this particular machine (https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342008 I guess) you already seem
to be using NInja + MSVC. so many small files access to ?makefiles? should not be the culprit.

That said whatever the system your build and test time seems to dominate the overall time (in the CI).

I have had great speed-up experience using CMake+ninja+ccache for our build in CI. We did go from 1 hour down to less than 10min.
We only have various linux build (no Windows) it is really a big time savior combo.

We even share the compil' cache between the CI and the developers. So everybody is feeding compile cache and takes benefits from others including CI.

You seem to be using that as well here:
https://dash.orfeo-toolbox.org/buildSummary.php?buildid=341993
and you get less than 5min build time which corroborate my suggestion.

The same machine build with clang: https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342033 is lasting 30min+
and is not using ccache.

Thanks for the tip about Ninja generator, it's indeed much faster. But switching the entire dev team + CI servers from make to ninja is not really an option for us right now because of a bug in ninja that affects our SuperBuild configuration (see here if you're interested in the details: https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625)

I see.
Not using ninja in the Superbuild case does not mean you cannot use it in other cases, right?
May be the one-size fits all is a little too much to ask at this point? My opinion though.
Did you report the issue upstream (ninja or cmake)?

I was impressed in the past by the amount of tests orfeo is doing, I'm still are !!
Are you running those tests in parallel (with ctest --parallel) ? It can be a time savior as well is the running machine has a fair amount of cores and memory.

[Eric Noulard]

Post by Poughon Victor
Yes you are correct on all your observations. We already use ninja and
ccache wherever we can. In fact we have an issue about the whole end-to-end
build performance where we track all effort on this throughout the project
https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1649)

End-to-end build performance is a subject of interest to me, I'll have a
look.

Post by Poughon Victor
There are indeed quick and easy wins to be had with those tools (and we
are working on it). But my original question is not about that. It's about
speeding up the configure step with makefile generator. I still don't think
it's normal that it does hundreds of thousands of I/O on files that are a
few bytes or even empty. However it's possible that it's because we do
something incorrect in our CMakeLists.txt and not CMake's fault.

I don't really know makefile-generator internals so I cannot tell.
For sur the current OTB build dir constructed with Makefile generator is
spitting out around 180 Makefile.
May be you can profile the cmake execution by building a debug version of
CMake and collect more precise insight on where the bottlenecks may reside.

You may track try_compile which could obviously be slow and may be
configure_file as well.
Otherwise I don't know, I guess you'll have to profile the cmake run.

It's true that a few minutes of configure is not much when doing a full

Post by Poughon Victor
build, but consider incremental builds where all you do is change one cxx
file (and cmake is triggered because you changed git branch or something).
Then the generate step is a significant part of the time you wait.

Yes that right.
Note however that changing git branch should nor per-se trigger a cmake run.
Only changing a CMakeLists.txt or configured file or explicitely specified
by
CMAKE_CONFIGURE_DEPENDS.

Otherwise changing branch with git that only modifies source shouldn't
trigger cmake rebuild.
I do that all the time in a classical git branch dev model.

[Cristian Adam]

Post by Poughon Victor
Hi all
I am working on a large C++ project using CMake (
https://www.orfeo-toolbox.org/) and our 'configure' step takes quite a
lot of time. This can be annoying for developers working on features where
the configure step needs to be re-run, or for CI in general.
We have looked into it a bit and it seems like one issue is that the
inotofy-wait -mr <build_dir>
shows that "cmake ." causes about 150K file access/modify, etc. There are
some especially guilty looking files like 'Makefile2.tmp' or 'progress.tmp'
that appear thousands of times in most of the log file.
I have an SSD which somewhat helps, but for colleagues still on mechanical
drives this is especially painful. Is there a way to disable writing all
those files to the disk?
Any other tips on optimizing the performance of 'cmake .'?
Thanks,
Victor Poughon

Hi,

if a subsequent CMake run is faster than the initial one, you can have a
look at
https://github.com/cristianadam/cmake-checks-cache

If you cache the results of the configure checks you can have the initial
run as fast as subsequent ones!

Cheers,
Cristian.