[CMake] Converting a large C++-Project to CMake

Hi,

Message: 1
Date: Sat, 30 Oct 2010 13:54:23 +0200
Subject: [CMake] Converting a large C++-Project to CMake
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Hello,
I'm working on a ~1.5Mio LOC C++ project and our buildsystem is a

"me too" ;)

hodgepodge of handcrafted Makefiles, shell scripts and qmake projects.
I tried to convert a subset to CMake and it looks very promising so far.
1) User 'nightly' builds versions of the project every night on several
development servers.
2) A developer is coming to the office in the morning, and
copies/hardlinks the nightly build to his home dir and probably patches
uncommited changes from yesterday into the newly copied build.
Our build is taking ages (almost a three hours on the fastest of our
servers) and it would be really painful if everybody needed to rebuild
everything for himself in the morning.

3 hours sounds quite excessively long.
On a bog-standard workstation (i.e., definitely _not_ "on the fastest of our
servers"), a (single-core) build here takes around 2 hours,
however a (partial) Linux CMake build is finished within 25(!) minutes
(make -j3) on a Core 2 Duo.
And this with some projects thrown in to the mix (of around 40 projects total)
that are _heavily_ templated i.e. rather more painful to build.

According to the FAQ, CMake does not support copying build trees around.

Yep, and I'd think that that's a pretty hard condition.
One really shouldn't try to bend things to work with specific build tree
data on multiple machines I'd think. That's just not the way that CMake
is supposed to be used.

I'd much rather suggest cutting down on rogue include file overhead
(check gcc -E to analyze amount to process on each file, and to see
which file gets included most often and/or in the most painful way).
This is a suitable way to decrease build pains in the case of
possibly more "chaotic" build trees (I once managed to get it down
from almost 3 hours to around 1:25 or so).

If however this doesn't cut it for you (i.e., tree size is a wee bit too large
for this to be feasible within office times),
then you should probably find a way to split sub projects into proper,
cleanly interfaced, external projects to be used for one large build project
as needed.
I.e., keep most parts as external dependencies (to be updated every
couple weeks/months when the "other" teams have a new achievement to
offer). And mainly work on the one CMake project (i.e., "your" project)
which then makes use of the other external projects.
Related CMake mechanisms here probably are ExternalProject_Add()
or Imported Targets.

Or, another way to achieve reduced build times might be things like
distcc, icecc, ccache.

On my side, I'm currently working on a flat CMake project space (and as said
above this is working sufficiently well),
however I might eventually change this to split off some suitable parts.

HTH, coming from someone with possibly rather similar conditions.

Andreas Mohr

Benjamin King

2010-10-31 12:42:52 UTC

Hi Andreas!

3 hours sounds quite excessively long.

Yepp, that's too long, really. Thanks for your suggestions on cutting
that down. We have tried a lot of them already, but there simply is no
short-term solution to make the separation of the modules clearer.
Always working on adding features, never time to refactor, meta template
programming all over the place. Same old, same old...

Our windows release build and packaging takes almost 8 hours (on a
virtual machine), which makes a quick fix of a bug reported by our test
team effectively impossible.

Using parallel compilation on multiple cores and/or machines would make
things easier for us. With linux, we are using distcc and make -jX. The
trouble with that is, that the Makefiles generated by qmake don't deal
well with generated code which we also use a lot. You have to restart
make a few times to keep things going, which is not practical for an
automated build and package script.

Windows and nmake don't offer any feasible way to parallelize, so we are
stuck there.

Maybe cmakes VS-Project output could improve the latter and cmake custom
command the former.

Post by Benjamin King
According to the FAQ, CMake does not support copying build trees around.

Alexander Neundorf

2010-10-31 12:58:06 UTC

Post by Benjamin King
Hi Andreas!

3 hours sounds quite excessively long.

Yepp, that's too long, really. Thanks for your suggestions on cutting
that down. We have tried a lot of them already, but there simply is no
short-term solution to make the separation of the modules clearer.
Always working on adding features, never time to refactor, meta template
programming all over the place. Same old, same old...
Our windows release build and packaging takes almost 8 hours (on a
virtual machine), which makes a quick fix of a bug reported by our test
team effectively impossible.
Using parallel compilation on multiple cores and/or machines would make
things easier for us. With linux, we are using distcc and make -jX. The
trouble with that is, that the Makefiles generated by qmake don't deal
well with generated code which we also use a lot. You have to restart
make a few times to keep things going, which is not practical for an
automated build and package script.

This is handled properly by cmake-generated makefiles and project files.
You use add_custom_command() to generate files, and if you really list all
files the custom command generates after the OUTPUT keyword, parallel builds
will work properly.
We do that all the time in KDE.

Post by Benjamin King
Windows and nmake don't offer any feasible way to parallelize, so we are
stuck there.

You could use GNU make with the MS compiler, or you can use jom, which is a
nmake compatible make for Windows, which supports parallel builds
(http://labs.qt.nokia.com/2009/03/27/speeding-up-visual-c-qt-builds/)

Alex

Benjamin King

2010-11-01 04:15:28 UTC

Hello Alex!

[Dependencies to generated files are] handled properly by cmake-generated
makefiles and project files.
You use add_custom_command() to generate files, and if you really list all
files the custom command generates after the OUTPUT keyword, parallel builds
will work properly.

Great, that's good to know.

Thanks again. I'll try both!

Cheers,
Benjamin

Pedro d'Aquino

2010-10-31 17:19:49 UTC

Post by Benjamin King
Hi Andreas!

3 hours sounds quite excessively long.

Yepp, that's too long, really. Thanks for your suggestions on cutting that
down. We have tried a lot of them already, but there simply is no
short-term solution to make the separation of the modules clearer. Always
working on adding features, never time to refactor, meta template
programming all over the place. Same old, same old...
Our windows release build and packaging takes almost 8 hours (on a virtual
machine), which makes a quick fix of a bug reported by our test team
effectively impossible.
Using parallel compilation on multiple cores and/or machines would make
things easier for us. With linux, we are using distcc and make -jX. The
trouble with that is, that the Makefiles generated by qmake don't deal well
with generated code which we also use a lot. You have to restart make a few
times to keep things going, which is not practical for an automated build
and package script.
Windows and nmake don't offer any feasible way to parallelize, so we are
stuck there.

If you haven't already, you might want to take a lot at JOM, which is
essentially parallel nmake: http://qt.gitorious.org/qt-labs/jom. JOM
is supported by CMake.

Post by Benjamin King
Maybe cmakes VS-Project output could improve the latter and cmake custom
command the former.

Post by Benjamin King
According to the FAQ, CMake does not support copying build trees around.

Understood, but we do not copy the tree to different machines, just from
/home/nightly to /home/developer1, /home/developer2 etc on the same machine.
Anyway, thanks for your impression on the situation!
Cheers,
Benjamin
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

Benjamin King

2010-11-01 04:18:08 UTC

Hi Pedro!

Post by Pedro d'Aquino

Post by Benjamin King
Windows and nmake don't offer any feasible way to parallelize, so we are
stuck there.

If you haven't already, you might want to take a lot at JOM, which is
essentially parallel nmake: http://qt.gitorious.org/qt-labs/jom. JOM
is supported by CMake.

Thank you Pedro. One of our developers already tried to plug jom into
our current build system, but some trouble with forward slashes in path
names occurred. I'll give it a shot with CMake!

Cheers,
Benjamin

Verweij, Arjen

2010-11-01 11:35:39 UTC

Benjamin,

Post by Pedro d'Aquino
If you haven't already, you might want to take a lot at JOM, which is
essentially parallel nmake: http://qt.gitorious.org/qt-labs/jom. JOM
is supported by CMake.

Thank you Pedro. One of our developers already tried to plug jom into
our current build system, but some trouble with forward slashes in path
names occurred. I'll give it a shot with CMake!

Make sure you take 0.9.4 or better, I haven't had much success with older versions, although I didn't try every version. This was the newest at the time and it works for me (tm).

Arjen

Michael Jackson

2010-11-01 12:56:49 UTC

There has since been a 1.0 release which seems to work very well.

ftp://ftp.qt.nokia.com/jom/jom.zip
___________________________________________________________
Mike Jackson www.bluequartz.net

Post by Verweij, Arjen
Benjamin,

Post by Pedro d'Aquino
If you haven't already, you might want to take a lot at JOM, which is
essentially parallel nmake: http://qt.gitorious.org/qt-labs/jom. JOM
is supported by CMake.

Thank you Pedro. One of our developers already tried to plug jom into
our current build system, but some trouble with forward slashes in path
names occurred. I'll give it a shot with CMake!

Make sure you take 0.9.4 or better, I haven't had much success with
older versions, although I didn't try every version. This was the
newest at the time and it works for me (tm).
Arjen

Alan W. Irwin

2010-10-30 22:21:52 UTC

Our build is taking ages (almost a three hours on the fastest of our servers)
and it would be really painful if everybody needed to rebuild everything for
himself in the morning.

Aren't you distributing your source code with some tool that preserves
the creation dates of source files? I think all of svn, rsync, tar, or
even cp -a do this.

If the above condition is met (no gratuitous changing of file dates
when copying source trees), then cmake configured builds are normally
very good about paying attention to target and file dependencies so
recompilations only occur for source code that has a later date than
the corresponding object code, for example.

To demonstrate this for yourself, run your configured build system
twice (e.g., run cmake-configured "make VERBOSE=1" twice) without
changing any source code file. The second run should take essentially
no time at all. Then touch one of your source code files and run make
a third time. You should see only the object file corresponding to
that source code being rebuilt.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Benjamin King

2010-10-31 12:48:14 UTC

Hi Alan!

Post by Alan W. Irwin

Aren't you distributing your source code with some tool that preserves
the creation dates of source files? I think all of svn, rsync, tar, or
even cp -a do this.

Yes, we do use cvs and copy build trees with rsync -a/cp -a.

The problem is that because our builds take that long, nobody does a
global cvs update very often, but rather only on the parts he is working on.

It's user nightly that does an automated fresh checkout every night and
sees if everything is fitting together and passes the tests. That makes
it more attractive to copy the nightly build to continue your own work,
because there is some guarantee that it builds, passes the tests and
incorporated the stuff from the other teams.

Cheers,
Benjamin

Alan W. Irwin

2010-10-31 16:48:17 UTC

Post by Benjamin King
Hi Alan!

Post by Alan W. Irwin

Aren't you distributing your source code with some tool that preserves
the creation dates of source files? I think all of svn, rsync, tar, or
even cp -a do this.

Yes, we do use cvs and copy build trees with rsync -a/cp -a.
The problem is that because our builds take that long, nobody does a global
cvs update very often, but rather only on the parts he is working on.
It's user nightly that does an automated fresh checkout every night and sees
if everything is fitting together and passes the tests. That makes it more
attractive to copy the nightly build to continue your own work, because there
is some guarantee that it builds, passes the tests and incorporated the stuff
from the other teams.

Hi Benjamin:

This sounds like you are describing dependency problems with your
current build system. That should be a prime motivation to implement a
complete CMake-based build system which allows you to specify all
dependencies between your software components with no gratuitious
rebuilding. With a properly implemented complete CMake-based build
system, your users would be controlling what gets rebuilt in a
flexible way depending on what they choose to cvs update. That is the
ideal result and therefore a great selling point for CMake-based build
systems.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Karl Wallner

2010-11-01 01:13:13 UTC

Post by Benjamin King
Hello,
I'm working on a ~1.5Mio LOC C++ project and our buildsystem is a hodgepodge of handcrafted Makefiles, shell scripts and
qmake projects.
I tried to convert a subset to CMake and it looks very promising so far.

I'm currently (as a consultant) converting ~2.5 Mio LOC C++/C from vcproj/Makefiles/qmake to cmake.
Working with big projects, different VCS, lots of developers is very different to using cmake in
small projects (few developers, single repository).
Here are some things I've learned so far:
- Split repositories (You can not put ~2.5 Mio LOC in one repository)
- Automate everything (You need: continuous build and delivery, use cdash or hudson)
- Complete build process (You need: checkout, build, packaging and install)
- Support developers (You need: developers want to work as they used to with their IDE)
- Fill gaps (You need: Find*-Modules, CMake-Macros)

Post by Benjamin King
1) User 'nightly' builds versions of the project every night on several development servers.
2) A developer is coming to the office in the morning, and copies/hardlinks the nightly build to his home dir and
probably patches uncommited changes from yesterday into the newly copied build.

Nightly builds are very useful to build releases. Developers should have their own source tree and update it on
their own (daily or more often updates from repository and incremental rebuilds). Systems for continuous
integration might also be useful.

Post by Benjamin King
Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if
everybody needed to rebuild everything for himself in the morning.
According to the FAQ, CMake does not support copying build trees around. On the development servers, we have the special
situation that all external tools needed for the build are either in /usr/bin, /usr/local/bin or
/opt/local/ourstuff/build/bin. Would it be feasible to copy a build tree in such a setting? Or could we copy everything,
throw the CmakeCache away, reconfigure and reuse the object files that already were build?

There are others ways to build faster, like ccache (http://ccache.samba.org/) or distcc (http://distcc.samba.org/).

Post by Benjamin King
Do you have any advice on this or other ideas? This is really a make-or-break feature for us and I'd like to present
some solution with CMake for it.

If you are going to change tools (introduce cmake), it is also time to review your development process (If in the
business world e.g. a SAP-Systems is introduced the complete work flow and business processes will get adapted).

Regards,

Karl

Benjamin King

2010-11-01 04:53:06 UTC

Hi Karl!

Post by Karl Wallner
I'm currently (as a consultant) converting ~2.5 Mio LOC C++/C from
vcproj/Makefiles/qmake to cmake.

Can you say something of the motivations for your client to do this?

I suspect that you are expected to keep most of their development
workflow and processes in order. Which parts are you allowed to redesign
and which have to stay?

For us, there are also strong non-technical factors that will affect the
decision to improve the build system, such as:
1) Is there some migration path from the old system?
2) Can we get around taking two weeks off where everybody needs to learn
a new tool?
3) Can we keep the knowledge that went into the scripts, makefiles and
qmake-projects of the current system?
4) Does it offer enough advantages to take the risk to change a matured
and operative build system (albeit the latter is running at its limits)?

There are more points, but I'll need to address them all in order to
convince anyone to even try cmake.

Right now, my plan is to convert a substantial part of the project to
cmake. It needs to work with all our code generators. It will include a
subproject that has a substantial amount of unit, integration and
regression test that all need to be passed afterwards. And the migration
path could be to just check in the CMakeLists.txt into the repository
and be grateful that cmake builds out of source.

Part of my non-technical review will be the amount of time and pain it
took me to do the conversion. The book (Mastering CMake) and the
superfriendly and professional mailing lists will surely be big selling
points there :-)

I'd be really interesting to know what non-technical issues you are
facing with your current project!

Post by Karl Wallner
- Automate everything (You need: continuous build and delivery, use cdash or hudson)
- Complete build process (You need: checkout, build, packaging and install)

We do that with bash scripts and virtual machines.
Which is why running cmake must be fully scriptable.
I understand that it is if you are passing it the required configuration
parameters with cmake -DTHIS_AND_THAT.

Post by Karl Wallner
There are others ways to build faster, like ccache
(http://ccache.samba.org/) or distcc (http://distcc.samba.org/).

We are using both for regular developer builds in linux. With release
builds, we are a little more reluctant and expect everything to build in
bearable time from scratch on a single machine. This works, just short
of the bearable part.

Post by Karl Wallner
If you are going to change tools (introduce cmake), it is also time to
review your development process (If in the business world e.g. a
SAP-Systems is introduced the complete work flow and business processes
will get adapted).

I'll collect all the suggestions from this list and summarize them in a
report.

Thanks a lot for being so helpful, that's much appreciated!

Cheers,
Benjamin

Michael Wild

2010-11-01 08:07:06 UTC

Post by Benjamin King
Hi Karl!

Post by Karl Wallner
I'm currently (as a consultant) converting ~2.5 Mio LOC C++/C from
vcproj/Makefiles/qmake to cmake.

Well, IMHO this really depends on your current build system. If it is flexible enough you should be able to port the parts of your project one by one.

Post by Benjamin King
2) Can we get around taking two weeks off where everybody needs to learn a new tool?

I don't think that it takes that long for everybody. There will be a few key-persons who will have to dig in deep, but the majority of the developers will be able to add and remove sources from the CMakeLists.txt files without even knowing the language. The same goes for include directories and link-libraries. Running CMake is another matter, but that should take a really short time to get a grasp on, and you could also provide a convenient wrapper script.

Post by Benjamin King
3) Can we keep the knowledge that went into the scripts, makefiles and qmake-projects of the current system?

I don't quite understand the point of this question. Also, it seems to be motivated by the "sunk costs fallacy". Apparently the current build system doesn't fulfill its requirements anymore and is fragile and from your statements I get the feeling that you don't think the issues can be easily rectified in the current framework. Cut your losses and move on. Whether that is CMake or something else that fits the bill doesn't matter so much.

Post by Benjamin King
4) Does it offer enough advantages to take the risk to change a matured and operative build system (albeit the latter is running at its limits)?

CMake itself is also quite mature and used for a large variety of projects, some of them huge (take KDE). Of course, the build-system you create with CMake will take some time to settle and grow into its full potential, but that is true for every other build system.

Post by Benjamin King
There are more points, but I'll need to address them all in order to convince anyone to even try cmake.
Right now, my plan is to convert a substantial part of the project to cmake. It needs to work with all our code generators.

What do you mean by a "code generator"?

Post by Benjamin King
It will include a subproject that has a substantial amount of unit, integration and regression test that all need to be passed afterwards. And the migration path could be to just check in the CMakeLists.txt into the repository and be grateful that cmake builds out of source.

Keeping the two systems side-by-side seems to be a reasonable approach. If you used a decent SCM (sorry, I'm extremely biased against CVS/SVN), you could easily do that in a feature-branch, not disturbing current development. But with CVS I think it's best to have them side-by-side.

Post by Benjamin King
Part of my non-technical review will be the amount of time and pain it took me to do the conversion. The book (Mastering CMake) and the superfriendly and professional mailing lists will surely be big selling points there :-)

Just remember that the book is a bit out-dated. Although everything in there should still work with current CMake (AFAIK), there are some notable new features which could make your life easier (such as the ExternalProject module).

Post by Benjamin King
I'd be really interesting to know what non-technical issues you are facing with your current project!

Post by Karl Wallner
- Automate everything (You need: continuous build and delivery, use cdash or hudson)
- Complete build process (You need: checkout, build, packaging and install)

If you're willing to use CTest+CDash and CPack, you've got it all. Also, for the scripting them -C option might be of interest for you. Instead of passing tons of -D options, you write a script containing "set(... CACHE ...)" instructions to populate the cache.

Post by Karl Wallner
There are others ways to build faster, like ccache
(http://ccache.samba.org/) or distcc (http://distcc.samba.org/).

We are using both for regular developer builds in linux. With release builds, we are a little more reluctant and expect everything to build in bearable time from scratch on a single machine. This works, just short of the bearable part.

CMake can speed up builds from scratch in my experience, but there's only so much you can do when building sequentially. What really helps is doing parallel builds with GNU Make (the -j option).