[CMake] visual studio parallel build

[j s]

By default visual studio uses 2 threads on my machine, a dual core AMD.

I notice when I have RUNTESTS (I can't remember) enabled, it may attempt to
run the tests before my project is even built.

If certain sub directories depend on a target in another directory already
being built, is it safe to use the multi-threaded option in visual studio?

Regards,

Juan

[John Drescher]

Post by j s
By default visual studio uses 2 threads on my machine, a dual core AMD.
I notice when I have RUNTESTS (I can't remember) enabled, it may attempt to
run the tests before my project is even built.
If certain sub directories depend on a target in another directory already
being built, is it safe to use the multi-threaded option in visual studio?

I have been using parallel builds with visual studio 2005 for over 1
year. It sometimes creates situations like this but since it saves me
10s of minutes per day I deal with the issues. Generally building the
same project a second time fixes any problems and is usually quick
since most stuff builds correctly on the first pas.

John

[John Drescher]

I guess I won't be using the parallel option in Visual Studio, then.

I have found it rare to have issues like this even on quad core
machines but it does happen. I build on 5 or so different machines
from 2 to 4 cores.

A
non-deterministic build order is not worth the risk.

I understand.

[Dominik Szczerba]

BTW do you know if it is possible to do a parallel build with nmake?
(like -jN with GNU make)
-- Dominik

Post by John Drescher

I guess I won't be using the parallel option in Visual Studio, then.

I have found it rare to have issues like this even on quad core
machines but it does happen. I build on 5 or so different machines
from 2 to 4 cores.

A
non-deterministic build order is not worth the risk.

I understand.
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

[j s]

I don't think so, but this article claims that by specifying multiple c++
files at the same time, the Visual C++ compiler will parallelize them:

http://stackoverflow.com/questions/601970/how-do-i-utilise-all-the-cores-for-nmake

Regards,

Juan

BTW do you know if it is possible to do a parallel build with nmake? (like
-jN with GNU make)
-- Dominik

Post by John Drescher

I guess I won't be using the parallel option in Visual Studio, then.
I have found it rare to have issues like this even on quad core

machines but it does happen. I build on 5 or so different machines
from 2 to 4 cores.
A

non-deterministic build order is not worth the risk.
I understand.

_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

--
d o m i n i k s z c z e r b a , p h d . . . . . . . . . . .
c o m p u t a t i o n a l l i f e s c i e n c e g r o u p
. . . . . . . i t ' i s r e s e a r c h f o u n d a t i o n
. . . . . . . . . . . . . . . . . . . . http://www.itis.ethz.ch

[Martin Apel]

Post by j s
I don't think so, but this article claims that by specifying multiple
http://stackoverflow.com/questions/601970/how-do-i-utilise-all-the-cores-for-nmake
<avk-blocked://stackoverflow.com/questions/601970/how-do-i-utilise-all-the-cores-for-nmake>
Regards,

As nmake cannot perform parallel builds on its own, it would be nice, if
CMake could use the possibility of the cl compiler itself to compile
multiple files in parallel.
Qt's qmake does this by creating a temporary file containing a single cl
invocation with multiple names of source files to be built. Wouldn't it
be possible for the NMake generator to
perform something similar?

Regards,

Martin

[David Cole]

No. There is no -j flag to nmake.

BTW do you know if it is possible to do a parallel build with nmake? (like
-jN with GNU make)
-- Dominik

Post by John Drescher

I guess I won't be using the parallel option in Visual Studio, then.
I have found it rare to have issues like this even on quad core

machines but it does happen. I build on 5 or so different machines
from 2 to 4 cores.
A

non-deterministic build order is not worth the risk.
I understand.

_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

--
d o m i n i k s z c z e r b a , p h d . . . . . . . . . . .
c o m p u t a t i o n a l l i f e s c i e n c e g r o u p
. . . . . . . i t ' i s r e s e a r c h f o u n d a t i o n
. . . . . . . . . . . . . . . . . . . . http://www.itis.ethz.ch
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

[Patrick Spendrin]

Post by Dominik Szczerba
BTW do you know if it is possible to do a parallel build with nmake?
(like -jN with GNU make)

It isn't.
A replacement tool has been developed though:
http://labs.trolltech.com/blogs/2009/03/27/speeding-up-visual-c-qt-builds/
current binaries can be found here:
http://winkde.org/pub/kde/ports/win32/repository/other/jom-20090729-bin.tar.bz2
reports about problems would be nice.

Post by Dominik Szczerba
-- Dominik

regards,
Patrick

Post by Dominik Szczerba

Post by John Drescher

I guess I won't be using the parallel option in Visual Studio, then.

I have found it rare to have issues like this even on quad core
machines but it does happen. I build on 5 or so different machines
from 2 to 4 cores.

A
non-deterministic build order is not worth the risk.

I understand.
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

[j s]

I guess I won't be using the parallel option in Visual Studio, then. A
non-deterministic build order is not worth the risk.

Juan

Post by j s

Post by j s
By default visual studio uses 2 threads on my machine, a dual core AMD.
I notice when I have RUNTESTS (I can't remember) enabled, it may attempt

to

Post by j s
run the tests before my project is even built.
If certain sub directories depend on a target in another directory

already

Post by j s
being built, is it safe to use the multi-threaded option in visual

studio?
I have been using parallel builds with visual studio 2005 for over 1
year. It sometimes creates situations like this but since it saves me
10s of minutes per day I deal with the issues. Generally building the
same project a second time fixes any problems and is usually quick
since most stuff builds correctly on the first pas.
John

[David Cole]

If something is non-deterministic then it means there is a missing
dependency somewhere...
This should work. We have many dashboards that run with parallel Visual
Studio builds and work reliably day after day. Whenever I have seen a
problem with it, there has always been a missing depedency expression in the
CMakeLists.txt files.

What exactly is it that does not build in the proper order? (Send along some
CMakeLists snippets perhaps...)

When you say "RUNTESTS (I can't remember) enabled" that does not really help
us help you. If you could just post the error output, perhaps the group can
spot something...


David

Post by j s
I guess I won't be using the parallel option in Visual Studio, then. A
non-deterministic build order is not worth the risk.
Juan

Post by j s

Post by j s
By default visual studio uses 2 threads on my machine, a dual core AMD.
I notice when I have RUNTESTS (I can't remember) enabled, it may attempt

to

Post by j s
run the tests before my project is even built.
If certain sub directories depend on a target in another directory

already

Post by j s
being built, is it safe to use the multi-threaded option in visual

studio?
I have been using parallel builds with visual studio 2005 for over 1
year. It sometimes creates situations like this but since it saves me
10s of minutes per day I deal with the issues. Generally building the
same project a second time fixes any problems and is usually quick
since most stuff builds correctly on the first pas.
John

_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

[j s]

Going into the build configuration and telling it to run tests, (I think the
checkbox is RUNTESTS, but I don't have my laptop here), causes it to run
tests as it is trying to compile the main executable for my project.

The directory structure is:
lib1
lib2
lib3
main (location of the main() entry point of the program)
testing

CTEST is being called before it is done compiling my application.

Regards,

Juan

Post by David Cole
If something is non-deterministic then it means there is a missing
dependency somewhere...
This should work. We have many dashboards that run with parallel Visual
Studio builds and work reliably day after day. Whenever I have seen a
problem with it, there has always been a missing depedency expression in the
CMakeLists.txt files.
What exactly is it that does not build in the proper order? (Send along
some CMakeLists snippets perhaps...)
When you say "RUNTESTS (I can't remember) enabled" that does not really
help us help you. If you could just post the error output, perhaps the group
can spot something...
David

Post by j s
I guess I won't be using the parallel option in Visual Studio, then. A
non-deterministic build order is not worth the risk.
Juan

Post by j s

Post by j s
By default visual studio uses 2 threads on my machine, a dual core AMD.
I notice when I have RUNTESTS (I can't remember) enabled, it may

attempt to

Post by j s
run the tests before my project is even built.
If certain sub directories depend on a target in another directory

already

Post by j s
being built, is it safe to use the multi-threaded option in visual

studio?
I have been using parallel builds with visual studio 2005 for over 1
year. It sometimes creates situations like this but since it saves me
10s of minutes per day I deal with the issues. Generally building the
same project a second time fixes any problems and is usually quick
since most stuff builds correctly on the first pas.
John

_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

[John Drescher]

Post by j s
Going into the build configuration and telling it to run tests, (I think the
checkbox is RUNTESTS, but I don't have my laptop here), causes it to run
tests as it is trying to compile the main executable for my project.
lib1
lib2
lib3
main (location of the main() entry point of the program)
testing
CTEST is being called before it is done compiling my application.

I can tell you that tests can be a problem because RUN_TEST is by
default not dependent on any of the other projects in your build.
There have been a couple of threads about this a few months back.

John

[j s]

To clarify, if my dependencies are properly spelled out correctly for all of
my targets, the build order will be handled properly in Visual Studio?

Regards,

Juan

Post by John Drescher

Post by j s
Going into the build configuration and telling it to run tests, (I think

the

Post by j s
checkbox is RUNTESTS, but I don't have my laptop here), causes it to run
tests as it is trying to compile the main executable for my project.
lib1
lib2
lib3
main (location of the main() entry point of the program)
testing
CTEST is being called before it is done compiling my application.

I can tell you that tests can be a problem because RUN_TEST is by
default not dependent on any of the other projects in your build.
There have been a couple of threads about this a few months back.
John

[Bill Hoffman]

Post by j s
To clarify, if my dependencies are properly spelled out correctly for
all of my targets, the build order will be handled properly in Visual
Studio?

Yes, and if you are running it from the command line, you should build
the ALL_BUILD target. Then build the RUN_TESTS. Currently, the
RUN_TESTS is expected to be run after the build is done, and should not
be built before.

-Bill

[Andrew Maclean]

I get this issue building VTK, ParaView and ITK on Windows. It often
happens if I haven't build for a week or so. Generally just re-running
the build again fixes everything.
To me it seems to be a Visual Studio issue not CMake.

Andrew

Post by j s
To clarify, if my dependencies are properly spelled out correctly for all
of my targets, the build order will be handled properly in Visual Studio?

Yes, and if you are running it from the command line, you should build the
ALL_BUILD target.  Then build the RUN_TESTS.  Currently, the RUN_TESTS is
expected to be run after the build is done, and should not be built before.
-Bill
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
http://www.cmake.org/Wiki/CMake_FAQ
http://www.cmake.org/mailman/listinfo/cmake

--
___________________________________________
Andrew J. P. Maclean
Centre for Autonomous Systems
The Rose Street Building J04
The University of Sydney 2006 NSW
AUSTRALIA
Ph: [phone]
Fax: [phone]
URL: http://www.acfr.usyd.edu.au/
___________________________________________