[CMake] Concurrency issues with copy_if_different

[Olivier Croquette]

Hello,

in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.

I see two options to fix this:

1. don't use copy_if_different, but a custom tool that can deal with the
concurrency

2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go

What do you think?

Cheers
Olivier
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[Robert Maynard]

Personally would go with option number 2

Post by Olivier Croquette
Hello,
in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.
1. don't use copy_if_different, but a custom tool that can deal with the
concurrency
2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go
What do you think?
Cheers
Olivier
--
Powered by www.kitware.com
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
https://cmake.org/mailman/listinfo/cmake

[J. Caleb Wherry]

Had the same problem and went with #2

-Caleb

Post by Olivier Croquette
Hello,
in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.
1. don't use copy_if_different, but a custom tool that can deal with the
concurrency
2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go
What do you think?
Cheers
Olivier
--
Powered by www.kitware.com
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
https://cmake.org/mailman/listinfo/cmake

[Olivier Croquette]

Post by J. Caleb Wherry

Post by Olivier Croquette
in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.
1. don't use copy_if_different, but a custom tool that can deal with the
concurrency
2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go
What do you think?

Had the same problem and went with #2

Hi Caleb,

could you give some more details how you did this? Filling up the list
is easy, but I am not sure how to define the final action that will copy
all the DLLs, and make sure it happens only once. Did you define a new
custom target for that?
--
Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

[J. Caleb Wherry]

Our case is pretty simplified in that almost every executable uses the
dependent DLLs/SOs I need copied. For C# apps it is log4net, NDesk.Options,
etc. For native apps it’s fmpeg shared libraries.

So when I create the list at configure time, I immediately just use CMake
to copy the files the where I want them in the build tree. This guarantees
that they are copied the one time when CMake is run instead of at build
time.

A few things that make this work for us:

* These libraries are checked into our repo so they aren’t changed by
automatically pulling new binaries or anything like that.

* We only have a handful of these dependent 3rd party libs, like 7.

* Couldn’t think of a better way to do it.

If you did want to move it to build time instead of configure time, you’d
just have to make a custom target that every other target depends on. In
some systems this isn’t easy but if you write wrappers around creating
targets then you can do extra book keeping when a target is create. In this
case, you just add another depends on the custom target that copies the
dependent libs.

-Caleb

Post by Olivier Croquette

Post by J. Caleb Wherry

Post by Olivier Croquette
in one of our projects, we use copy_if_different to copy some DLLs
required by the runtime. It's called as post-build action. The problem
is that several targets want to copy the same DLLs, and when using
parallelized builds, the different "cmake -E copy_if_different" can
conflict, leading the whole build to fail.
1. don't use copy_if_different, but a custom tool that can deal with the
concurrency
2. instead of adding post-build actions to different targets, fill up a
list containing all the files required, and add a single post-build
action to a single target that copies all the required files in one go
What do you think?

Had the same problem and went with #2

Hi Caleb,
could you give some more details how you did this? Filling up the list
is easy, but I am not sure how to define the final action that will copy
all the DLLs, and make sure it happens only once. Did you define a new
custom target for that?
--

Sent from my iPhone SE