This will become available in Clang 21:
> ThreadSafetyAnalysis now supports -Wthread-safety-pointer, which
> enables warning on passing or returning pointers to guarded variables
> as function arguments or return value respectively. Note that
> ThreadSafetyAnalysis still does not perform alias analysis. The
> feature will be default-enabled with -Wthread-safety in a future release.
See
https://github.com/llvm/llvm-project/blob/main/clang/docs/ReleaseNotes.rst.
a5ac43d98d doc: Add release notes describing bitcoin wrapper executable (Ryan Ofsky)
258bda80c0 doc: Mention bitcoin wrapper executable in documentation (Ryan Ofsky)
d2739d75c9 build: add bitcoin.exe to windows installer (Sjors Provoost)
ba649c0006 ci: Run multiprocess tests through wrapper executable (Ryan Ofsky)
29bdd743bb test: Support BITCOIN_CMD environment variable (Ryan Ofsky)
9c8c68891b multiprocess: Add bitcoin wrapper executable (Ryan Ofsky)
5076d20fdb util: Add cross-platform ExecVp and GetExePath functions (Ryan Ofsky)
Pull request description:
Intended to make bitcoin command line features more discoverable and allow installing new multiprocess binaries in libexec/ instead of bin/ so they don't cause confusion.
Idea and implementation of this were discussed in https://github.com/bitcoin/bitcoin/issues/30983.
---
Initial implementation of this feature is deliberately minimal so the UX can evolve in response to feedback and there are not too many details to debate and discuss in a single PR. But many improvements are possible or planned:
- Adding manpage and bash completions.
- Showing nicer error messages that detect if an executable isn't installed and suggest how to fix [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2073194474)
- Showing wrapper command lines in subcommand in help output [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2077800405). This could be done conditionally as suggested in the comment or be unconditional.
- Showing wrapper command lines in subcommand error output. There is a bitcoin-cli error pointed out in [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2091152243) that is needlessly confusing.
- Integrating help so `bitcoin help subcommand` invokes `bitcoin subcommand -h`. `bitcoin -h subcommand` should also be supported and be equivalent [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2093116725)
- Adding support for `bitcoin-util` subcommands. Ideal interface would probably be more like `bitcoin grind` not `bitcoin util grind` but this has been punted for now. Supporting subcommands directly would require some ArgsManager modifications
- Adding a dedicated python functional test for the wrapper. Right now there is some CI coverage by setting the `BITCOIN_CMD` variable, but this doesn't cover things like the help output and version output, and support for different directory layouts.
- Better `--multiprocess` (`-m`) / `--monolithic` (`-M`) default selection. Right now, default is monolithic but it probably makes sense to chose more intelligently depending on whether -ipc options are enabled and what binaries are available.
- Maybe parsing `bitcoin.conf` and supporting options to control wrapper behavior like custom locations or preferences or aliases.
- Better command command line usability. Allow combining short options like (`-ah`). Allow fuzzy matching of subcommands or suggestions if you misspell. (suggested by stickies in review club)
- Not directly related to this PR but `bitcoin-cli named` implementation used by the wrapper should do a better job disambiguating named arguments from base64 arguments ending in = as pointed out in [(comment)](https://github.com/bitcoin/bitcoin/pull/31375#discussion_r2091886628)
---
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722). A review club meeting for it took place in https://bitcoincore.reviews/31375
ACKs for top commit:
Sjors:
utACK a5ac43d98d
achow101:
ACK a5ac43d98d
vasild:
ACK a5ac43d98d
theStack:
ACK a5ac43d98d
ismaelsadeeq:
fwiw my last review implied an ACK a5ac43d98d
hodlinator:
ACK a5ac43d98d
Tree-SHA512: 570e6a4ff8bd79ef6554da3d01f36c0a7c6d2dd7dace8f8732eca98f4a8bc2284474a9beadeba783114fe2f3dd08b2041b3da7753bae0b7f881ec50668cb821f
Windows application manifests provide several benefits. However, on the
master branch, the linker generates and embeds manifests only when
building with MSVC.
This change unifies manifest embedding for both native and
cross-compilation.
Intended to make bitcoin command line features more discoverable and allow
installing new multiprocess binaries in libexec/ instead of bin/ so they don't
cause confusion.
Idea and implementation of this were discussed in
https://github.com/bitcoin/bitcoin/issues/30983
Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
This change makes the `WITH_DBUS` option available on all Unix-like
systems, not just Linux, thereby fixing a regression that was
overlooked during the migration from Autotools.
Note: Enabling D-Bus support on macOS still makes no sense, since the
`Notificator` class uses the User Notification Center regardless.
edde96376a cmake: Respect user-provided configuration-specific flags (Hennadii Stepanov)
Pull request description:
This PR addresses [this](https://github.com/bitcoin/bitcoin/issues/31491#issuecomment-2542140874) comment:
> I suppose that should only happen if the `-O3` isn't coming from an explicitly set `CMAKE_CXX_FLAGS_RELEASE`.
With this PR:
```
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="-O3"
<snip>
C++ compiler flags .................... -O3 -std=c++20 -fPIC -fno-extended-identifiers -fdebug-prefix-map=/home/hebasto/dev/bitcoin/src=. -fmacro-prefix-map=/home/hebasto/dev/bitcoin/src=. -fstack-reuse=none -Wall -Wextra -Wformat -Wformat-security -Wvla -Wredundant-decls -Wdate-time -Wduplicated-branches -Wduplicated-cond -Wlogical-op -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wbidi-chars=any -Wundef -Wno-unused-parameter -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection
Linker flags .......................... -O3 -fstack-reuse=none -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie
```
and
```
$ cmake -B build -DCMAKE_BUILD_TYPE=Release
<snip>
C++ compiler flags .................... -O2 -std=c++20 -fPIC -fno-extended-identifiers -fdebug-prefix-map=/home/hebasto/dev/bitcoin/src=. -fmacro-prefix-map=/home/hebasto/dev/bitcoin/src=. -fstack-reuse=none -Wall -Wextra -Wformat -Wformat-security -Wvla -Wredundant-decls -Wdate-time -Wduplicated-branches -Wduplicated-cond -Wlogical-op -Woverloaded-virtual -Wsuggest-override -Wimplicit-fallthrough -Wunreachable-code -Wbidi-chars=any -Wundef -Wno-unused-parameter -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -Wstack-protector -fstack-protector-all -fcf-protection=full -fstack-clash-protection
Linker flags .......................... -O2 -fstack-reuse=none -fstack-protector-all -fcf-protection=full -fstack-clash-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,separate-code -fPIE -pie
```
When calling `cmake` repeatedly using the same build directory, each newly provided `CMAKE_CXX_FLAGS_RELEASE` value will be accommodated. In such a scenario, if the user wishes to revert to the build system defaults, they should unset the `CMAKE_CXX_FLAGS_RELEASE` variable by passing `-UCMAKE_CXX_FLAGS_RELEASE` to `cmake`.
---
This PR does not aim to resolve _all_ issues mentioned in https://github.com/bitcoin/bitcoin/issues/31491.
ACKs for top commit:
purpleKarrot:
ACK edde96376a
janb84:
ACK [edde963](edde96376a)
ryanofsky:
Code review ACK edde96376a
Tree-SHA512: 1fbc879bd02cf0be726ced490f65985e728f0686ccb3a32cd38787b56377aa666e1965448e5069515abc814df49a0083c8000bc3f6f322f5f395695638168fb6
77e553ab6a build: refactor: hardening flags -> core_interface (David Gumberg)
00ba3ba303 build: Drop option for disabling hardening (David Gumberg)
f57db75e91 build: Use `-z noseparate-code` on NetBSD < 11.0 (David Gumberg)
Pull request description:
Follow up to #32038 which dropped `NO_HARDEN` from depends builds, this PR drops the `ENABLE_HARDENING` build option since disabling hardening of binaries should not be a supported or maintained use case. With this change, hardening flags are always enabled.
Individual hardening flags and options can still be disabled by appending flags, e.g.:
```bash
cmake -B build \
-DAPPEND_CPPFLAGS='-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -fno-stack-protector -fcf-protection=none -fno-stack-clash-protection' \
-DAPPEND_LDFLAGS='-Wl,-z,lazy -Wl,-z,norelro -Wl,-z,noseparate-code'
```
There is an issue with NetBSD 10.0's dynamic linker that makes one of the hardening linker flags, `-z separate-code`, [problematic](https://github.com/bitcoin/bitcoin/pull/28724#issuecomment-2589347934), so this PR also introduces a check to prevent the use of this flag in NetBSD versions < 11.0, (where this issue is [fixed](acf7fb3abf)). The fix for this [might be backported](https://mail-index.netbsd.org/tech-userlevel/2023/01/05/msg013670.html) to NetBSD 10.0.
I suggest reviewing the diff with whitespace changes hidden (`git diff -w` or using github's hide whitespace option)
ACKs for top commit:
hebasto:
re-ACK 77e553ab6a.
laanwj:
re-ACK 77e553ab6a
janb84:
ACK [77e553a](77e553ab6a)
vasild:
ACK 77e553ab6a
musaHaruna:
tested ACK [77e553](77e553ab6a)
Tree-SHA512: b149fb0371d12312c140255bf674c2bdc9f5272a5750a5b9ec5f192323364bb2ea8e164af13b9ab981ab3aa7ceb91b7a64785081e7458470e81c2f5228abf7b1
This change is technically not needed to add libmultiprocess as a subtree, but
it avoids a CI failure in followup PR #30975 which enables multiprocess build
option in more CI jobs. In that PR, several jobs fail due to the mptest
executable not being built by default, as reported
https://github.com/bitcoin/bitcoin/pull/30975#issuecomment-2623801480
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
When ENABLE_IPC option is on, build with libmultiprocess subtree and
`add_subdirectory(src/ipc/libmultiprocess)` instead of external package
and `find_package(Libmultiprocess)` by default.
Behavior can be toggled with `WITH_EXTERNAL_LIBMULTIPROCESS` option. Using a
subtree should be more convenient for most bitcoin developers, but using an
external package is more convenient for developing in the libmultiprocess
repository.
The `WITH_EXTERNAL_LIBMULTIPROCESS` option is also used to avoid needing to
changing the depends build here. But in later commits, the depends build is
switched to use the add_subdirectory build as well.
Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
Rename WITH_MULTIPROCESS to ENABLE_IPC, because ENABLE_IPC is a more accurate
name for the feature. It controls whether the src/ipc/ directory is built and
whether IPC features like -ipcbind, -ipcconnect, and -ipcfd are available. It
does NOT currently enable multiprocess features which are implemented in #10102
building on top of the IPC features. It will also no longer (as of the next
commit), control whether a find_package call is made so the "WITH_" prefix is
also inappropriate.
-BEGIN VERIFY SCRIPT-
git grep -l WITH_MULTIPROCESS | xargs sed -i s/WITH_MULTIPROCESS/ENABLE_IPC/g
-END VERIFY SCRIPT-
57d8b1f1b3 cmake: Avoid fuzzer "multiple definition of `main'" errors (Ryan Ofsky)
Pull request description:
This change builds libraries with `-fsanitize=fuzzer-no-link` instead of `-fsanitize=fuzzer` when the cmake `-DSANITIZERS=fuzzer` option is specified. This is necessary to make fuzzing and IPC cmake options compatible with each other and avoid CI failures in #30975 which enables IPC in the fuzzer CI build:
https://cirrus-ci.com/task/5366255504326656?logs=ci#L2817https://cirrus-ci.com/task/5233064575500288?logs=ci#L2384
The failures can also be reproduced by checking out #31741 and building with `cmake -B build -DBUILD_FOR_FUZZING=ON -DSANITIZERS=fuzzer -DENABLE_IPC=ON` with this fix reverted.
The fix updates the cmake build so when `-DSANITIZERS=fuzzer` is specified, the fuzz test binary is built with `-fsanitize=fuzzer` (so it can use libFuzzer's main function), and libraries are built with `-fsanitize=fuzzer-no-link` (so they can be linked into other executables with their own main functions).
Previously when `-DSANITIZERS=fuzzer` was specified, `-fsanitize=fuzzer` was applied to ALL libraries and executables. This was inappropriate because it made it impossible to build any executables other than the fuzz test executable without triggering link errors:
- `` multiple definition of `main' ``
- `` "undefined reference to `LLVMFuzzerTestOneInput' ``
if they depended on any libraries instrumented for fuzzing.
This was especially a problem when the `ENABLE_IPC` option was set because it made building the `mpgen` code generator impossible so nothing else that depended on generated sources, including the fuzz test binary, could be built either.
This commit was previously part of https://github.com/bitcoin/bitcoin/pull/31741 and had some discussion there starting in https://github.com/bitcoin/bitcoin/pull/31741#pullrequestreview-2619682385
---
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722).
ACKs for top commit:
hebasto:
ACK 57d8b1f1b3, tested on Ubuntu 24.04.
Tree-SHA512: 4011adbc0b08742e83cf7c0560d3d5b5694a863358e6ac9a21239626b4a8fedceca66db34b5a46136a7b26849bb1d8710c894689322ae97e1c407687c3f57d50
52ac17757e cmake: Add `NO_CACHE_IF_FAILED` option for checking linker flags (Hennadii Stepanov)
Pull request description:
Use it for checking `-fsanitize`.
This change improves the user experience when the configuration step fails due to a missing library. Now, there is no need to manually clean the CMake cache after installing the required library.
Addresses [this](https://github.com/bitcoin/bitcoin/issues/31942#issuecomment-2703801270) comment from https://github.com/bitcoin/bitcoin/issues/31942.
ACKs for top commit:
fanquake:
ACK 52ac17757e
Tree-SHA512: 4004110585413792faa01551cf5a5b3b0de7f213c7a1dd333647107741f84abf626fd0ed067fc17e4c5a523de549432738d3752facf25d1e3dab240be8d13d03
Building unhardened executables is not a supported use case that should
be maintained and those that want unhardened executables can still
override them by appending disable flags.
For example:
cmake -B build -DAPPEND_CPPFLAGS='-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -fno-stack-protector -fcf-protection=none -fno-stack-clash-protection' -DAPPEND_LDFLAGS='-Wl,-z,lazy -Wl,-z,norelro -Wl,-z,noseparate-code'
36b6f36ac4 build: require sqlite when building the wallet (Sjors Provoost)
Pull request description:
Require that sqlite is available in order to compile the wallet. Removes instances of `USE_SQLITE` since it is no longer possible to not have sqlite available.
The `NO_SQLITE` option is dropped from depends.
This is another step towards dropping the legacy wallet, extracted from #31250.
ACKs for top commit:
m3dwards:
ACK 36b6f36ac4
davidgumberg:
crACK 36b6f36ac4
hebasto:
re-ACK 36b6f36ac4.
Tree-SHA512: 870a0135671c80c4f28602119eb8637a1ed43b51b1673bfe88425782fb62ec6ef0f3d6baf0d5984d6a243779b0f63423fd4c4dc324ef87bffba13d63e05ad793
Require that sqlite is available in order to compile the wallet. Removes
instances of USE_SQLITE since it is no longer possible to not have
sqlite available.
The NO_SQLITE option is dropped from depends.
Co-authored-by: Ava Chow <github@achow101.com>
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
568fcdddae scripted-diff: Adjust documentation per top-level target output location (Hennadii Stepanov)
026bb226e9 cmake: Set top-level target output locations (Hennadii Stepanov)
Pull request description:
This PR sets the target output locations to the `bin` and `lib` subdirectories within the build tree, creating a directory structure that mirrors that of the installed targets.
This approach is widely adopted by the large projects, such as [LLVM](e146c1867e/lldb/cmake/modules/LLDBStandalone.cmake (L128-L130)):
```cmake
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX})
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib${LLVM_LIBDIR_SUFFIX})
```
The `libsecp256k1` project has also recently [adopted](https://github.com/bitcoin-core/secp256k1/pull/1553) this approach.
With this PR, all binaries are conveniently located. For example, run:
```
$ ./build/bin/fuzz
```
instead of:
```
$ ./build/src/test/fuzz/fuzz
```
On Windows, all required DLLs are now located in the same directory as the executables, allowing to run `bitcoin-chainstate.exe` (which loads `bitcoinkernel.dll`) without the need to copy DLLs or modify the `PATH` variable.
The idea was briefly discussed among the build team during the recent CoreDev meeting.
---
**Warning**: This PR changes build locations of newly built executables like `bitcoind` and `test_bitcoin` from `src/` to `bin/` without deleting previously built executables. A clean build is recommended to avoid accidentally running old binaries.
ACKs for top commit:
theStack:
Light re-ACK 568fcdddae
ryanofsky:
Code review ACK 568fcdddae. Only change since last review was rebasing. I'm ok with this PR in its current form if other developers are happy with it. I just personally think it is inappropriate to \*silently\* break an everyday developer workflow like `git pull; make bitcoind`. I wouldn't have a problem with this PR if it triggered an explicit error, or if the problem was limited to less common workflows like changing cmake options in an existing build.
TheCharlatan:
Re-ACK 568fcdddae
theuni:
ACK 568fcdddae
Tree-SHA512: 1aa5ecd3cd49bd82f1dcc96c8e171d2d19c58aec8dade4bc329df89311f9e50cbf6cf021d004c58a0e1016c375b0fa348ccd52761bcdd179c2d1e61c105e3b9f
Use it for checking `-fsanitize`.
This change improves the user experience when the configuration step
fails due to a missing library. Now, there is no need to manually clean
the CMake cache after installing the required library.
This change builds libraries with -fsanitize=fuzzer-no-link instead of
-fsanitize=fuzzer when the cmake -DSANITIZERS=fuzzer option is specified. This
is necessary to make fuzzing and IPC cmake options compatible with each other
and avoid CI failures in #30975 which enables IPC in the fuzzer CI build:
https://cirrus-ci.com/task/5366255504326656?logs=ci#L2817https://cirrus-ci.com/task/5233064575500288?logs=ci#L2384
The failures can also be reproduced by checking out #31741 and building with
`cmake -B build -DBUILD_FOR_FUZZING=ON -DSANITIZERS=fuzzer -DENABLE_IPC=ON`
with this fix reverted.
The fix updates the cmake build so when -DSANITIZERS=fuzzer is specified, the
fuzz test binary is built with -fsanitize=fuzzer (so it can use libFuzzer's
main function), and libraries are built with -fsanitize=fuzzer-no-link (so they
can be linked into other executables with their own main functions).
Previously when -DSANITIZERS=fuzzer was specified, -fsanitize=fuzzer was
applied to ALL libraries and executables. This was inappropriate because it
made it impossible to build any executables other than the fuzz test executable
without triggering link errors:
- "multiple definition of `main'"
- "undefined reference to `LLVMFuzzerTestOneInput'"
if they depended on any libraries instrumented for fuzzing.
This was especially a problem when the ENABLE_IPC option was set because it
made building the mpgen code generator impossible so nothing else that depended
on generated sources, including the fuzz test binary, could be built either.
This commit was previously part of
https://github.com/bitcoin/bitcoin/pull/31741 and had some discussion there
starting in
https://github.com/bitcoin/bitcoin/pull/31741#pullrequestreview-2619682385
This change:
1. Collects build artifacts in dedicated locations.
2. Allows running bitcoin-chainstate.exe with bitcoinkernel.dll directly
from the build tree on Windows.
2c4b229c90 cmake: Introduce `FUZZ_LIBS` (Hennadii Stepanov)
ea929c0848 scripted-diff: Rename CMake helper module (Hennadii Stepanov)
8d238c1dfd cmake: Delete `check_cxx_source_links*` macros (Hennadii Stepanov)
71bf8294a9 cmake: Convert `check_cxx_source_compiles_with_flags` to a function (Hennadii Stepanov)
88ee6800c9 cmake: Delete `check_cxx_source_links_with_flags` macro (Hennadii Stepanov)
09e8fd25b1 build: Don't override CMake's default try_compile target (Hennadii Stepanov)
Pull request description:
This was requested in https://github.com/bitcoin/bitcoin/pull/31359#issuecomment-2515287092.
From https://github.com/bitcoin/bitcoin/pull/31359#issuecomment-2511246212:
> (Almost?) every CMake check internally uses the [`try_compile()`](https://cmake.org/cmake/help/latest/command/try_compile.html) command, whose behaviour, in turn, depends on the [`CMAKE_TRY_COMPILE_TARGET_TYPE`](https://cmake.org/cmake/help/latest/variable/CMAKE_TRY_COMPILE_TARGET_TYPE.html) variable:
>
> 1. The default value, `EXECUTABLE`, enables both compiler and linker checks.
>
> 2. The `STATIC_LIBRARY` value enables only compiler checks.
>
>
> To mimic Autotools' behaviour, we [disabled](d3f42fa08f/cmake/module/CheckSourceCompilesAndLinks.cmake (L9-L10)) linker checks by setting `CMAKE_TRY_COMPILE_TARGET_TYPE` to `STATIC_LIBRARY` globally (perhaps not the best design). This effectively separates the entire CMake script into regions where `CMAKE_TRY_COMPILE_TARGET_TYPE` is:
>
> * unset
>
> * set to `STATIC_LIBRARY`
>
> * set to `EXECUTABLE`
From https://github.com/bitcoin/bitcoin/pull/31359#issuecomment-2515287092:
> > This seems very fragile and unintuitive, and the fact that this could silently break at any point is not documented in any way. I don't think other bad design decisions should lead to us having to write even more boilerplate code to fix things that should "just work" (minus the upstream bugs).
>
> Agreed. I forgot that we set `CMAKE_TRY_COMPILE_TARGET_TYPE` globally. And even worse, it's buried in a module. If that upsets CMake internal tests, I think we should undo that.
This PR ensures that `CMAKE_TRY_COMPILE_TARGET_TYPE` is modified only within local scopes.
Additionally, the `FUZZ_LIBS` variable has been introduced to handle additional libraries required for linking, rather than link options, in certain build environment, such as OSS-Fuzz.
ACKs for top commit:
TheCharlatan:
Re-ACK 2c4b229c90
theuni:
utACK 2c4b229c90
Tree-SHA512: f72ffa8f50f216fc1a2f8027ba8ddfd4acd42b94ff6c1cb2138f2da51eb8f945660e97d3c247d7f3f7ec8dfebbccec3ab84347d6ae2e3f8a40f3d7aa8b14cde9
CMake distinguishes recommended methods for handling (1) linker options
and (2) libraries used during linking. Therefore, it is both reasonable
and consistent to introduce a dedicated variable for the latter,
particularly when a build environment, such as OSS-Fuzz, requires
linking against additional libraries.
81c174e318 cmake: Refer to the configure log instead of printing PIE test error (Hennadii Stepanov)
65a0920ca6 cmake: Add `CheckLinkerSupportsPIE` module (Hennadii Stepanov)
Pull request description:
This new module is a wrapper around CMake's `CheckPIESupported` module that fixes an upstream bug.
See: https://gitlab.kitware.com/cmake/cmake/-/issues/26463.
Fixes https://github.com/bitcoin/bitcoin/issues/30771.
ACKs for top commit:
theuni:
utACK 81c174e318.
vasild:
ACK 81c174e318
Tree-SHA512: 77d7022238551a4e69c59d1fe6b78975bb552cbbed5339459853d7ebf0086813036081f464fed230be330b3bd7d6cf8590b536b064028d2f786d6ae40f342f95
dead908654 cmake: Improve compatibility with Python version managers (Hennadii Stepanov)
Pull request description:
This PR resolves the issue [highlighted](https://github.com/bitcoin/bitcoin/pull/31411#issuecomment-2516745547) in https://github.com/bitcoin/bitcoin/pull/31411:
> Here's another case where CMake just picks some other Python...
The fix leverages two [hints](https://cmake.org/cmake/help/latest/module/FindPython3.html#hints) for the CMake `FindPython3` module:
1. `Python3_FIND_FRAMEWORK` is set to `LAST`. This ensures that Unix-style package components are preferred over frameworks on macOS. As a side effect, the `FindPython3` module reports a shim or symlink (e.g., from `pyenv`) rather than the underlying framework's binary. The module's output aligns with the result of the `which` command.
2. `Python3_FIND_UNVERSIONED_NAMES` is set to `FIRST`. This supports scenarios where tools like `pyenv`—which use shims—have multiple Python versions installed.
Here are examples of output on my macOS 15.1.1 (Intel) with installed Homebrew's [Python 3.13.0](https://formulae.brew.sh/formula/python@3.13):
- without any Python version manager:
```
% which -a python3
/usr/local/bin/python3
/usr/bin/python3
% cmake -B build
<snip>
-- Found Python3: /usr/local/bin/python3 (found suitable version "3.13.0", minimum required is "3.10") found components: Interpreter
<snip>
```
- using `pyenv`:
```
% pyenv versions
system
* 3.10.14 (set by /Users/hebasto/dev/bitcoin/.python-version)
3.12.8
3.13.1
% which -a python3
/Users/hebasto/.pyenv/shims/python3
/usr/local/bin/python3
/usr/bin/python3
% cmake -B build
<snip>
-- Found Python3: /Users/hebasto/.pyenv/shims/python3 (found suitable version "3.10.14", minimum required is "3.10") found components: Interpreter
<snip>
```
Both variables, `Python3_FIND_FRAMEWORK` and `Python3_FIND_UNVERSIONED_NAMES`, can still be overridden by the user via the command line if needed.
ACKs for top commit:
theuni:
No opinion on the python selection changes themselves, but code-review ACK dead908654
willcl-ark:
ACK dead908654
Tree-SHA512: 69f8541223e5b6c35c892b4ba2a2dcfc24b41a10cf20accc75d3008b16434db8a9240c99c886c3a4566ba24269c5b0e0d856357891811f0a77b39f4afbee3634
e3c0152769 cmake: Copy `cov_tool_wrapper.sh.in` to the build tree (Hennadii Stepanov)
Pull request description:
This PR ensures that `cov_tool_wrapper.sh.in` is available when invoking the `Coverage.cmake` script from any directory.
Here is an example of usage on Ubuntu 24.10 with the default GCC 14.2.0:
```
$ cmake -B build -DCMAKE_BUILD_TYPE=Coverage -DCMAKE_CXX_FLAGS="-fprofile-update=atomic" -DCMAKE_C_FLAGS="-fprofile-update=atomic"
$ cmake --build build -j $(nproc)
$ cd ..
$ cmake -DJOBS=$(nproc) -DLCOV_OPTS="--ignore-errors inconsistent,inconsistent" -P bitcoin/build/Coverage.cmake
```
Fixes https://github.com/bitcoin/bitcoin/issues/31638.
ACKs for top commit:
theuni:
utACK e3c0152769
Tree-SHA512: ccfc8e893567f199d9b05ea3916cac06fca89c5892cc7492d5251c331c35408222fd918ed08017515e2dfca10970ae3f633b3917bfb7037db539559e71d7f711
12fa9511b5 build: simplify dependency graph (Cory Fields)
c4e498300c build: avoid unnecessary dependencies on generated headers (Cory Fields)
Pull request description:
These changes speed up my build (default config/options/targets) by roughly 10%. I suspect the difference may be more significant in other build configs.
Before:
> $ time cmake --build build -j24
> real3m26.932s
After:
> $ time cmake --build build -j24
> real3m7.556s
Generally they allow for jobservers (either `make -jX` or `ninja`) to be better utilized. This can be verified using `top` while building and looking at the number of compiles running at any given time before/after these changes. Before, it's easy to observe periods of stalling when only one or two compiles are happening. After these changes, the compiler process count should mostly match the number of jobs given (`-jX`) until it falls off at the end.
---
The first commit sets [DEPENDS_EXPLICIT_ONLY](https://cmake.org/cmake/help/latest/command/add_custom_command.html#command:add_custom_command) for commands which generate our test header files. Without this option, `test_bitcoin`'s generated headers won't be built until all of its other dependencies have been built. This introduces a significant stall in the build, though currently only Ninja benefits from this being set, and only CMake >= 3.27 understands it.
Example from a generated `build.ninja`:
Before:
> \# Custom command for src/test/data/base58_encode_decode.json.h
>
> build src/test/data/base58_encode_decode.json.h | ${cmake_ninja_workdir}src/test/data/base58_encode_decode.json.h: CUSTOM_COMMAND /home/cory/dev/bitcoin/src/test/data/base58_encode_decode.json /home/cory/dev/bitcoin/cmake/script/GenerateHeaderFromJson.cmake || libcrc32c.a libcrc32c_sse42.a libleveldb.a libminisketch.a minisketch_clmul src/bitcoin_clientversion src/crypto/libbitcoin_crypto.a src/crypto/libbitcoin_crypto_avx2.a src/crypto/libbitcoin_crypto_sse41.a src/crypto/libbitcoin_crypto_x86_shani.a src/generate_build_info src/libbitcoin_cli.a src/libbitcoin_common.a src/libbitcoin_consensus.a src/libbitcoin_node.a src/secp256k1/src/libsecp256k1.a src/secp256k1/src/secp256k1_precomputed src/test/util/libtest_util.a src/univalue/libunivalue.a src/util/libbitcoin_util.a src/wallet/libbitcoin_wallet.a src/zmq/libbitcoin_zmq.a
After:
> \# Custom command for src/test/data/base58_encode_decode.json.h
>
> build src/test/data/base58_encode_decode.json.h | ${cmake_ninja_workdir}src/test/data/base58_encode_decode.json.h: CUSTOM_COMMAND /home/cory/dev/bitcoin/src/test/data/base58_encode_decode.json /home/cory/dev/bitcoin/cmake/script/GenerateHeaderFromJson.cmake
---
The second commit is more significant. It sets [CMAKE_OPTIMIZE_DEPENDENCIES](https://cmake.org/cmake/help/latest/prop_tgt/OPTIMIZE_DEPENDENCIES.html) globally, which allows the objects of static libs to be built in parallel when one lib depends on the other. This can be set as a per-lib property, ~but I don't see any need for that as we don't currently have any edge-cases where this wouldn't be ok. If those should arise, we could always disable on a per-lib basis~.
Edit: turns out this triggers an [upstream bug](https://gitlab.kitware.com/cmake/cmake/-/issues/24058), which I guess can be considered an edge-case until fixed in CMake. I've added 2 per-lib opt-outs as a result.
Example:
Before:
> \# Link the static library src/libbitcoin_cli.a
>
> build src/libbitcoin_cli.a: CXX_STATIC_LIBRARY_LINKER__bitcoin_cli_RelWithDebInfo src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o || src/univalue/libunivalue.a
After:
> \# Link the static library src/libbitcoin_cli.a
>
> build src/libbitcoin_cli.a: CXX_STATIC_LIBRARY_LINKER__bitcoin_cli_RelWithDebInfo src/CMakeFiles/bitcoin_cli.dir/compat/stdin.cpp.o src/CMakeFiles/bitcoin_cli.dir/rpc/client.cpp.o
>
ACKs for top commit:
l0rinc:
utACK 12fa9511b5
hebasto:
ACK 12fa9511b5.
Tree-SHA512: f85f507e70cdc06acd07542161d9f9b8edf9ba866f08c8ef17aaaed770fa11530a27521c4413456d863463a6e77d4d6983fa623a64e17bbd602c2bc70aacc112
56a9b847bb build: set build type and per-build-type flags as early as possible (Cory Fields)
f605f7a9c2 build: refactor: set debug definitions in main CMakeLists (Cory Fields)
Pull request description:
This ensures that most compiler tests are not run with the wrong build type's flags. The initial c++ checks are an exception to that because many internal CMake variables are unset until a language is selected, so it's problematic to change our build type before that.
The difference can be seen in `build/CMakeFiles/CMakeConfigureLog.yaml`. Before, `Debug` was used for many of the earlly checks. After this PR, it's only the first 2 checks.
ACKs for top commit:
hebasto:
ACK 56a9b847bb.
Tree-SHA512: 87947352d6d4fd08554515822cb13634ed3be33fcda2af817c22ef943b1d0856ceb39311ddc01b8221397528fdc09f630dc7e74fc92f5a4a073f09c4ae493596
With the exception of the first c++ checks, this ensures that compiler tests
are never run with the wrong build type's flags.
Co-Authored-By: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
fanquake
Sjors Provoost
MarcoFalke
Hennadii Stepanov
Ava Chow
Ryan Ofsky
David Gumberg
glozow
Cory Fields