Understanding CROSSTOOL

To get hands-on with CROSSTOOL, see the Configuring CROSSTOOL tutorial.

Overview

CROSSTOOL is a text file containing a protocol buffer that provides the necessary level of granularity for configuring the behavior of Bazel’s C++ rules. By default, Bazel automatically configures CROSSTOOL for your build, but you have the option to configure it manually. You reference the CROSSTOOL file in your BUILD file(s) using a cc_toolchain target and check it into source control alongside your project. You can share a single CROSSTOOL file across multiple projects or create separate per-project files.

When a C++ target enters the analysis phase, Bazel selects the appropriate cc_toolchain target based on the BUILD file, then reads the corresponding toolchain definition from the CROSSTOOL file. The cc_toolchain target passes information from the CROSSTOOL proto to the C++ target through a CcToolchainProvider.

For example, a compile or link action, instantiated by a rule such as cc_binary or cc_library, needs the following information:

  • The compiler or linker to use
  • Command-line flags for the compiler/linker
  • Configuration flags passed through the --copt/--linkopt options
  • Environment variables
  • Artifacts needed in the sandbox in which the action executes

All of the above information except the artifacts required in the sandbox is specified in the CROSSTOOL proto.

The artifacts to be shipped to the sandbox are declared in the cc_toolchain target. For example, with the cc_toolchain.linker_files attribute you can specify the linker binary and toolchain libraries to ship into the sandbox.

CROSSTOOL proto structure

The CROSSTOOL proto has the following structure:

  • Map from --cpu to toolchain (to be used when --compiler is not specified or when cc_toolchain_suite.toolchains omits the cpu entry)

  • Toolchain for a particular --cpu and --compiler combination (1)
    • Static toolchain:
      • compiler_flags
      • linker_flags
      • compilation_mode_flags
      • linking_mode_flags
    • Dynamic toolchain:
      • features

  • Toolchain for a particular --cpu and --compiler combination (2)

  • Toolchain for a particular --cpu and --compiler combination (3)

  • …

Toolchain selection

The toolchain selection logic operates as follows:

  1. User specifies a cc_toolchain_suite target in the BUILD file and points Bazel to the target using the --crosstool_top option. The CROSSTOOL file must reside in the same directory as the BUILD file containing the cc_toolchain_suite target.

  2. The cc_toolchain_suite target and the CROSSTOOL file reference multiple toolchains. The values of the --cpu and --compiler flags determine which of those toolchains is selected, either based only on the --cpu flag value, or based on a joint --cpu | --compiler value. The selection process is as follows:

  • If the --compiler option is specified, Bazel selects the corresponding entry from the cc_toolchain_suite.toolchains attribute with --cpu | --compiler. If Bazel does not find a corresponding entry, it throws an error.

  • If the --compiler option is not specified, Bazel selects the corresponding entry from the cc_toolchain_suite.toolchains attribute with just --cpu.

    However, if Bazel does not find a corresponding entry and the --incompatible_disable_cc_toolchain_label_from_crosstool_proto option is disabled, Bazel iterates through default_toolchains in the CROSSTOOL file until it finds an entry where the default_toolchain.cpu value matches the specified --cpu option value. Bazel then reads the toolchain_identifier value to identify the corresponding toolchain, and selects the appropriate entry in the cc_toolchain_suite.toolchains attribute using toolchain.target_cpu | toolchain.compiler.

  • If no flags are specified, Bazel inspects the host system and selects a --cpu value based on its findings. See the inspection mechanism code.

Once a toolchain has been selected, corresponding feature and action_config messages in the CROSSTOOL file govern the configuration of the build (that is, items described earlier in this document). These messages allow the implementation of fully fledged C++ features in Bazel without modifying the Bazel binary. C++ rules support multiple unique actions documented in detail in the Bazel source code.

CROSSTOOL features

A feature is an entity that requires non-default command-line flags, actions, constraints on the execution environment, or dependency alterations. A feature can be something as simple as allowing BUILD files to select configurations of flags, such as treat_warnings_as_errors, or interact with the C++ rules and include new compile actions and inputs to the compilation, such as header_modules or thin_lto.

Ideally, a toolchain definition consists of a set of features, where each feature consists of multiple flag groups, each defining a list of flags that apply to specific Bazel actions.

A feature is specified by name, which allows full decoupling of the CROSSTOOL configuration from Bazel releases. In other words, a Bazel release does not affect the behavior of CROSSTOOL configurations as long as those configurations do not require the use of new features.

A feature is enabled only when both Bazel and the CROSSTOOL configuration support it.

Features can have interdependencies, depend on command line flags, BUILD file settings, and other variables.

Feature relationships

Dependencies are typically managed directly with Bazel, which simply enforces the requirements and manages conflicts intrinsic to the nature of the features defined in the build. The toolchain specification allows for more granular constraints for use directly within the CROSSTOOL file that govern feature support and expansion. These are:

</table> ## `CROSSTOOL` actions `CROSSTOOL` actions provide the flexibility to modify the circumstances under which an action executes without assuming how the action will be run. An `action_config` specifies the tool binary that an action invokes, while a `feature` specifies the configuration (flags) that determine how that tool behaves when the action is invoked. [Features](#crosstool-features) reference `CROSSTOOL` actions to signal which Bazel actions they affect since `CROSSTOOL` actions can modify the Bazel action graph. The `CROSSTOOL` file includes actions that have flags and tools associated with them, such as `c++-compile`. Flags are assigned to each action by associating them with a feature. Each `CROSSTOOL` action name represents a single type of action performed by Bazel, such as compiling or linking. There is, however, a many-to-one relationship between `CROSSTOOL` actions and Bazel action types, where a Bazel action type refers to a Java class that implements an action (such as `CppCompileAction`). In particular, the "assembler actions" and "compiler actions" in the table below are `CppCompileAction`, while the link actions are `CppLinkAction`. ### Assembler actions
Constraint Description
requires: ['feature1', 'feature2'] Feature-level. The feature is supported only if the specified required features are enabled. For example, when a feature is only supported in certain build modes (features opt, dbg, or fastbuild). Multiple `requires` statements are satisfied all at once if any one of them is satisfied.
implies: 'feature' Feature-level. This feature implies the specified feature. For example, a module compile implies the need for module maps, which can be implemented by a repeated implies string in the feature where each of the strings names a specific feature. Enabling a feature also implicitly enables all features implied by it (that is, it functions recursively).

Also provides the ability to factor common subsets of functionality out of a set of features, such as the common parts of sanitizers. Implied features cannot be disabled. </td> </tr>

provides: 'feature' Feature-level. Indicates that this feature is one of several mutually exclusive alternate features. For example, all of the sanitizers could specify provides: "sanitizer".

This improves error handling by listing the alternatives if the user asks for two or more mutually exclusive features at once. </td> </tr>

with_feature: {feature: 'feature1', not_feature: 'feature2'] Flag set-level. A feature can specify multiple flag sets with multiple with_feature statements. When with_feature is specified, the flag set will only expand to the build command if all of the feature in the specified feature: set are enabled, and all the features specified in not_feature: set are disabled.
Action Description
preprocess-assemble Assemble with preprocessing. Typically for .S files.
assemble Assemble without preprocessing. Typically for .s files.
### Compiler actions
Action Description
cc-flags-make-variable Propagates CC_FLAGS to genrules.
c-compile Compile as C.
c++-compile Compile as C++.
c++-header-parsing Run the compiler's parser on a header file to ensure that the header is self-contained, as it will otherwise produce compilation errors. Applies only to toolchains that support modules.
### Link actions
Action Description
c++-link-dynamic-library Link a shared library containing all of its dependencies.
c++-link-nodeps-dynamic-library Link a shared library only containing cc_library sources.
c++-link-executable Link a final ready-to-run library.
### AR actions AR actions assemble object files into archive libraries (`.a` files) via `ar` and encode some semantics into the name.
Action Description
c++-link-static-library Create a static library (archive).
### LTO actions
Action Description
lto-backend ThinLTO action compiling bitcodes into native objects.
lto-index ThinLTO action generating global index.
## `CROSSTOOL` `action_config` A `CROSSTOOL` `action_config` is a proto message that describes a Bazel action by specifying the tool (binary) to invoke during the action and sets of flags, defined by features, that apply constraints to the action's execution. A `CROSSTOOL` action takes the following attributes:
Attribute Description
action_name The Bazel action to which this CROSSTOOL action corresponds. Bazel uses this attribute to discover per-action tool and execution requirements.
tool The executable to invoke. This can depend on a feature. A default must be provided.
flag_set A set of flags that applies to a group of actions. Same as for a feature.
env_set A set of environment constraints that applies to a group of actions. Same as for a feature.
A `CROSSTOOL` `action_config` can require and imply other features and action_configs as dictated by the [feature relationships](#feature-relationships) described earlier. This behavior is similar to that of a feature. The last two attributes are redundant against the corresponding attributes on features and are included because some Bazel actions require certain flags or environment variables and we want to avoid unnecessary `action_config`+`feature` pairs. Typically, sharing a single feature across multiple `action_config`s is preferred. You can not define more than one `CROSSTOOL` `action_config` with the same `action_name` within the same toolchain. This prevents ambiguity in tool paths and enforces the intention behind `action_config` - that an action's properties are clearly described in a single place in the toolchain. ### `tool` messages A `CROSSTOOL` `action_config` can specify a set of tools via `tool` messages. A `tool` message consists of the following fields:
Field Description
tool_path Path to the tool in question (relative to the CROSSTOOL file).
with_feature A set of features that must be enabled for this tool to apply.
For a given `CROSSTOOL` `action_config`, only a single `tool` message applies its tool path and execution requirements to the Bazel action. A tool is selected by sequentially parsing `tool` messages on an `action_config` until a tool with a `with_feature` set matching the feature configuration is found (see [Feature relationships](#feature-relationships) earlier in this document for more information). We recommend that you end your tool lists with a default tool that corresponds to an empty feature configuration. ### Example usage Features and `CROSSTOOL` actions can be used together to implement Bazel actions with diverse cross-platform semantics. For example, debug symbol generation on macOS requires generating symbols in the compile action, then invoking a specialized tool during the link action to create compressed dsym archive, and then decompressing that archive to produce the application bundle and `.plist` files consumable by Xcode. With Bazel, this process can instead be implemented as follows, with `unbundle-debuginfo` being a Bazel action: ``` toolchain { action_config { config_name: "c++-link-executable" action_name: "c++-link-executable" tool { with_feature { feature: "generate-debug-symbols" } tool_path: "toolchain/mac/ld-with-dsym-packaging" } tool { tool_path: "toolchain/mac/ld" } } feature { name: "generate-debug-symbols" flag_set { action: "c-compile" action: "c++-compile" flag_group { flag: "-g" } } implies: { feature: "unbundle-debuginfo" } } } ``` This same feature can be implemented entirely differently for Linux, which uses `fission`, or for Windows, which produces `.pdb` files. For example, the implementation for `fission`-based debug symbol generation might look as follows: ``` toolchain { action_config { name: "c++-compile" tool { tool_path: "toolchain/bin/gcc" } } feature { name: "generate-debug-symbols" requires { feature: "dbg" } flag_set { action: "c++-compile" flag_group { flag: "-gsplit-dwarf" } } flag_set { action: "c++-link-executable" flag_group { flag: "-Wl" flag: "--gdb-index" } } } } } ``` ### Flag groups `CROSSTOOL` allows you to bundle flags into groups that serve a specific purpose. You can specify a flag within the `CROSSTOOL` file using pre-defined variables within the flag value, which the compiler expands when adding the flag to the build command. For example: ``` flag_group { flag: "%{output_file_path} } ``` In this case, the contents of the flag will be replaced by the output file path of the action. Flag groups are expanded to the build command in the order in which they appear in the `CROSSTOOL` file, top-to-bottom, left-to-right. For flags that need to repeat with different values when added to the build command, the flag group can iterate variables of type `list`. For example, the variable `include_path` of type `list`: ``` flag_group { iterate_over: "include_paths" flag: "-I%{include_paths}" } ``` expands to `-I` for each path element in the `include_paths` list. All flags (or `flag_group`s) in the body of a flag group declaration are expanded as a unit. For example: ``` flag_group { iterate_over: "include_paths" flag: "-I" flag: "%{include_paths}" } ``` expands to `-I ` for each path element in the `include_paths` list. A variable can repeat multiple times. For example: ``` flag_group { iterate_over: "include_paths" flag: "-iprefix=%{include_paths}" flag: "-isystem=%{include_paths}" } ``` expands to: ``` -iprefix= -isystem= -iprefix= -isystem= ``` Variables can correspond to structures accessible using dot-notation. For example: ``` flag_group { flag: "-l%{libraries_to_link.name}" } ``` Structures can be nested and may also contain sequences. To prevent name clashes and to be explicit, you must specify the full path through the fields. For example: ``` flag_group { iterate_over: "libraries_to_link" flag_group { iterate_over: "libraries_to_link.shared_libraries" flag: "-l%{libraries_to_link.shared_libraries.name}" } } ``` ### Conditional expansion Flag groups support conditional expansion based on the presence of a particular variable or its field using the `expand_if_all_available`, `expand_if_none_available`, `expand_if_true`, `expand_if_false`, or `expand_if_equal` messages. For example: ``` flag_group { iterate_over: "libraries_to_link" flag_group { iterate_over: "libraries_to_link.shared_libraries" flag_group { expand_if_all_available: "libraries_to_link.shared_libraries.is_whole_archive" flag: "--whole_archive" } flag_group { flag: "-l%{libraries_to_link.shared_libraries.name}" } flag_group { expand_if_all_available: "libraries_to_link.shared_libraries.is_whole_archive" flag: "--no_whole_archive" } } } ``` **Note:** The `--whole_archive` and `--no_whole_archive` options are added to the build command only when a currently iterated library has an `is_whole_archive` field. ## `CROSSTOOL` reference This section provides a reference of build variables, features, and other information required to successfully configure `CROSSTOOL`. ### `CROSSTOOL` build variables The following is a reference of `CROSSTOOL` build variables. **Note:** `[action]` indicates the relevant action type.
Variable Description
source_file [compile] Source file to compile.
input_file [strip] Artifact to strip.
output_file [compile] Compilation output.
output_assembly_file [compile] Emitted assembly file. Applies only when the compile action emits assembly text, typically when using the --save_temps flag. The contents are the same as for output_file.
output_preprocess_file [compile] Preprocessed output. Applies only to compile actions that only preprocess the source files, typically when using the --save_temps flag. The contents are the same as for output_file.
includes [compile] Sequence of files the compiler must unconditionally include in the compiled source.
include_paths [compile] Sequence directories in which the compiler searches for headers included using #include<foo.h> and #include "foo.h".
quote_include_paths [compile] Sequence of -iquote includes - directories in which the compiler searches for headers included using #include<foo.h>.
system_include_paths [compile] Sequence of -isystem includes - directories in which the compiler searches for headers included using #include "foo.h".
dependency_file [compile] The .d dependency file generated by the compiler.
preprocessor_defines [compile] Sequence of defines, such as --DDEBUG.
pic [compile] Compiles the output as position-independent code.
gcov_gcno_file [compile] The gcov coverage file.
per_object_debug_info_file [compile] The per-object debug info (.dwp) file.
stripotps [strip] Sequence of stripopts.
legacy_compile_flags [compile] Sequence of flags from legacy CROSSTOOL fields such as compiler_flag, optional_compiler_flag, cxx_flag, and optional_cxx_flag.
user_compile_flags [compile] Sequence of flags from either the copt rule attribute or the --copt, --cxxopt, and --conlyopt flags.
unfiltered_compile_flags [compile] Sequence of flags from the unfiltered_cxx_flag legacy CROSSTOOL field or the unfiltered _compile_flags feature. These are not filtered by the nocopts rule attribute.
sysroot The sysroot.
runtime_library_search_directories [link] Entries in the linker runtime search path (usually set with the -rpath flag).
library_search_directories [link] Entries in the linker search path (usually set with the -L flag).
libraries_to_link [link] Flags providing files to link as inputs in the linker invocation.
def_file_path [link] Location of def file used on Windows with MSVC.
linker_param_file [link] Location of linker param file created by bazel to overcome command line length limit.
output_execpath [link] execpath of the output of the linker.
generate_interface_library [link] "yes"|"no" depending on whether interface library should be generated.
interface_library_builder_path [link] Path to the interface library builder tool.
interface_library_input_path [link] Input for the interface library ifso builder tool.
interface_library_output_path [link] Path where to generate interface library using the ifso builder tool.
legacy_link_flags [link] Linker flags coming from the legacy CROSSTOOL.
user_link_flags [link] Linker flags coming from the --linkopt or linkopts attribute.
symbol_counts_output [link] Path to which to write symbol counts.
linkstamp_paths [link] A build variable giving linkstamp paths.
force_pic [link] Presence of this variable indicates that PIC code should be generated.
strip_debug_symbols [link] Presence of this variable indicates that the debug symbols should be stripped.
is_cc_test [link] Truthy when current action is a cc_test linking action, false otherwise.
is_using_fission [link] Presence of this variable indicates that files were compiled with fission. Debug info is in .dwo files instead of .o files and the linker needs to know this.
### CROSSTOOL features The following is a reference of `CROSSTOOL` features and their activation conditions.
Feature Activation Condition
opt | dbg | fastbuild Enabled by default based on compilation mode.
static_linking_mode | dynamic_linking_mode Enabled by default based on linking mode.
random_seed Enabled by default.
dependency_file Enabled by default.
per_object_debug_info Enabled if the supports_fission attribute is set in the `CROSSTOOL` file and the current compilation mode is specified in the --fission flag.
pic Required if the target needs PIC objects for dynamic libraries. Enabled by default - the `pic` variable is present whenever PIC compilation is requested.