compilers survive garbage collection and get modules

CLAUDE.md

@@ -254,7 +254,7 @@ Writes all files to the build path. Key responsibilities:
 JSON schema validation using `jsonschema`. The `validator()` function extends the validator to auto-inject `default` values from schemas into parsed instances, so downstream code can rely on optional fields always being present. `check_config_version` enforces version 3 and gives a clear error for v1/v2 recipes pointing to the `releases/v6` branch.
 
 ### `etc/compiler-config.py`
-Replaces `spack compiler find`. Run as `spack -e BUILD_ROOT python compiler-config.py OUTPUT_YAML COMPILER...`. Uses `spack.store.STORE.db.query()` to find installed compiler packages, walks the install prefix to locate binaries, and writes (or merges into) a `packages.yaml` with `extra_attributes.compilers` entries. This is how compilers become available to downstream Spack users without a `compilers.yaml`.
+Replaces `spack compiler find`. Run as `spack -e BUILD_ROOT python compiler-config.py OUTPUT_YAML COMPILER...`. Uses `spack.store.STORE.db.query()` (with no `explicit=` filter, since a compiler exposed by a view is installed explicitly) to find installed compiler packages, walks the install prefix to locate binaries, and writes (or merges into) a `packages.yaml` with `extra_attributes.compilers` entries. This is how compilers become available to downstream Spack users without a `compilers.yaml`.
 
 ### `etc/envvars.py`
 A standalone CLI tool (copied into the build directory) with two subcommands:
@@ -294,11 +294,13 @@ The build runs inside a bwrap sandbox (`bwrap-mutable-root.sh`) that:
 
 The unified `spack.yaml` uses Spack 1.2 spec groups to express the build order and per-group concretizer settings. Structure:
 
-- **gcc group**: `explicit: false`, override sets static-library variants for gcc's dependencies (mpc, gmp, mpfr, zstd, zlib)
-- **nvhpc/llvm/llvm-amdgpu/intel-oneapi/intel-classic groups**: `explicit: false`, `needs: [gcc]`, `reuse: false`
+- **gcc group**: `explicit` follows `Recipe.provided_compilers`, override sets static-library variants for gcc's dependencies (mpc, gmp, mpfr, zstd, zlib)
+- **nvhpc/llvm/llvm-amdgpu/intel-oneapi/intel-classic groups**: `explicit` follows `Recipe.provided_compilers`, `needs: [gcc]`, `reuse: false`
 - **uenv_tools group**: `explicit: false`, `needs: [gcc]`, installs `squashfs`
 - **user environment groups**: `needs: [compiler list]`, override sets `concretizer.unify`, `concretizer.duplicates.strategy`, `packages.all.prefer`, `packages.all.variants`, and `packages.mpi.require` per-environment
 
+A compiler group is `explicit: true` when some view exposes it (`uenv:add_compilers`, the default). Explicit groups are gc roots, so `cleanup: runtime` does not collect them, and Spack generates their module files. A compiler that no view exposes was built only to compile something else: it stays `explicit: false` and remains collectable.
+
 Per-group `override:` blocks are pushed as the highest-priority config scope during that group's concretization, so `unify` and `duplicates.strategy` settings are truly per-group.
 
 ## Build Cache
@@ -339,6 +341,7 @@ The test coverage is limited — the schema validators and their default-injecti
 - **`mirrors.yaml` in recipes is unsupported**: use `--cache` CLI flag instead.
 - **`default-view` must exist**: if set in `config.yaml`, the named view must be defined in `environments.yaml` (or be `modules`/`spack`).
 - **`prefer` is auto-set**: if `null` in the recipe, Stackinator generates a `prefer` constraint using Spack's `%[when=...]` syntax to pin the default compiler.
-- **`uenv_tools` is reserved**: a spec group named `uenv_tools` is hardcoded in the spack.yaml template to install `squashfs`. Recipe authors must not use this environment name.
+- **`uenv_tools` is reserved**: a spec group named `uenv_tools` is hardcoded in the spack.yaml template to install `squashfs`. Recipe authors must not use this environment name. It is `explicit: false`, so `cleanup: runtime` collects it — a recipe that needs `squashfs` in its image must list it in its own `specs`.
+- **Views set `CC`/`CXX`/`FC`/`F77`**: `envvars.py` sets them from the first entry of the environment's `compiler` list. Spack's `activate.sh` only sets them for a compiler it installed, so an external (`gcc: version: "system"`) would otherwise leave them unset and cmake would fall back to `/usr/bin/cc`.
 - **`packages` field in environments.yaml is ignored**: in v3 recipes, the `packages` list (formerly used to drive `spack external find`) is silently ignored with a warning. Add external packages to `packages.yaml` instead.
 - **compiler-config.py must run inside spack python**: it imports `spack.store` and must be invoked as `spack -e BUILD_ROOT python compiler-config.py` to access the correct spack DB.

stackinator/etc/compiler-config.py

@@ -53,7 +53,9 @@ def build_compiler_packages(compiler_names):
 
     packages = {}
     for name in compiler_names:
-        specs = list(spack.store.STORE.db.query(name, explicit=False))
+        # no explicit= filter: a compiler exposed by a view is installed explicitly.
+        # externals are dropped below, which is what actually narrows this query.
+        specs = list(spack.store.STORE.db.query(name))
         if not specs:
             print(f"  compiler-config: no installed specs found for '{name}'", file=sys.stderr)
             continue

stackinator/etc/envvars.py

@@ -512,6 +512,7 @@ def view_impl(args):
     # (nvhpc, llvm, etc.) the binary names are already canonical, so we fall
     # back to os.path.basename which gives the right name (nvc, clang, etc.).
     _GCC_ROLE_CANONICAL = {"c": "gcc", "cxx": "g++", "fortran": "gfortran"}
+    _ROLE_ENVVARS = {"c": ["CC"], "cxx": ["CXX"], "fortran": ["FC", "F77"]}
 
     if args.compilers is not None:
         if not os.path.isfile(args.compilers):
@@ -523,9 +524,13 @@ def view_impl(args):
 
         # Restrict the symlinked compilers to those wired to this view's
         # environment (environments.yaml: compiler). When None, link them all.
-        allowed = None
-        if args.compiler_names:
-            allowed = {n for n in args.compiler_names.split(",") if n}
+        # Order matters: the first is the environment's default compiler, and
+        # the one CC/CXX/FC/F77 point at below.
+        ordered_names = [n for n in (args.compiler_names or "").split(",") if n]
+        allowed = set(ordered_names) if args.compiler_names else None
+
+        # package name -> {role: symlink name}, for setting CC/CXX/FC/F77 below
+        linked = {}
 
         for pkg_name, pkg_data in data["packages"].items():
             if allowed is not None and pkg_name not in allowed:
@@ -549,6 +554,29 @@ def view_impl(args):
                     if os.path.exists(dst):
                         os.remove(dst)
                     os.symlink(src, dst)
+                    linked.setdefault(pkg_name, {})[role] = link_name
+
+        # Set CC/CXX/FC/F77 ourselves. Spack's activate.sh only sets them for a
+        # compiler it installed, so an external one ('gcc: version: "system"')
+        # leaves them unset - and cmake then falls back to /usr/bin/cc, which is
+        # not what the stack was built with. Recipe env_vars are merged after
+        # this in meta_impl, so a recipe can still override.
+        for pkg_name in ordered_names:
+            roles = linked.get(pkg_name)
+            if not roles:
+                continue
+            for role, link_name in roles.items():
+                value = os.path.join(bin_path, link_name)
+                for var in _ROLE_ENVVARS.get(role, []):
+                    print(f"compiler env: {var}={value}")
+                    envvars.set_scalar(var, value)
+            break
+    else:
+        # No compilers in this view (uenv:add_compilers is off), but spack's
+        # activate.sh still points CC/CXX/FC/F77 at paths inside it, which
+        # nothing creates. Drop them: a dangling CC breaks cmake outright.
+        for var in (v for vs in _ROLE_ENVVARS.values() for v in vs):
+            envvars.scalars.pop(var, None)
 
     if args.prefix_paths:
         for p in args.prefix_paths.split(","):

stackinator/recipe.py

@@ -64,6 +64,7 @@ def __init__(self, args):
 
         # optional modules.yaml file
         self.modules = None
+        self._module_types = []
         modules_path = self.path / "modules.yaml"
         self._logger.debug(f"opening {modules_path}")
         if modules_path.is_file():
@@ -107,6 +108,9 @@ def __init__(self, args):
                         )
                     roots[module_type] = (self.mount / "modules").as_posix()
 
+                # the module rules depend on the environments, which are read below
+                self._module_types = module_types
+
         self._logger.debug("creating packages")
 
         # load recipe/packages.yaml -> recipe_packages (if it exists)
@@ -168,6 +172,9 @@ def __init__(self, args):
             self._check_environments_v3(raw)
             self.generate_environment_specs(raw)
 
+        if self.with_modules:
+            self._apply_module_rules()
+
         # check that the default view exists (if one has been set)
         self._default_view = self.config["default-view"]
         if self._default_view is not None:
@@ -390,6 +397,68 @@ def compiler_names(self):
         """
         return [c.get("package", name) for name, c in self.compilers.items() if not c.get("system", False)]
 
+    def _is_system(self, compiler):
+        return self.compilers.get(compiler, {}).get("system", False)
+
+    @property
+    def exposed_compilers(self):
+        """Recipe keys of the compilers that a view exposes to users.
+
+        A compiler no view exposes was built only to compile something else: it
+        gets no module file, and `spack gc` may collect it.
+        """
+        exposed = set()
+        for config in self.environments.values():
+            if any(v["extra"]["add_compilers"] for v in config["views"]):
+                exposed.update(config["compiler"])
+        return exposed
+
+    @property
+    def provided_compilers(self):
+        """Exposed compilers whose spec group is marked explicit.
+
+        Explicit groups are gc roots, and spack generates their module files.
+        """
+        return {c for c in self.exposed_compilers if not self._is_system(c)}
+
+    @property
+    def compiler_module_specs(self):
+        """Exposed compilers that need an "include" entry to get a module file.
+
+        Only externals: they have no spec group, so unlike provided_compilers
+        they cannot be made explicit.
+        """
+        return sorted(self.compilers[c].get("package", c) for c in self.exposed_compilers if self._is_system(c))
+
+    def _apply_module_rules(self):
+        """Add the include/exclude entries that stackinator owns in modules.yaml.
+
+        Merges into whatever the recipe set, rather than replacing it.
+        """
+        defaults = self.modules["modules"]["default"]
+
+        # "include" wins over "exclude_implicits", which would otherwise hide an
+        # external compiler.
+        include = self.compiler_module_specs
+
+        # squashfs is in the uenv_tools group, which is explicit so that `spack gc`
+        # cannot break the image step. That also makes spack generate a module for
+        # it, which is internal noise - unless the recipe asked for squashfs itself.
+        exclude = []
+        if not any(s.split()[0] == "squashfs" for c in self.environments.values() for s in c["specs"]):
+            exclude = ["squashfs"]
+
+        for module_type in self._module_types:
+            type_config = defaults.setdefault(module_type, {})
+            for key, values in (("include", include), ("exclude", exclude)):
+                current = type_config.setdefault(key, [])
+                current.extend(v for v in values if v not in current)
+
+        if include:
+            self._logger.debug(f"modules: including {', '.join(include)}")
+        if exclude:
+            self._logger.debug(f"modules: excluding {', '.join(exclude)}")
+
     # creates the self.environments field that describes the full specifications
     # for all of the environments sets, grouped in environments, from the raw
     # environments.yaml input.
@@ -576,6 +645,10 @@ def spack_yaml(self):
 
         has_views = any(env_cfg["views"] for env_cfg in self.environments.values())
 
+        provided = self.provided_compilers
+        for name, c in self.compilers.items():
+            c["provided"] = name in provided
+
         template = env.get_template("spack.yaml")
         return template.render(
             compilers=self.compilers,

stackinator/templates/spack.yaml

@@ -7,7 +7,7 @@ spack:
 {% if not system_gcc %}
   # gcc is built first using the system compiler.
   - group: gcc
-    explicit: false
+    explicit: {{ compilers.gcc.provided | string | lower }}
     specs:
 {% for spec in compilers.gcc.specs %}
     - '{{ spec }}'
@@ -23,7 +23,7 @@ spack:
 {% endif %}
 {% if compilers.nvhpc %}
   - group: nvhpc
-    explicit: false
+    explicit: {{ compilers.nvhpc.provided | string | lower }}
 {% if not system_gcc %}
     needs: [gcc]
 {% endif %}
@@ -39,7 +39,7 @@ spack:
 {% endif %}
 {% if compilers.llvm %}
   - group: llvm
-    explicit: false
+    explicit: {{ compilers.llvm.provided | string | lower }}
 {% if not system_gcc %}
     needs: [gcc]
 {% endif %}
@@ -55,7 +55,7 @@ spack:
 {% endif %}
 {% if compilers.get('llvm-amdgpu') %}
   - group: llvm-amdgpu
-    explicit: false
+    explicit: {{ compilers['llvm-amdgpu'].provided | string | lower }}
 {% if not system_gcc %}
     needs: [gcc]
 {% endif %}
@@ -71,7 +71,7 @@ spack:
 {% endif %}
 {% if compilers.get('intel-oneapi') %}
   - group: intel-oneapi
-    explicit: false
+    explicit: {{ compilers['intel-oneapi'].provided | string | lower }}
 {% if not system_gcc %}
     needs: [gcc]
 {% endif %}
@@ -87,7 +87,7 @@ spack:
 {% endif %}
 {% if compilers.get('intel-classic') %}
   - group: intel-classic
-    explicit: false
+    explicit: {{ compilers['intel-classic'].provided | string | lower }}
 {% if not system_gcc %}
     needs: [gcc]
 {% endif %}
@@ -103,9 +103,10 @@ spack:
 
 {% endif %}
   # ---- Internal tools group ----
-  # squashfs is required to create the final squashfs image.
+  # squashfs is required to create the final squashfs image. explicit, so that
+  # `spack gc` cannot collect it before the image step runs.
   - group: uenv_tools
-    explicit: false
+    explicit: true
 {% if not system_gcc %}
     needs: [gcc]
 {% endif %}

unittests/test_recipe.py

@@ -91,3 +91,59 @@ def test_generate_compiler_specs_system_gcc():
     recipe.generate_compiler_specs({"gcc": {"version": "system", "spec": "+nvptx"}})
     assert recipe.compilers["gcc"] == {"system": True}
     assert recipe.use_system_gcc
+
+
+def _env(compilers, add_compilers):
+    return {"compiler": compilers, "views": [{"extra": {"add_compilers": add_compilers}}], "specs": []}
+
+
+def test_compiler_module_specs_built_compiler():
+    """A built compiler is made explicit instead, so it needs no include entry."""
+    recipe = make_recipe()
+    recipe.generate_compiler_specs({"gcc": {"version": "14", "spec": None}})
+    recipe.environments = {"a": _env(["gcc"], True)}
+    assert recipe.compiler_module_specs == []
+
+
+def test_compiler_module_specs_system_gcc():
+    """The system gcc is an external, so it needs an include entry to get a module."""
+    recipe = make_recipe()
+    recipe.generate_compiler_specs({"gcc": {"version": "system", "spec": None}})
+    recipe.environments = {"a": _env(["gcc"], True)}
+    assert recipe.compiler_module_specs == ["gcc"]
+
+
+def test_compiler_module_specs_not_exposed():
+    """A compiler no view exposes gets no module, system or not."""
+    recipe = make_recipe()
+    recipe.generate_compiler_specs({"gcc": {"version": "system", "spec": None}})
+    recipe.environments = {"a": _env(["gcc"], False)}
+    assert recipe.compiler_module_specs == []
+    assert recipe.provided_compilers == set()
+
+
+def test_provided_compilers():
+    """A compiler is provided when a view with add_compilers exposes it."""
+    recipe = make_recipe()
+    recipe.generate_compiler_specs({"gcc": {"version": "14", "spec": None}, "nvhpc": {"version": "25.1", "spec": None}})
+    recipe.environments = {"a": _env(["gcc"], True), "b": _env(["nvhpc"], False)}
+    assert recipe.provided_compilers == {"gcc"}
+
+
+def test_provided_compilers_excludes_system():
+    """A system compiler is an external, so there is no group to mark explicit."""
+    recipe = make_recipe()
+    recipe.generate_compiler_specs(
+        {"gcc": {"version": "system", "spec": None}, "llvm": {"version": "20", "spec": None}}
+    )
+    recipe.environments = {"a": _env(["llvm", "gcc"], True)}
+    assert recipe.provided_compilers == {"llvm"}
+    assert recipe.exposed_compilers == {"llvm", "gcc"}
+
+
+def test_provided_compilers_no_views():
+    """No view exposing compilers means none are provided."""
+    recipe = make_recipe()
+    recipe.generate_compiler_specs({"gcc": {"version": "14", "spec": None}})
+    recipe.environments = {"a": {"compiler": ["gcc"], "views": [], "specs": []}}
+    assert recipe.provided_compilers == set()