Avoiding rebuilds for Python packages on only slightly different interpreters

In Nix, Python packages are typically managed by the Nix package manager as seperate units with each package referencing the Python interpreter version it has been built for.

While this ensures that packages are relatively certain to be compatible with the interpreter they have been built for, it also means that using a package with an even slightly different interpreter built will rebuild the package, even if it is a pure source package with no binary modules. While we’d ideally only rebuild only those packages with binary modules while using pure-Python packages as-is, the approach here shows how to swap out the interpreter for a Python virtual environment built by Nixpkg tools wholesale:

let
    # Helper: Rebuild Python environment to target a different interpreter
    #
    # sourceEnv: Original Python environment to reference
    # targetPython: Override interpreter to use instead with package set
    overridePythonEnvInterpreter = sourceEnv: targetPython: pkgs.stdenv.mkDerivation {
        name = "patched-env";
        nativeBuildInputs = with pkgs; [
            makeBinaryWrapper
        ];

        dontUnpack = true;
        dontBuild = true;

        installPhase = ''
            runHook preInstall

            mkdir -p "$out/bin" "$out/lib"
            makeBinaryWrapper "${targetPython}/bin/python" "$out/bin/python" --inherit-argv0 --set PYTHONUSERSITE true
            ln -s "${targetPython}/include" "$out/include";
            ln -s "${targetPython}/share"   "$out/share";
            for libpath in "${targetPython}"/lib/*;
            do
                if [[ "''${libpath##*/}" = python3.* ]];
                then
                    mkdir "$out/lib/''${libpath##*/}"
                    for pylibpath in "$libpath"/*;
                    do
                        if [[ "''${pylibpath##*/}" = "site-packages" ]];
                        then
                            ln -s "${sourceEnv}"/lib/python3.*/site-packages "$out/lib/''${libpath##*/}/site-packages"
                        else
                            ln -s "$pylibpath" "$out/lib/''${libpath##*/}/''${pylibpath##*/}"
                        fi
                    done
                else
                    ln -s "$libpath" "$out/lib/''${libpath##*/}"
                fi
            done

            runHook postInstall
        '';
    };

    # Use helper to use regular Python 3.13 packages (as built by Hydra) with
    # the free-threading interpreter without rebuilding every package
    pythonEnv = overridePythonEnvInterpreter (pkgs.python313.withPackages(pp: with pp; [
        # … list of Python packages …
    ]) pkgs.python313FreeThreading;
in {
    # … use `pythonEnv` here …
}

What the overridePythonEnvInterpreter function here does is iterate over the of the target Python interpreter installation, placing symlinks to it were appropriate, while creating the “lib/pythonX.Y/” directory structure and then crucially having its “site-packages” directory instead point to the “site-package” of that the Nixpkg tools have so kindly already assembled for us. (The “site-packages” directory is also were tools like pip install thrid-party packages on user-managed Python installations.)

A version that rebuilds the binary packages would likely start with one Python environment containing the entire package set from the offical Hydra set (as above), and one “minimal set” containing only the packages needing to be rebuilt for the target Python interpreter. The process would be like above, but instead of directly symlinking the “site-packages” to the target environment, it would instead also be created as a regular directory, with each directory inside "${sourceEnv}"/lib/python3.*/site-packages being first checked on whether it also exists in the minimal override side and then symlinked from the override set if it exists there and symlinked from the original set otherwise.