ci: preserve Apple Recovery daemon during read-only readiness probe

docs/macos-native-diagnostic.md

@@ -18,7 +18,9 @@ dotnet run --file tools/ci/MacOsNativeDiagnostic.cs -- --cleanup --output artifa
 
 Dependencies are the existing Linux/x64 runner, .NET 10 SDK, Git, Bash and Docker CLI/socket. The actual execution downloads public Dockur source, upstream build assets, Docker images and Apple Recovery; it does not use workstation credentials. The existing upstream Python UDIF patcher and the Bash hook are retained because they run inside the pinned Linux/macOS boot integration. Independent orchestration and validation remain C#.
 
-The helper clones Dockur commit `16a5b470cdd601bae8b05b02d748d7edfb36c12e`, verifies its exact Recovery patcher hash, and makes three narrowly verified source edits. The early `rc.cdrom.sh` hook only mounts the existing state share and returns. A same-length XML replacement makes the existing `com.apple.recoveryosd` LaunchDaemon execute `/bin/bash /Volumes/installstate/launch.sh` after boot tasks. The staged `launch.sh` is replaced entirely by the checked-in read-only readiness probe. All replacement counts are exact; an upstream mismatch fails. The two imported QEMU image digests are pinned and the final image/source/Recovery hashes are retained. Other upstream Dockerfile downloads are observed through the resulting image identity rather than asserted to be immutable.
+The helper clones Dockur commit `16a5b470cdd601bae8b05b02d748d7edfb36c12e`, verifies its exact Recovery patcher and staging-script hashes, and makes narrowly verified source edits. The early `rc.cdrom.sh` hook only mounts the existing state share and returns. A same-length XML replacement makes the existing `com.apple.recoveryosd` LaunchDaemon execute `/bin/bash /Volumes/installstate/launch.sh` after boot tasks. In this separate bootstrap A/B candidate, that file is the small `tools/ci/macos-native-bootstrap.sh` wrapper: it starts `/bin/bash /Volumes/installstate/readiness.sh` in the background, then `exec /usr/libexec/recoveryosd`. The original Apple executable therefore replaces the wrapper under the same launchd job/PID if exec succeeds. The staging copy and comparison transport the second script through the existing 9p share. All replacement counts are exact; an upstream mismatch fails. The two imported QEMU image digests are pinned and the final image/source/Recovery hashes are retained. Other upstream Dockerfile downloads are observed through the resulting image identity rather than asserted to be immutable.
+
+The experiment starts from commit `40281b57a5e80bbe699ae50d8927e629d09d05ad`. The readiness script is byte-identical to that baseline; validation enforces SHA-256 `4d428f594dac14eff64ed87b172c81ecf85ac91da8c5460cd6ec4b1d310800c3`. This changes only whether Apple's original Recovery daemon runs alongside the same probe. The probe now has that daemon as its parent and competes with its work for guest CPU/I/O. If the job restarts, the wrapper could start another read-only probe. Those lifecycle and scheduling effects are part of the experiment, not proof of a CoreFoundation or DiskArbitration dependency. No original Apple daemon implementation or such dependency is established by the retained plist.
 
 The VM uses TCG (`KVM=N`), slirp networking, a 4-GiB guest, two virtual CPUs and a sparse 64-GiB data disk. Its container has a 6-GiB memory/swap ceiling and a two-CPU limit. The existing Docker daemon must report at least two CPUs and 6 GiB total memory, the runner must have at least 5 GiB available memory, and the Docker filesystem must have at least 8 GiB free before Recovery downloads or boot. Native commands have 45-second watchdogs, except the single UID gate's targeted 180-second timing experiment. The ten-minute disk-readiness phase, 40-minute host deadline and 45-minute workflow limit remain unchanged.
 
@@ -28,11 +30,11 @@ The next probe runs mandatory architecture, root identity and platform gates bef
 
 After an initial platform failure the hook collects native launchd context and repeats the identical `sw_vers` command once, with the same 45-second limit. Native product version and all original identity/service/disk gates remain required. Optional process and CPU diagnostics run only after a gate fails. The upstream AVX2 warning reads host flags; run 4159 observed AVX2 in the actual guest. No host or guest CPU settings change.
 
-Actual run 4161 separated native wait from timer cleanup: architecture passed after 39 seconds, but the UID gate was terminated by its 45-second watchdog (51-second fork/exec/wait duration). Native ps commands passed after 34-42 seconds; output flushes took 289 and 76 seconds. The next diagnostic changes only the UID gate's watchdog to 180 seconds while retaining exit-zero/exact-root checks. This tests whether the measured short limit caused that failure; it does not establish a guest startup or service cause, and it does not qualify native CI. The earlier local 16-case harness qualified the previous 45-second timer/cancellation/output behavior, not this new timing experiment or the actual emulated guest.
+Actual run 4161 separated native wait from timer cleanup: architecture passed after 39 seconds, but the UID gate was terminated by its 45-second watchdog (51-second fork/exec/wait duration). Native ps commands passed after 34-42 seconds; output flushes took 289 and 76 seconds. Run 4163 used the targeted UID watchdog of 180 seconds but returned UID 0 with exit zero after 38 seconds, so it did not establish a need for that longer limit. Architecture passed after 46 seconds; the initial `sw_vers`, system-domain and DiskArbitration queries were terminated. The recovery-label query passed after 31 seconds but described the replacement Bash job, not Apple's executable. The identical warm `sw_vers` retry had no final exit before the 40-minute host deadline. This bootstrap A/B preserves all those limits and native pass conditions; no cause or native readiness is claimed. The earlier local 16-case harness qualified the previous 45-second timer/cancellation/output behavior, not this experiment or the actual emulated guest.
 
 ## Evidence and cleanup
 
-Evidence is written under the requested output directory: run identity and candidate commit, Docker/runner resources, exact source patch artifacts and hashes, image/container inspection, Recovery hash, native platform/process/launchctl/diskutil logs, machine-readable guest result, outcome and cleanup receipt. The workflow retains these as a seven-day artifact. Phase names and up to 512 KiB of the final native proof also appear in CI stdout, on success or failure, with the run token replaced; no environment or credential dump is printed. A Docker start/build exit zero is not a successful native result. A missing, stale, unsupported-platform, read-only or wrong-size guest receipt fails.
+Evidence is written under the requested output directory: run identity and candidate commit, Docker/runner resources, exact source patch artifacts and hashes, image/container inspection, Recovery hash, native platform/process/launchctl/diskutil logs, machine-readable guest result, outcome and cleanup receipt. `guest-launch.sh` records the wrapper and its original `recoveryosd` exec path; `guest-readiness.sh` records the separate token-bound probe; `image.sh.patched` records the copy/comparison seam. `source-hashes.json` distinguishes all three. These source artifacts alone do not prove that Apple's executable actually ran. The workflow retains these as a seven-day artifact. Phase names and up to 512 KiB of the final native proof also appear in CI stdout, on success or failure, with the run token replaced; no environment or credential dump is printed. A Docker start/build exit zero is not a successful native result. A missing, stale, unsupported-platform, read-only or wrong-size guest receipt fails.
 
 While Recovery readiness is pending, a minute heartbeat reports elapsed guest time and the container's running state. Before final cleanup, an optional ten-second capture rechecks the saved container ID/ownership label and uses the pinned image's existing Unix HMP socket, `nc.openbsd` and a five-second `timeout` to collect only [`info status` and `screendump`](https://www.qemu.org/docs/master/system/monitor.html), retaining the command transcript, exit codes and fresh bounded PPM screenshot. Capture failure is visible and never changes native readiness success.
 

tools/ci/MacOsNativeDiagnostic.cs

@@ -168,6 +168,8 @@ static class NativeDiagnostic
 
     static void ValidateContracts()
     {
+        if (Hash(File.ReadAllBytes(Path.Combine("tools", "ci", "macos-native-readiness.sh"))) != "4d428f594dac14eff64ed87b172c81ecf85ac91da8c5460cd6ec4b1d310800c3")
+            throw new InvalidOperationException("Bootstrap A/B requires the unchanged 40281b readiness probe and limits.");
         XDocument.Parse(DiagnosticDaemon);
         if (Encoding.UTF8.GetByteCount(DiagnosticDaemon) > Encoding.UTF8.GetByteCount(OriginalDaemon + "\n")) throw new InvalidOperationException("Daemon replacement exceeds original file.");
         var good = JsonSerializer.Serialize(new { token = "validation", success = true, osVersion = "14.6.1", architecture = "x86_64", uid = 0, disk = "/dev/disk1", diskBytes = GuestDiskBytes, readOnly = false, systemExit = 0, diskArbitrationExit = 0, recoveryExit = 0, diskListExit = 0 });
@@ -199,16 +201,26 @@ static class NativeDiagnostic
         var entry = ReplaceOnce(File.ReadAllText(entryPath), "set -Eeuo pipefail\n", "set -Eeuo pipefail\n\n# Diagnostic budget: inspect existing Docker storage before Recovery download/boot.\ndf -Pk /storage\nfree_kib=$(df -Pk /storage | awk 'NR==2 {print $4}')\n[[ \"$free_kib\" =~ ^[0-9]+$ ]] && (( free_kib >= 8 * 1024 * 1024 )) || { echo 'Existing Docker storage has less than the 8-GiB diagnostic budget.' >&2; exit 1; }\n");
         var hookPath = Path.Combine("tools", "ci", "macos-native-readiness.sh");
         var hook = ReplaceOnce(File.ReadAllText(hookPath), "@@PROOF_TOKEN@@", token);
-        foreach (var pair in new[] { ("recovery-patch.py", patch), ("Dockerfile.patched", dockerfile), ("container-entry.sh", entry), ("guest-launch.sh", hook), ("recoveryosd-original.plist", daemon), ("recoveryosd-diagnostic.plist", DiagnosticDaemon), ("early-bootstrap.sh", MountOnlyBootstrap) })
+        var wrapper = File.ReadAllText(Path.Combine("tools", "ci", "macos-native-bootstrap.sh"));
+        var imagePath = Path.Combine(source, "src", "image.sh");
+        var originalImage = File.ReadAllText(imagePath);
+        if (Hash(Encoding.UTF8.GetBytes(originalImage)) != "c08bf9436fb8b72ea82fdf0e677641ab2fc42a0a59e2cf0309c00df519884c5c") throw new InvalidOperationException("Pinned Recovery staging script hash mismatch.");
+        var image = ReplaceOnce(originalImage, "  if ! cp -f \"$IMAGE_TOOLS/recovery/launch.sh\" \"$script\"; then\n", "  if ! cp -f \"$IMAGE_TOOLS/recovery/launch.sh\" \"$script\" ||\n     ! cp -f \"$IMAGE_TOOLS/recovery/readiness.sh\" \"${script%/*}/readiness.sh\"; then\n");
+        image = ReplaceOnce(image, "  if ! cmp -s \"$IMAGE_TOOLS/recovery/launch.sh\" \"$script\" ||\n", "  if ! cmp -s \"$IMAGE_TOOLS/recovery/launch.sh\" \"$script\" ||\n     ! cmp -s \"$IMAGE_TOOLS/recovery/readiness.sh\" \"$state/readiness.sh\" ||\n");
+        foreach (var pair in new[] { ("recovery-patch.py", patch), ("Dockerfile.patched", dockerfile), ("container-entry.sh", entry), ("guest-launch.sh", wrapper), ("guest-readiness.sh", hook), ("image.sh.patched", image), ("recoveryosd-original.plist", daemon), ("recoveryosd-diagnostic.plist", DiagnosticDaemon), ("early-bootstrap.sh", MountOnlyBootstrap) })
             File.WriteAllText(Path.Combine(output, pair.Item1), pair.Item2, new UTF8Encoding(false));
-        Save(Path.Combine(output, "source-hashes.json"), Directory.GetFiles(output).Where(path => Path.GetFileName(path) is "recovery-patch.py" or "Dockerfile.patched" or "container-entry.sh" or "guest-launch.sh" or "recoveryosd-original.plist" or "recoveryosd-diagnostic.plist" or "early-bootstrap.sh").ToDictionary(path => Path.GetFileName(path)!, path => Hash(File.ReadAllBytes(path))));
+        Save(Path.Combine(output, "source-hashes.json"), Directory.GetFiles(output).Where(path => Path.GetFileName(path) is "recovery-patch.py" or "Dockerfile.patched" or "container-entry.sh" or "guest-launch.sh" or "guest-readiness.sh" or "image.sh.patched" or "recoveryosd-original.plist" or "recoveryosd-diagnostic.plist" or "early-bootstrap.sh").ToDictionary(path => Path.GetFileName(path)!, path => Hash(File.ReadAllBytes(path))));
         await Command("bash", ["-n", Path.Combine(output, "guest-launch.sh")], output, "guest-hook-syntax", cancellation);
+        await Command("bash", ["-n", Path.Combine(output, "guest-readiness.sh")], output, "guest-readiness-syntax", cancellation);
+        await Command("bash", ["-n", Path.Combine(output, "image.sh.patched")], output, "guest-staging-syntax", cancellation);
         await Command("bash", ["-n", Path.Combine(output, "container-entry.sh")], output, "entry-syntax", cancellation);
         if (!writeSource) return;
         File.WriteAllText(patchPath, patch, new UTF8Encoding(false));
         File.WriteAllText(dockerPath, dockerfile, new UTF8Encoding(false));
         File.WriteAllText(entryPath, entry, new UTF8Encoding(false));
-        File.WriteAllText(Path.Combine(source, "src/install/recovery/launch.sh"), hook, new UTF8Encoding(false));
+        File.WriteAllText(imagePath, image, new UTF8Encoding(false));
+        File.WriteAllText(Path.Combine(source, "src/install/recovery/launch.sh"), wrapper, new UTF8Encoding(false));
+        File.WriteAllText(Path.Combine(source, "src/install/recovery/readiness.sh"), hook, new UTF8Encoding(false));
     }
 
     static string ReplaceOnce(string text, string oldValue, string newValue) => ReplaceAllExact(text, oldValue, newValue, 1);

tools/ci/macos-native-bootstrap.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+# Apple Recovery has Bash before any SDK is installed. Preserve the original
+# daemon under its launchd label/PID while the unchanged read-only probe runs.
+/bin/bash /Volumes/installstate/readiness.sh &
+exec /usr/libexec/recoveryosd