retain bounded disk stack and resource pressure evidence

docs/macos-native-diagnostic.md

@@ -57,9 +57,11 @@ No new force/beta argument is needed for actual no-AVX2 CPUs. Baseline arguments
 
 The Apple wrapper is byte-identical to baseline: background `/Volumes/installstate/readiness.sh` then `exec /usr/libexec/recoveryosd` under the same launchd job/PID. Source evidence does not prove Apple's executable ran.
 
-The disk IPC continuation adds six explicitly marked observation blocks and limits disk enumeration to two attempts. Validation removes only those marked blocks, restores the former attempt condition and normalizes macOS 13 to 14 before requiring baseline SHA256 `4d428f594dac14eff64ed87b172c81ecf85ac91da8c5460cd6ec4b1d310800c3`. The receipt explicitly records these exclusions and the two-attempt limit. Architecture, UID, native service exits, disk size/writability/uniqueness, proof bounds, existing command watchdogs and native-wait/cleanup/flush metrics remain identical. Limits stay 45 seconds per required native command, 180 seconds for UID, ten minutes maximum disk readiness, 40 minutes host and 45 minutes workflow.
+The disk IPC continuation contains seven explicitly marked diagnostic blocks and limits disk enumeration to one attempt. Its disk query receives 120 seconds so the owned sample can finish while the query is still running. Validation removes only those marked blocks, including that command-budget exception, restores the former attempt condition and normalizes macOS 13 to 14 before requiring baseline SHA256 `4d428f594dac14eff64ed87b172c81ecf85ac91da8c5460cd6ec4b1d310800c3`. The receipt explicitly records the exclusions, attempt limit and sample/query budgets. Architecture, UID, native service exits, disk size/writability/uniqueness, proof bounds and native-wait/cleanup/flush metrics remain identical. All other required native commands retain 45 seconds, UID retains 180 seconds, and outer limits remain ten minutes maximum disk readiness, 40 minutes host and 45 minutes workflow.
 
-Run 4175 at `94a70b200508d3ba295124896d923fbb785d1658` reached macOS 13.6, x86_64 and UID 0 with KVM enabled; its nine `diskutil list physical` attempts timed out. This identifies a disk-readiness failure without proving whether SATA/IOMedia, service IPC or the probe context is responsible. Before the first attempt the continuation records bounded `launchctl print` output for `com.apple.diskarbitrationd` and `com.apple.diskmanagementd`, plus `ioreg -r -c IOMedia -l -w 0`. During that first owned diskutil process it captures process state, optionally runs `/usr/bin/sample <owned-child-pid> 3 10 -file <owned-output>`, then observes both service jobs again. The separate sample report is flushed into the proof alongside command output. Each optional observer command has an eight-second watchdog plus the existing two-second TERM/KILL grace. Missing sample tooling or an already completed diskutil is reported explicitly; nonzero observation exits are logged and cannot satisfy any native gate.
+Run 4175 at `94a70b200508d3ba295124896d923fbb785d1658` reached macOS 13.6, x86_64 and UID 0 with KVM enabled; its nine `diskutil list physical` attempts timed out. Run 4185 at `25989cf0eb7205f30ac0bb44eb279aa3b463f12c` proved the whole writable 64-GiB target as IOMedia `disk2`; it measured approximately 14 seconds for the final native process listing and 33 seconds for IOMedia. Both Apple disk jobs were running; DiskManagement's endpoint was still inactive. The former eight-second observer allowance was shorter than observed native startup, so its killed sample did not establish an IPC wait point. A missing target is ruled out for that run; service initialization, IPC or resource delays remain unresolved.
+
+Before the only disk attempt the continuation records bounded `launchctl print` output for `com.apple.diskarbitrationd` and `com.apple.diskmanagementd`, plus `ioreg -r -c IOMedia -l -w 0`. Its observer starts only `/usr/bin/sample <owned-diskutil-child-pid> 3 100 -file <owned-output>`, with no preceding process list or additional service query. Three seconds at a 100-millisecond interval reduces sampling overhead. The sample has 60 seconds for startup/reporting plus the existing two-second TERM/KILL grace; its separate report is flushed into the proof alongside command output. Missing sample tooling or an already completed diskutil is reported explicitly; nonzero observation exits are logged and cannot satisfy any native gate.
 
 The observer owns its command/timer PIDs and is stopped when the disk query completes or the probe is canceled. Its output enters the existing 512-KiB per-output and 4-MiB proof budgets. The hook only reads media/service/process state and writes its existing diagnostic files: it does not load, restart, erase or modify any service or disk. Bash remains necessary because Apple Recovery runs this hook before a .NET SDK is installed. The CPU, Recovery, QEMU, Apple wrapper and container profile are unchanged. These observations are prepared diagnostics, not a new successful guest or full native CI receipt.
 
@@ -69,4 +71,6 @@ Only device mapping: exactly `/dev/kvm:/dev/kvm:rw`. Inspection rejects other de
 
 Evidence retains run/profile identity, source/assets, EFI staging, container/resources, macOS 13 Recovery hash, native proof/result/outcome and cleanup. `[recovery-original]` logs the exact download's size/SHA256 before modifying it, including when patch failure later deletes the source. `guest-container-resources.last-success.stdout.log` and its timestamp/hash receipt preserve the last successful resource snapshot independently of a later failed stopped-container `docker exec`. Optional final Unix HMP capture includes `info kvm`, `info status` and a bounded PPM exported from `/tmp`; capture success passes no native gate.
 
+Optional 20-second resource snapshots before, during and after the guest probe retain cgroup CPU usage/throttling/pressure, memory events/pressure/statistics and host page-fault/swap counters. These observations test resource contention as a hypothesis; no resource failure is established by the existing guest timing alone. The large Recovery image hash is captured once after compatibility-profile staging is observed, with its successful receipt retained, instead of repeatedly hashing the image while collecting guest progress. Snapshot or hash observation failure cannot satisfy a native readiness gate.
+
 Both cleanup paths keep exact token/label/ID checks. `docker rm --force --volumes` removes only the owned container and anonymous volume, then its exact image; no unrelated objects or pruning. Evidence stays seven days. Full native CI still needs a subsequent actual installed remote guest to build/sign helpers and pass the full suite, including five native tests without skips.

tools/ci/MacOsNativeDiagnostic.cs

@@ -69,7 +69,7 @@ static class NativeDiagnostic
                 await PrepareSource(Path.GetFullPath(source), output, "validation", false, Option(args, "--cryptex-archive"), CancellationToken.None);
                 await ValidateResourceRetention(output);
                 await ValidateRecoveryPatch(output);
-                Save(Path.Combine(output, "validation.json"), new { success = true, profile = "kvm-host-ventura-cryptex", helperSha256 = Hash(File.ReadAllBytes(Path.Combine("tools", "ci", "MacOsNativeDiagnostic.cs"))), udifChecksumBindingSha256 = Hash(File.ReadAllBytes(Path.Combine("tools", "ci", "macos-native-udif-checksums.py"))), baselineReadinessNormalized = true, readinessDiagnosticBlocksExcluded = 6, diskReadinessAttemptLimit = 2, resultNegativeCases = 6, containerNegativeCases = 11, recoveryPositiveCases = 4, recoveryNegativeCases = 12, independentFixtureCrc32Readback = true, resourceSnapshotRetention = true, cryptexArchiveVerified = true, configurationAndStagingContractsVerified = true, templateIsoDownloaded = false, activeLiluRuntimeChecked = false, sourceModified = false, dockerExecuted = false, guestExecuted = false, completedUtc = DateTimeOffset.UtcNow });
+                Save(Path.Combine(output, "validation.json"), new { success = true, profile = "kvm-host-ventura-cryptex", helperSha256 = Hash(File.ReadAllBytes(Path.Combine("tools", "ci", "MacOsNativeDiagnostic.cs"))), udifChecksumBindingSha256 = Hash(File.ReadAllBytes(Path.Combine("tools", "ci", "macos-native-udif-checksums.py"))), baselineReadinessNormalized = true, readinessDiagnosticBlocksExcluded = 7, diskReadinessAttemptLimit = 1, diskCommandLimitSeconds = 120, diskSampleLimitSeconds = 60, diskSampleDurationSeconds = 3, diskSampleIntervalMilliseconds = 100, resultNegativeCases = 6, containerNegativeCases = 11, recoveryPositiveCases = 4, recoveryNegativeCases = 12, independentFixtureCrc32Readback = true, resourceSnapshotRetention = true, cryptexArchiveVerified = true, configurationAndStagingContractsVerified = true, templateIsoDownloaded = false, activeLiluRuntimeChecked = false, sourceModified = false, dockerExecuted = false, guestExecuted = false, completedUtc = DateTimeOffset.UtcNow });
             }
             Console.WriteLine("Source patch contracts and diagnostic result validation passed; no Docker or guest execution occurred.");
             return 0;
@@ -131,13 +131,21 @@ static class NativeDiagnostic
             await Command("docker", ["inspect", id], output, "container-created", deadline.Token);
             AssertContainer(File.ReadAllText(Path.Combine(output, "container-created.stdout.log")), token);
             await Command("docker", ["start", id], output, "docker-start", deadline.Token);
+            await CapturePressure(id, output, "before", deadline.Token);
             Console.WriteLine("The owned restricted KVM/host-CPU macOS 13 compatibility guest is starting. Success requires native macOS 13+/x86_64 and a writable 64-GiB disk; no installer will run. This is not a single-variable causal test.");
             var recoveryStarted = Stopwatch.StartNew();
             var heartbeat = Stopwatch.StartNew();
+            var diskPressureCaptured = false;
             while (true)
             {
                 deadline.Token.ThrowIfCancellationRequested();
                 await CaptureGuest(id, output, deadline.Token);
+                var proofPath = Path.Combine(output, "guest-proof.log");
+                if (!diskPressureCaptured && File.Exists(proofPath) && File.ReadAllText(proofPath).Contains("[proof-start] disks", StringComparison.Ordinal))
+                {
+                    diskPressureCaptured = true;
+                    await CapturePressure(id, output, "during", deadline.Token);
+                }
                 var resultPath = Path.Combine(output, "guest-result.json");
                 if (File.Exists(resultPath))
                 {
@@ -167,6 +175,7 @@ static class NativeDiagnostic
             Console.CancelKeyPress -= cancelHandler;
             using var captureDeadline = new CancellationTokenSource(TimeSpan.FromSeconds(45));
             try { await CaptureGuest(state.ContainerId ?? state.ContainerName, output, captureDeadline.Token, true, state.Token); } catch (Exception exception) { Console.Error.WriteLine("Final evidence capture: " + exception.Message); }
+            await CapturePressure(state.ContainerId ?? state.ContainerName, output, "after", captureDeadline.Token);
             try { PrintGuestProof(output, state.Token); } catch (Exception exception) { Console.Error.WriteLine("Native proof output: " + exception.Message); }
             var clean = await Cleanup(output);
             if (!clean) { outcome = "failed"; error = (error ?? "") + " Owned-resource cleanup failed; inspect cleanup evidence."; }
@@ -186,9 +195,9 @@ static class NativeDiagnostic
         var readiness = File.ReadAllText(Path.Combine("tools", "ci", "macos-native-readiness.sh"));
         var baseline = NormalizeReadinessDiagnostics(readiness);
         baseline = ReplaceOnce(baseline, "(( ${os_version%%.*} >= 13 ))", "(( ${os_version%%.*} >= 14 ))");
-        baseline = ReplaceOnce(baseline, "while (( attempt < 2 && SECONDS - readiness_start < 600 )); do", "while (( SECONDS - readiness_start < 600 )); do");
+        baseline = ReplaceOnce(baseline, "while (( attempt < 1 && SECONDS - readiness_start < 600 )); do", "while (( SECONDS - readiness_start < 600 )); do");
         if (Hash(Encoding.UTF8.GetBytes(baseline)) != "4d428f594dac14eff64ed87b172c81ecf85ac91da8c5460cd6ec4b1d310800c3")
-            throw new InvalidOperationException("Outside six explicit diagnostic blocks, the macOS minimum and two-attempt limit, baseline identity/service/disk gates and watchdogs must remain identical.");
+            throw new InvalidOperationException("Outside seven explicit diagnostic blocks, the macOS minimum and one-attempt limit, baseline identity/service/disk gates and watchdogs must remain identical.");
         if (Hash(File.ReadAllBytes(Path.Combine("tools", "ci", "macos-native-bootstrap.sh"))) != "94f069e116fdc7685a4d233cab6fa50df9f39274386bb82157674061e74fadb5")
             throw new InvalidOperationException("Compatibility profile must preserve the baseline Apple recoveryosd wrapper.");
         if (Hash(Encoding.UTF8.GetBytes(OriginalDaemon13)) != "af9d7f6c1948079bd4384d27b6882678d6fb4e338fcf6a8be8f84fceef174ad6") throw new InvalidOperationException("macOS 13 allowlist bytes differ from the independently read comparison plist.");
@@ -229,8 +238,8 @@ static class NativeDiagnostic
             source = source.Remove(from, to + end.Length - from);
             blocks++;
         }
-        if (blocks != 6 || source.Contains(end, StringComparison.Ordinal))
+        if (blocks != 7 || source.Contains(end, StringComparison.Ordinal))
-            throw new InvalidOperationException("Readiness must contain exactly six explicit disk IPC diagnostic blocks.");
+            throw new InvalidOperationException("Readiness must contain exactly seven explicit disk IPC diagnostic blocks.");
         return source;
     }
 
@@ -477,7 +486,40 @@ static class NativeDiagnostic
             var result = await Command("docker", ["exec", id, "cat", "/dev/shm/installstate/" + file.Item1], output, "capture-" + file.Item1, cancellation, requireSuccess: false);
             if (result.ExitCode == 0 && !string.IsNullOrWhiteSpace(result.Output)) File.WriteAllText(Path.Combine(output, file.Item2), result.Output);
         }
-        await Command("docker", ["exec", id, "sh", "-c", "printf '[qemu]\n'; qemu-system-x86_64 --version | head -n 1; printf '[Recovery hash]\n'; test ! -f /storage/13/setup.dmg || sha256sum /storage/13/setup.dmg; printf '[resources]\n'; df -Pk /storage; cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max 2>/dev/null || true"], output, "guest-container-resources", cancellation, requireSuccess: false, retainSuccessful: true);
+        // The immutable Recovery image is complete only after this staging marker.
+        // Hash it once instead of rereading 710 MB on every twenty-second poll.
+        if (logs.Output.Contains("[compatibility-profile] accelerator=kvm", StringComparison.Ordinal)
+            && !File.Exists(Path.Combine(output, "guest-container-resources.last-success.json")))
+            await Command("docker", ["exec", id, "sh", "-c", "printf '[qemu]\n'; qemu-system-x86_64 --version | head -n 1; printf '[Recovery hash]\n'; test -f /storage/13/setup.dmg && sha256sum /storage/13/setup.dmg || exit 1; printf '[resources]\n'; df -Pk /storage; cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max 2>/dev/null || true"], output, "guest-container-resources", cancellation, requireSuccess: false, retainSuccessful: true);
+    }
+
+    static async Task CapturePressure(string id, string output, string phase, CancellationToken cancellation)
+    {
+        using var snapshotDeadline = CancellationTokenSource.CreateLinkedTokenSource(cancellation);
+        snapshotDeadline.CancelAfter(TimeSpan.FromSeconds(20));
+        const string snapshot = """
+            printf '[snapshot UTC]\n'; date -u '+%Y-%m-%dT%H:%M:%SZ'
+            for path in /proc/meminfo /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io \
+                /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/cpu.pressure \
+                /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.peak \
+                /sys/fs/cgroup/memory.events /sys/fs/cgroup/memory.stat /sys/fs/cgroup/memory.pressure \
+                /sys/fs/cgroup/memory.swap.current; do
+                printf '\n[%s]\n' "$path"
+                if [ -r "$path" ]; then cat "$path"; else printf 'unavailable\n'; fi
+            done
+            printf '\n[host paging counters]\n'
+            awk '/^(pgmajfault|pswpin|pswpout) / {print}' /proc/vmstat
+            """;
+        try
+        {
+            await Command("docker", ["exec", id, "sh", "-c", snapshot], output, "capture-pressure-" + phase, snapshotDeadline.Token, requireSuccess: false, retainSuccessful: true);
+        }
+        catch (Exception exception)
+        {
+            // Optional evidence must not replace the guest outcome or prevent cleanup.
+            try { Save(Path.Combine(output, "capture-pressure-" + phase + ".unavailable.json"), new { phase, error = exception.Message, capturedUtc = DateTimeOffset.UtcNow }); }
+            catch (Exception evidenceError) { Console.Error.WriteLine("Optional pressure evidence: " + evidenceError.Message); }
+        }
     }
 
     static async Task CaptureMonitor(string id, string output, string token, CancellationToken cancellation)

tools/ci/macos-native-readiness.sh

@@ -126,8 +126,8 @@ observe_disk_query() {
     observation_child=$!
     (
       trap 'exit 0' TERM INT
-      IFS= read -r -t 8 -u 9 unused || :
+      IFS= read -r -t 60 -u 9 unused || :
-      printf '[disk-observation-timeout] %s child=%s limit=8s\n' "$name" "$observation_child" >> "$output"
+      printf '[disk-observation-timeout] %s child=%s limit=60s\n' "$name" "$observation_child" >> "$output"
       kill -TERM "$observation_child" 2>/dev/null || :
       IFS= read -r -t 2 -u 9 unused || :
       kill -KILL "$observation_child" 2>/dev/null || :
@@ -141,18 +141,15 @@ observe_disk_query() {
   }
   trap cancel_observation TERM INT
   printf '[disk-observation] owned-diskutil-child=%s parent-shell=%s\n' "$disk_process" "$$" >> "$output"
-  observe_command processes /bin/ps -axo pid,ppid,state,comm
   if [ -x /usr/bin/sample ]; then
     if kill -0 "$disk_process" 2>/dev/null; then
-      observe_command diskutil-sample /usr/bin/sample "$disk_process" 3 10 -file "$sample_output"
+      observe_command diskutil-sample /usr/bin/sample "$disk_process" 3 100 -file "$sample_output"
     else
       printf '[disk-observation-unavailable] diskutil already exited before sample\n' >> "$output"
     fi
   else
     printf '[disk-observation-unavailable] /usr/bin/sample is unavailable\n' >> "$output"
   fi
-  observe_command arbitration /bin/launchctl print system/com.apple.diskarbitrationd
-  observe_command management /bin/launchctl print system/com.apple.diskmanagementd
 }
 
 stop_disk_observation() {
@@ -186,6 +183,9 @@ run_command() {
   # Run 4161: even native uname/ps startup took 34-42s under TCG.
   # Isolate only the failed UID gate; every other watchdog remains unchanged.
   [[ "$name" != uid ]] || command_limit=180
+# BEGIN disk IPC diagnostic
+  if [[ "$name" == disks && "${attempt:-0}" == 1 ]]; then command_limit=120; fi
+# END disk IPC diagnostic
   LAST_OUTPUT="/tmp/native-diagnostic-$name.out"
   printf '\n[proof-command] %s:' "$name" >&3
   printf ' %s' "$@" >&3
@@ -298,7 +298,7 @@ flush_outputs || finish false diagnostic_log_budget_exceeded
 # Bound readiness independently of the host's 40-minute overall deadline.
 readiness_start=$SECONDS
 attempt=0
-while (( attempt < 2 && SECONDS - readiness_start < 600 )); do
+while (( attempt < 1 && SECONDS - readiness_start < 600 )); do
   attempt=$((attempt + 1))
   printf '\n[readiness-attempt] %s\n' "$attempt" >&3
   run_command disks /usr/sbin/diskutil list physical