ci: accept QEMU paused prelaunch state and retire one-time cleanup

.gitea/workflows/macos-kvm-diagnostic.yaml

@@ -26,10 +26,6 @@ jobs:
         if: always()
         run: dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup --output artifacts/existing-kvm
 
-      - name: Remove only run 4165's unused inherited anonymous volume
-        if: always()
-        run: dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup-run4165-volume --output artifacts/existing-kvm/run4165-volume-cleanup
-
       - name: Preserve KVM diagnostic evidence
         if: always()
         uses: actions/upload-artifact@v3

docs/existing-kvm-diagnostic.md

@@ -8,10 +8,10 @@ The entry point is the .NET 10 file-based app `tools/ci/ExistingKvmDiagnostic.cs
 dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --help
 dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --run --output artifacts/existing-kvm
 dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup --output artifacts/existing-kvm
-dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup-run4165-volume --output artifacts/existing-kvm/run4165-volume-cleanup
+dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --validate-evidence /path/to/downloaded-kvm-artifact
 ```
 
-`--run` requires an empty output directory, the existing Docker CLI/daemon and Git. It records the source commit/helper SHA-256, Docker context/server identity, exact image metadata, commands, raw stdout/stderr, exit/state evidence, result and cleanup receipts. The workflow `.gitea/workflows/macos-kvm-diagnostic.yaml` is dispatched manually and always uploads these files. `--help` invokes no Docker command.
+`--run` requires an empty output directory, the existing Docker CLI/daemon and Git. It records the source commit/helper SHA-256, Docker context/server identity, exact image metadata, commands, raw stdout/stderr, exit/state evidence, result and cleanup receipts. The workflow `.gitea/workflows/macos-kvm-diagnostic.yaml` is dispatched manually and always uploads these files. `--help` invokes no Docker command. `--validate-evidence` reads `qemu-monitor.stdout.log`, `qemu-monitor.result.json` and `container-exited.stdout.log` through the same success parser as `--run`; it invokes no Docker command and writes no files. This mode checks only saved HMP protocol and clean exit interpretation, without requalifying source, ownership or the container boundary.
 
 The image is pinned to `qemux/qemu:7.50@sha256:e7f6fda52503a546fd649670ba46e4bc23dc6dcef275bc3fac48877fbbc430df`; if absent it may be pulled into the existing daemon's cache. One random-name/label container invokes `/usr/bin/qemu-system-x86_64` directly with KVM only, `-cpu host`, `-S`, no default devices, no display and HMP on stdin. It attaches no OS, disk or persistent volume and never continues the paused CPU. A read-only 4-KiB tmpfs at `/storage` (`ro,nosuid,nodev,noexec,size=4096,mode=0555`) replaces the image's inherited `VOLUME /storage`. Inspect must prove `Mounts=[]` and precisely that sole tmpfs entry; otherwise a specific mount/tmpfs error is retained before QEMU starts. The only host device mapping is `/dev/kvm:/dev/kvm:rw`. The filesystem is read-only, network is `none`, all Linux capabilities are dropped, and no-new-privileges is set. Limits are 0.5 CPU, 256 MiB container RAM/no additional swap, 32 PIDs, and 64 MiB paused guest RAM. There are no binds, ports, privileged mode, added capabilities or host networking.
 
@@ -19,8 +19,10 @@ The helper has a 95-second operation budget and a separate 20-second cleanup bud
 
 Actual run 4165 failed the original zero-mount guard because this pinned image declared `/storage` as a volume and Docker created an anonymous writable volume. QEMU never started: the saved container state was `created`, PID zero and `StartedAt` zero. Its exact container ID was `1753f95ef334244e7a1b393a839f218ea885363de7d5335eec53132d64627010`, owner token `43b7f4676c514f2a95c63c02577ac36e`, and anonymous volume `ef7daa62ef89a2ffb8aae50a9b7803f1d9b3075ee509aa3183f3e170f69ce595`. The original cleanup proved container removal; it did not prove volume removal.
 
-The temporary `--cleanup-run4165-volume` mode has a separate 20-second budget and accepts no target parameters. It records the frozen run/owner/volume target, artifact hashes, each command and its outcome under the supplied evidence directory. It requires the original Docker daemon ID `528941c8-73ac-49ff-8eb7-69113eb4a2a1`; absence of the old full container ID and random container name; exactly the recorded volume with local driver/scope and no options; and no container reference found by `docker ps --all --filter volume=...`. It uses `docker volume rm` without force, so Docker also refuses a reference introduced after the check. Already absent is an idempotent success only on the original daemon after old-container absence checks. A different daemon, attachment or missing evidence fails closed. No generalized orphan cleanup is provided. This temporary workflow step can be removed after actual removal/absence is verified. The source evidence is artifact ZIP SHA-256 `6745d90e8b81c867740405c99b4364cc165c47ebb165455052314459d5cd547b` and created-container inspect SHA-256 `7afdfc6c30c933bee2ef1d6c18ed011c8b2f709d1a5e88531928f9f40471c055`.
+Run 4167's temporary, frozen-target cleanup verified the original daemon ID `528941c8-73ac-49ff-8eb7-69113eb4a2a1`, absence of the old container ID/name, the exact local volume and absence of container references. Removal without force exited zero; a subsequent inspect returned `no such volume`, and the receipt recorded `outcome=removed` at `2026-10-03T18:03:06.5767095Z`. The one-time cleanup mode and workflow step have therefore been removed. No generalized orphan cleanup is provided. The original source evidence is run 4165 artifact ZIP SHA-256 `6745d90e8b81c867740405c99b4364cc165c47ebb165455052314459d5cd547b` and created-container inspect SHA-256 `7afdfc6c30c933bee2ef1d6c18ed011c8b2f709d1a5e88531928f9f40471c055`.
 
-`kvm_usable` requires QEMU to report both `kvm support: enabled` and `VM status: paused`, followed by clean monitor/container exit after `quit`. A device path alone is insufficient. Other receipts distinguish a Docker-reported missing daemon-host device, observed access denial, an unavailable QEMU KVM backend, an initialization error, and inconclusive evidence. These categories describe the observed output; they do not diagnose BIOS, nested virtualization, policy or hardware causes. All failures remain failed workflow runs with retained raw evidence. Even a usable result proves only this blank paused KVM initialization, not macOS boot, installation, native build or tests.
+`kvm_usable` requires QEMU to report the complete line `kvm support: enabled` and exactly `VM status: paused` or `VM status: paused (prelaunch)`, followed by clean monitor/container exit after `quit`. The monitor command must exit zero without timeout or error, and the container must be stopped with exit zero and no OOM. A device path alone is insufficient. Other receipts distinguish a Docker-reported missing daemon-host device, observed access denial, an unavailable QEMU KVM backend, an initialization error, and inconclusive evidence. These categories describe the observed output; they do not diagnose BIOS, nested virtualization, policy or hardware causes. All failures remain failed workflow runs with retained raw evidence. Even a usable result proves only this blank paused KVM initialization, not macOS boot, installation, native build or tests.
+
+Actual run 4167 successfully initialized KVM and reported `VM status: paused (prelaunch)` before clean exit. Its original workflow still failed because the parser accepted only `paused`. The read-only CLI evidence mode reproduced that behavioral failure against the actual artifact (exit one, `usable=false`); after the narrow state-parser correction, the identical artifact passed (exit zero, `usable=true`). This reinterprets retained evidence and does not claim that the original workflow result changed or that another VM was run.
 
 The CLI and monitor behavior follow the primary [QEMU command-line reference](https://www.qemu.org/docs/master/system/qemu-manpage.html) and [QEMU monitor reference](https://www.qemu.org/docs/master/system/monitor.html). Device/container options follow the [Docker create reference](https://docs.docker.com/reference/cli/docker/container/create/) and [Docker tmpfs reference](https://docs.docker.com/engine/storage/tmpfs/). Moby 28.3.3's [volume creation](https://raw.githubusercontent.com/moby/moby/v28.3.3/daemon/create_unix.go) and [mount detection](https://raw.githubusercontent.com/moby/moby/v28.3.3/container/container_unix.go) explicitly skip an inherited anonymous volume when that destination already has the tmpfs entry.

tools/ci/ExistingKvmDiagnostic.cs

@@ -19,15 +19,15 @@ static class ExistingKvmDiagnostic
     {
         if (args.SequenceEqual(new[] { "--help" }))
         {
-            Console.WriteLine("ExistingKvmDiagnostic.cs --run|--cleanup|--cleanup-run4165-volume --output DIRECTORY\nRequires .NET 10 and the existing Docker CLI/daemon. --help never calls Docker.\n--run tests only a paused, diskless QEMU with existing /dev/kvm; retains evidence and cleans up its own container. --cleanup retries that saved cleanup.\n--cleanup-run4165-volume removes only run 4165's frozen unused anonymous volume after daemon, old-container absence and reference checks.");
+            Console.WriteLine("ExistingKvmDiagnostic.cs --run|--cleanup --output DIRECTORY\nExistingKvmDiagnostic.cs --validate-evidence DIRECTORY\nRequires .NET 10; --run/--cleanup also require the existing Docker CLI/daemon. --help never calls Docker.\n--run tests only a paused, diskless QEMU with existing /dev/kvm; retains evidence and cleans up its own container. --cleanup retries that saved cleanup.\n--validate-evidence reads saved monitor/result/container evidence through the run's success parser; no Docker commands or file writes.");
             return 0;
         }
-        if (args.Length != 3 || args[0] is not ("--run" or "--cleanup" or "--cleanup-run4165-volume") || args[1] != "--output")
+        if (args.Length == 2 && args[0] == "--validate-evidence") return ValidateEvidence(Path.GetFullPath(args[1]));
-            throw new ArgumentException("Use --help or --run|--cleanup|--cleanup-run4165-volume --output DIRECTORY.");
+        if (args.Length != 3 || args[0] is not ("--run" or "--cleanup") || args[1] != "--output")
+            throw new ArgumentException("Use --help, --validate-evidence DIRECTORY or --run|--cleanup --output DIRECTORY.");
         var output = Path.GetFullPath(args[2]);
         Directory.CreateDirectory(output);
         if (args[0] == "--cleanup") return await Cleanup(output) ? 0 : 1;
-        if (args[0] == "--cleanup-run4165-volume") return await CleanupRun4165Volume(output) ? 0 : 1;
         if (Directory.EnumerateFileSystemEntries(output).Any()) throw new InvalidOperationException("Run output must be empty; existing receipts cannot be reused.");
 
         var started = DateTimeOffset.UtcNow;
@@ -69,11 +69,7 @@ static class ExistingKvmDiagnostic
                 var exited = await InspectOwned(output, owner, "container-exited", budget.Token);
                 var state = exited.GetProperty("State");
                 var text = Read(output, "qemu-monitor", "stdout");
-                var cleanExit = monitor.ExitCode == 0 && !monitor.TimedOut && !state.GetProperty("Running").GetBoolean()
+                status = KvmUsable(text, monitor, state) ? "kvm_usable" : ClassifyFailure(Read(output, "qemu-monitor", "stderr") + "\n" + state.GetProperty("Error").GetString());
-                    && state.GetProperty("ExitCode").GetInt32() == 0 && !state.GetProperty("OOMKilled").GetBoolean();
-                var enabled = Regex.IsMatch(text, @"(?m)^kvm support: enabled\r?$", RegexOptions.CultureInvariant);
-                var paused = Regex.IsMatch(text, @"(?m)^VM status: paused\r?$", RegexOptions.CultureInvariant);
-                status = cleanExit && enabled && paused ? "kvm_usable" : ClassifyFailure(Read(output, "qemu-monitor", "stderr") + "\n" + state.GetProperty("Error").GetString());
                 detail = status == "kvm_usable" ? "QEMU initialized KVM, reported enabled and paused, and exited successfully after quit. No guest CPU or OS was run."
                     : "KVM initialization or its enabled/paused/clean-exit proof did not pass; inspect raw monitor output and container state.";
             }
@@ -85,6 +81,31 @@ static class ExistingKvmDiagnostic
         return status == "kvm_usable" && cleaned ? 0 : 1;
     }
 
+    static bool KvmUsable(string text, CommandResult monitor, JsonElement state)
+    {
+        var cleanExit = monitor.ExitCode == 0 && !monitor.TimedOut && monitor.Error is null && !state.GetProperty("Running").GetBoolean()
+            && state.GetProperty("ExitCode").GetInt32() == 0 && !state.GetProperty("OOMKilled").GetBoolean();
+        var enabled = Regex.IsMatch(text, @"(?m)^kvm support: enabled\r?$", RegexOptions.CultureInvariant);
+        var paused = Regex.IsMatch(text, @"(?m)^VM status: paused(?: \(prelaunch\))?\r?$", RegexOptions.CultureInvariant);
+        return cleanExit && enabled && paused;
+    }
+
+    static int ValidateEvidence(string evidence)
+    {
+        try
+        {
+            using var monitorDocument = JsonDocument.Parse(File.ReadAllText(Path.Combine(evidence, "qemu-monitor.result.json")));
+            var result = monitorDocument.RootElement;
+            var monitor = new CommandResult(result.GetProperty("ExitCode").GetInt32(), result.GetProperty("TimedOut").GetBoolean(), result.GetProperty("Error").GetString());
+            using var exited = JsonDocument.Parse(Read(evidence, "container-exited", "stdout"));
+            if (exited.RootElement.GetArrayLength() != 1) throw new InvalidOperationException("Expected exactly one saved exited container.");
+            var usable = KvmUsable(Read(evidence, "qemu-monitor", "stdout"), monitor, exited.RootElement[0].GetProperty("State"));
+            Console.WriteLine(JsonSerializer.Serialize(new { usable, scope = "Saved HMP protocol and clean exit interpretation only; source, ownership and container boundary are not requalified." }));
+            return usable ? 0 : 1;
+        }
+        catch (Exception error) { Console.Error.WriteLine("Evidence interpretation failed: " + error.Message); return 1; }
+    }
+
     static string ClassifyFailure(string text)
     {
         if (text.Contains("/dev/kvm", StringComparison.Ordinal) && text.Contains("error gathering device information", StringComparison.OrdinalIgnoreCase)
@@ -177,72 +198,6 @@ static class ExistingKvmDiagnostic
         }
     }
 
-    static async Task<bool> CleanupRun4165Volume(string output)
-    {
-        // Frozen from actual run 4165: the exact owner-labelled container never started (created/PID 0/zero StartedAt).
-        // Its image's VOLUME instruction created this anonymous /storage volume before the mount guard failed.
-        const string daemonId = "528941c8-73ac-49ff-8eb7-69113eb4a2a1";
-        const string containerId = "1753f95ef334244e7a1b393a839f218ea885363de7d5335eec53132d64627010";
-        const string token = "43b7f4676c514f2a95c63c02577ac36e";
-        const string volumeName = "ef7daa62ef89a2ffb8aae50a9b7803f1d9b3075ee509aa3183f3e170f69ce595";
-        const string containerName = "meeting-assistant-kvm-" + token;
-        var prefix = "run4165-volume-" + Guid.NewGuid().ToString("N");
-        Save(output, prefix + ".target.json", new { runId = 4165, daemonId, containerId, containerName, token, volumeName,
-            archiveSha256 = "6745d90e8b81c867740405c99b4364cc165c47ebb165455052314459d5cd547b",
-            createdInspectSha256 = "7afdfc6c30c933bee2ef1d6c18ed011c8b2f709d1a5e88531928f9f40471c055",
-            ownerReceiptSha256 = "d96f15588412a5928ebe8a64b115764f113adaaaf6d164209fd87de8a4505572",
-            priorState = "created", priorPid = 0, priorStartedAt = "0001-01-01T00:00:00Z" });
-        using var budget = new CancellationTokenSource(TimeSpan.FromSeconds(20));
-        var success = false;
-        var outcome = "refused";
-        string? detail = null;
-        try
-        {
-            await Require(Command("docker", ["info", "--format", "{{.ID}}"], output, prefix + "-daemon", budget.Token));
-            if (Read(output, prefix + "-daemon", "stdout").Trim() != daemonId) throw new InvalidOperationException("Frozen run 4165 daemon ID differs; refusing volume cleanup.");
-            foreach (var selector in new[] { containerId, containerName })
-            {
-                var step = prefix + (selector == containerId ? "-old-id" : "-old-name");
-                var inspection = await Command("docker", ["inspect", "--type", "container", selector], output, step, budget.Token);
-                if (inspection.TimedOut || inspection.Error != null || inspection.ExitCode == 0 || !Read(output, step, "stderr").Contains("No such container", StringComparison.OrdinalIgnoreCase)
-                    || !Read(output, step, "stderr").Contains(selector, StringComparison.Ordinal)) throw new InvalidOperationException("Old run 4165 container absence was not proved for " + selector + ".");
-            }
-            var inspectStep = prefix + "-inspect";
-            var inspect = await Command("docker", ["volume", "inspect", volumeName], output, inspectStep, budget.Token);
-            if (VolumeAbsent(inspect, output, inspectStep, volumeName)) { success = true; outcome = "already-absent"; }
-            else
-            {
-                if (inspect.ExitCode != 0 || inspect.TimedOut || inspect.Error != null) throw new InvalidOperationException("Cannot inspect the exact run 4165 volume.");
-                using var document = JsonDocument.Parse(Read(output, inspectStep, "stdout"));
-                var values = document.RootElement;
-                if (values.GetArrayLength() != 1 || values[0].GetProperty("Name").GetString() != volumeName
-                    || values[0].GetProperty("Driver").GetString() != "local" || values[0].GetProperty("Scope").GetString() != "local"
-                    || !Empty(values[0].GetProperty("Options"))) throw new InvalidOperationException("Frozen volume name/local driver/scope/options boundary failed.");
-                await Require(Command("docker", ["ps", "--all", "--no-trunc", "--filter", "volume=" + volumeName, "--format", "{{.ID}}"], output, prefix + "-references", budget.Token));
-                if (Read(output, prefix + "-references", "stdout").Trim().Length != 0) throw new InvalidOperationException("A container references the frozen run 4165 volume; refusing removal.");
-                // No --force: Docker rejects an attachment made after the reference check, too.
-                await Require(Command("docker", ["volume", "rm", volumeName], output, prefix + "-remove", budget.Token));
-                var afterStep = prefix + "-after";
-                var after = await Command("docker", ["volume", "inspect", volumeName], output, afterStep, budget.Token);
-                if (!VolumeAbsent(after, output, afterStep, volumeName)) throw new InvalidOperationException("Volume absence after removal was not proved.");
-                success = true;
-                outcome = "removed";
-            }
-        }
-        catch (Exception error) { detail = error.Message; }
-        var receipt = new { runId = 4165, daemonId, volumeName, success, outcome, detail, finished = DateTimeOffset.UtcNow };
-        Save(output, prefix + ".receipt.json", receipt);
-        Console.WriteLine(JsonSerializer.Serialize(receipt));
-        return success;
-    }
-
-    static bool VolumeAbsent(CommandResult result, string output, string step, string name)
-    {
-        var error = Read(output, step, "stderr");
-        return result.ExitCode is not (null or 0) && !result.TimedOut && result.Error == null
-            && error.Contains("no such volume", StringComparison.OrdinalIgnoreCase) && error.Contains(name, StringComparison.Ordinal);
-    }
-
     static async Task<CommandResult> Command(string program, string[] arguments, string output, string step, CancellationToken cancellation, string? input = null)
     {
         var started = DateTimeOffset.UtcNow;