ci: replace inherited probe volume with read-only tmpfs

.gitea/workflows/macos-kvm-diagnostic.yaml

@@ -26,6 +26,10 @@ jobs:
         if: always()
         run: dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup --output artifacts/existing-kvm
 
+      - name: Remove only run 4165's unused inherited anonymous volume
+        if: always()
+        run: dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup-run4165-volume --output artifacts/existing-kvm/run4165-volume-cleanup
+
       - name: Preserve KVM diagnostic evidence
         if: always()
         uses: actions/upload-artifact@v3

docs/existing-kvm-diagnostic.md

@@ -8,14 +8,19 @@ The entry point is the .NET 10 file-based app `tools/ci/ExistingKvmDiagnostic.cs
 dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --help
 dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --run --output artifacts/existing-kvm
 dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup --output artifacts/existing-kvm
+dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup-run4165-volume --output artifacts/existing-kvm/run4165-volume-cleanup
 ```
 
 `--run` requires an empty output directory, the existing Docker CLI/daemon and Git. It records the source commit/helper SHA-256, Docker context/server identity, exact image metadata, commands, raw stdout/stderr, exit/state evidence, result and cleanup receipts. The workflow `.gitea/workflows/macos-kvm-diagnostic.yaml` is dispatched manually and always uploads these files. `--help` invokes no Docker command.
 
-The image is pinned to `qemux/qemu:7.50@sha256:e7f6fda52503a546fd649670ba46e4bc23dc6dcef275bc3fac48877fbbc430df`; if absent it may be pulled into the existing daemon's cache. One random-name/label container invokes `/usr/bin/qemu-system-x86_64` directly with KVM only, `-cpu host`, `-S`, no default devices, no display and HMP on stdin. It attaches no OS, disk or volume and never continues the paused CPU. The only host device mapping is `/dev/kvm:/dev/kvm:rw`. The filesystem is read-only, network is `none`, all Linux capabilities are dropped, and no-new-privileges is set. Limits are 0.5 CPU, 256 MiB container RAM/no additional swap, 32 PIDs, and 64 MiB paused guest RAM. There are no binds, ports, privileged mode, added capabilities or host networking.
+The image is pinned to `qemux/qemu:7.50@sha256:e7f6fda52503a546fd649670ba46e4bc23dc6dcef275bc3fac48877fbbc430df`; if absent it may be pulled into the existing daemon's cache. One random-name/label container invokes `/usr/bin/qemu-system-x86_64` directly with KVM only, `-cpu host`, `-S`, no default devices, no display and HMP on stdin. It attaches no OS, disk or persistent volume and never continues the paused CPU. A read-only 4-KiB tmpfs at `/storage` (`ro,nosuid,nodev,noexec,size=4096,mode=0555`) replaces the image's inherited `VOLUME /storage`. Inspect must prove `Mounts=[]` and precisely that sole tmpfs entry; otherwise a specific mount/tmpfs error is retained before QEMU starts. The only host device mapping is `/dev/kvm:/dev/kvm:rw`. The filesystem is read-only, network is `none`, all Linux capabilities are dropped, and no-new-privileges is set. Limits are 0.5 CPU, 256 MiB container RAM/no additional swap, 32 PIDs, and 64 MiB paused guest RAM. There are no binds, ports, privileged mode, added capabilities or host networking.
 
-The helper has a 95-second operation budget and a separate 20-second cleanup budget, plus at most two seconds to drain killed command output. The workflow permits five minutes including SDK setup, compilation and upload. Cleanup checks the saved random name/label and exact full container ID before stopping or removing that container. An interrupted create can recover its ID only from the saved random name with the exact ownership label. Cleanup does not remove images, prune resources, or touch another container.
+The helper has a 95-second operation budget and a separate 20-second cleanup budget, plus at most two seconds to drain killed command output. The workflow permits five minutes including SDK setup, compilation and upload. Cleanup checks the saved random name/label and exact full container ID before stopping or removing that container. An interrupted create can recover its ID only from the saved random name with the exact ownership label. `docker rm --volumes` also removes any anonymous volume attached to that exact owned container if creation did not match the expected tmpfs boundary. Cleanup does not remove images, prune resources, or touch another container.
+
+Actual run 4165 failed the original zero-mount guard because this pinned image declared `/storage` as a volume and Docker created an anonymous writable volume. QEMU never started: the saved container state was `created`, PID zero and `StartedAt` zero. Its exact container ID was `1753f95ef334244e7a1b393a839f218ea885363de7d5335eec53132d64627010`, owner token `43b7f4676c514f2a95c63c02577ac36e`, and anonymous volume `ef7daa62ef89a2ffb8aae50a9b7803f1d9b3075ee509aa3183f3e170f69ce595`. The original cleanup proved container removal; it did not prove volume removal.
+
+The temporary `--cleanup-run4165-volume` mode has a separate 20-second budget and accepts no target parameters. It records the frozen run/owner/volume target, artifact hashes, each command and its outcome under the supplied evidence directory. It requires the original Docker daemon ID `528941c8-73ac-49ff-8eb7-69113eb4a2a1`; absence of the old full container ID and random container name; exactly the recorded volume with local driver/scope and no options; and no container reference found by `docker ps --all --filter volume=...`. It uses `docker volume rm` without force, so Docker also refuses a reference introduced after the check. Already absent is an idempotent success only on the original daemon after old-container absence checks. A different daemon, attachment or missing evidence fails closed. No generalized orphan cleanup is provided. This temporary workflow step can be removed after actual removal/absence is verified. The source evidence is artifact ZIP SHA-256 `6745d90e8b81c867740405c99b4364cc165c47ebb165455052314459d5cd547b` and created-container inspect SHA-256 `7afdfc6c30c933bee2ef1d6c18ed011c8b2f709d1a5e88531928f9f40471c055`.
 
 `kvm_usable` requires QEMU to report both `kvm support: enabled` and `VM status: paused`, followed by clean monitor/container exit after `quit`. A device path alone is insufficient. Other receipts distinguish a Docker-reported missing daemon-host device, observed access denial, an unavailable QEMU KVM backend, an initialization error, and inconclusive evidence. These categories describe the observed output; they do not diagnose BIOS, nested virtualization, policy or hardware causes. All failures remain failed workflow runs with retained raw evidence. Even a usable result proves only this blank paused KVM initialization, not macOS boot, installation, native build or tests.
 
-The CLI and monitor behavior follow the primary [QEMU command-line reference](https://www.qemu.org/docs/master/system/qemu-manpage.html) and [QEMU monitor reference](https://www.qemu.org/docs/master/system/monitor.html). Device/container options follow the [Docker create reference](https://docs.docker.com/reference/cli/docker/container/create/).
+The CLI and monitor behavior follow the primary [QEMU command-line reference](https://www.qemu.org/docs/master/system/qemu-manpage.html) and [QEMU monitor reference](https://www.qemu.org/docs/master/system/monitor.html). Device/container options follow the [Docker create reference](https://docs.docker.com/reference/cli/docker/container/create/) and [Docker tmpfs reference](https://docs.docker.com/engine/storage/tmpfs/). Moby 28.3.3's [volume creation](https://raw.githubusercontent.com/moby/moby/v28.3.3/daemon/create_unix.go) and [mount detection](https://raw.githubusercontent.com/moby/moby/v28.3.3/container/container_unix.go) explicitly skip an inherited anonymous volume when that destination already has the tmpfs entry.

tools/ci/ExistingKvmDiagnostic.cs

@@ -10,6 +10,7 @@ static class ExistingKvmDiagnostic
 {
     const string Image = "qemux/qemu:7.50@sha256:e7f6fda52503a546fd649670ba46e4bc23dc6dcef275bc3fac48877fbbc430df";
     const string Label = "cloud.schweigert.meeting-assistant.existing-kvm-probe";
+    const string StorageTmpfsOptions = "ro,nosuid,nodev,noexec,size=4096,mode=0555";
     static readonly string[] QemuArguments = ["-machine", "pc", "-accel", "kvm", "-cpu", "host", "-m", "64", "-smp", "1", "-S", "-nodefaults",
         "-display", "none", "-monitor", "stdio", "-serial", "none", "-parallel", "none", "-nic", "none"];
     static readonly JsonSerializerOptions Json = new() { WriteIndented = true };
@@ -18,14 +19,15 @@ static class ExistingKvmDiagnostic
     {
         if (args.SequenceEqual(new[] { "--help" }))
         {
-            Console.WriteLine("ExistingKvmDiagnostic.cs --run|--cleanup --output DIRECTORY\nRequires .NET 10 and the existing Docker CLI/daemon. --help never calls Docker.\n--run tests only a paused, diskless QEMU with existing /dev/kvm; retains evidence and cleans up its own container. --cleanup retries that saved cleanup.");
+            Console.WriteLine("ExistingKvmDiagnostic.cs --run|--cleanup|--cleanup-run4165-volume --output DIRECTORY\nRequires .NET 10 and the existing Docker CLI/daemon. --help never calls Docker.\n--run tests only a paused, diskless QEMU with existing /dev/kvm; retains evidence and cleans up its own container. --cleanup retries that saved cleanup.\n--cleanup-run4165-volume removes only run 4165's frozen unused anonymous volume after daemon, old-container absence and reference checks.");
             return 0;
         }
-        if (args.Length != 3 || args[0] is not ("--run" or "--cleanup") || args[1] != "--output")
+        if (args.Length != 3 || args[0] is not ("--run" or "--cleanup" or "--cleanup-run4165-volume") || args[1] != "--output")
-            throw new ArgumentException("Use --help or --run|--cleanup --output DIRECTORY.");
+            throw new ArgumentException("Use --help or --run|--cleanup|--cleanup-run4165-volume --output DIRECTORY.");
         var output = Path.GetFullPath(args[2]);
         Directory.CreateDirectory(output);
         if (args[0] == "--cleanup") return await Cleanup(output) ? 0 : 1;
+        if (args[0] == "--cleanup-run4165-volume") return await CleanupRun4165Volume(output) ? 0 : 1;
         if (Directory.EnumerateFileSystemEntries(output).Any()) throw new InvalidOperationException("Run output must be empty; existing receipts cannot be reused.");
 
         var started = DateTimeOffset.UtcNow;
@@ -51,6 +53,7 @@ static class ExistingKvmDiagnostic
                 "--label", Label + "=" + token, "--cidfile", Path.Combine(output, "container.id"), "--interactive", "--read-only",
                 "--network", "none", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--cpus", "0.5",
                 "--memory", "256m", "--memory-swap", "256m", "--pids-limit", "32", "--no-healthcheck",
+                "--tmpfs", "/storage:" + StorageTmpfsOptions,
                 "--device", "/dev/kvm:/dev/kvm:rw", "--entrypoint", "/usr/bin/qemu-system-x86_64", Image, .. QemuArguments], output, "container-create", budget.Token);
             if (create.ExitCode != 0)
             {
@@ -96,10 +99,15 @@ static class ExistingKvmDiagnostic
     {
         var host = container.GetProperty("HostConfig");
         var devices = host.GetProperty("Devices");
+        if (container.GetProperty("Mounts").GetArrayLength() != 0)
+            throw new InvalidOperationException("Container mount boundary failed: persistent volumes or binds were created; expected Mounts=[] with only the read-only /storage tmpfs.");
+        if (!host.TryGetProperty("Tmpfs", out var tmpfs) || tmpfs.ValueKind != JsonValueKind.Object || tmpfs.EnumerateObject().Count() != 1
+            || !tmpfs.TryGetProperty("/storage", out var options) || options.GetString() != StorageTmpfsOptions)
+            throw new InvalidOperationException("Container tmpfs boundary failed: expected only /storage:" + StorageTmpfsOptions + ".");
         if (imageId == null || container.GetProperty("Image").GetString() != imageId || container.GetProperty("Config").GetProperty("Image").GetString() != Image
             || container.GetProperty("Path").GetString() != "/usr/bin/qemu-system-x86_64" || !container.GetProperty("Args").EnumerateArray().Select(x => x.GetString()).SequenceEqual(QemuArguments)
             || host.GetProperty("Privileged").GetBoolean() || !host.GetProperty("ReadonlyRootfs").GetBoolean()
-            || host.GetProperty("NetworkMode").GetString() != "none" || container.GetProperty("Mounts").GetArrayLength() != 0
+            || host.GetProperty("NetworkMode").GetString() != "none"
             || devices.GetArrayLength() != 1 || devices[0].GetProperty("PathOnHost").GetString() != "/dev/kvm"
             || devices[0].GetProperty("PathInContainer").GetString() != "/dev/kvm" || devices[0].GetProperty("CgroupPermissions").GetString() != "rw"
             || !host.GetProperty("CapDrop").EnumerateArray().Any(x => x.GetString() == "ALL")
@@ -151,7 +159,7 @@ static class ExistingKvmDiagnostic
                 await Command("docker", ["stop", "--time", "1", id], output, prefix + "-stop", budget.Token);
                 await InspectOwned(output, owner, prefix + "-reinspect", budget.Token);
             }
-            await Require(Command("docker", ["rm", "--force", id], output, prefix + "-remove", budget.Token));
+            await Require(Command("docker", ["rm", "--force", "--volumes", id], output, prefix + "-remove", budget.Token));
             var receipt = new { cleaned = true, id, finished = DateTimeOffset.UtcNow };
             Save(output, prefix + ".receipt.json", receipt);
             Save(output, "cleanup.json", receipt);
@@ -169,6 +177,72 @@ static class ExistingKvmDiagnostic
         }
     }
 
+    static async Task<bool> CleanupRun4165Volume(string output)
+    {
+        // Frozen from actual run 4165: the exact owner-labelled container never started (created/PID 0/zero StartedAt).
+        // Its image's VOLUME instruction created this anonymous /storage volume before the mount guard failed.
+        const string daemonId = "528941c8-73ac-49ff-8eb7-69113eb4a2a1";
+        const string containerId = "1753f95ef334244e7a1b393a839f218ea885363de7d5335eec53132d64627010";
+        const string token = "43b7f4676c514f2a95c63c02577ac36e";
+        const string volumeName = "ef7daa62ef89a2ffb8aae50a9b7803f1d9b3075ee509aa3183f3e170f69ce595";
+        const string containerName = "meeting-assistant-kvm-" + token;
+        var prefix = "run4165-volume-" + Guid.NewGuid().ToString("N");
+        Save(output, prefix + ".target.json", new { runId = 4165, daemonId, containerId, containerName, token, volumeName,
+            archiveSha256 = "6745d90e8b81c867740405c99b4364cc165c47ebb165455052314459d5cd547b",
+            createdInspectSha256 = "7afdfc6c30c933bee2ef1d6c18ed011c8b2f709d1a5e88531928f9f40471c055",
+            ownerReceiptSha256 = "d96f15588412a5928ebe8a64b115764f113adaaaf6d164209fd87de8a4505572",
+            priorState = "created", priorPid = 0, priorStartedAt = "0001-01-01T00:00:00Z" });
+        using var budget = new CancellationTokenSource(TimeSpan.FromSeconds(20));
+        var success = false;
+        var outcome = "refused";
+        string? detail = null;
+        try
+        {
+            await Require(Command("docker", ["info", "--format", "{{.ID}}"], output, prefix + "-daemon", budget.Token));
+            if (Read(output, prefix + "-daemon", "stdout").Trim() != daemonId) throw new InvalidOperationException("Frozen run 4165 daemon ID differs; refusing volume cleanup.");
+            foreach (var selector in new[] { containerId, containerName })
+            {
+                var step = prefix + (selector == containerId ? "-old-id" : "-old-name");
+                var inspection = await Command("docker", ["inspect", "--type", "container", selector], output, step, budget.Token);
+                if (inspection.TimedOut || inspection.Error != null || inspection.ExitCode == 0 || !Read(output, step, "stderr").Contains("No such container", StringComparison.OrdinalIgnoreCase)
+                    || !Read(output, step, "stderr").Contains(selector, StringComparison.Ordinal)) throw new InvalidOperationException("Old run 4165 container absence was not proved for " + selector + ".");
+            }
+            var inspectStep = prefix + "-inspect";
+            var inspect = await Command("docker", ["volume", "inspect", volumeName], output, inspectStep, budget.Token);
+            if (VolumeAbsent(inspect, output, inspectStep, volumeName)) { success = true; outcome = "already-absent"; }
+            else
+            {
+                if (inspect.ExitCode != 0 || inspect.TimedOut || inspect.Error != null) throw new InvalidOperationException("Cannot inspect the exact run 4165 volume.");
+                using var document = JsonDocument.Parse(Read(output, inspectStep, "stdout"));
+                var values = document.RootElement;
+                if (values.GetArrayLength() != 1 || values[0].GetProperty("Name").GetString() != volumeName
+                    || values[0].GetProperty("Driver").GetString() != "local" || values[0].GetProperty("Scope").GetString() != "local"
+                    || !Empty(values[0].GetProperty("Options"))) throw new InvalidOperationException("Frozen volume name/local driver/scope/options boundary failed.");
+                await Require(Command("docker", ["ps", "--all", "--no-trunc", "--filter", "volume=" + volumeName, "--format", "{{.ID}}"], output, prefix + "-references", budget.Token));
+                if (Read(output, prefix + "-references", "stdout").Trim().Length != 0) throw new InvalidOperationException("A container references the frozen run 4165 volume; refusing removal.");
+                // No --force: Docker rejects an attachment made after the reference check, too.
+                await Require(Command("docker", ["volume", "rm", volumeName], output, prefix + "-remove", budget.Token));
+                var afterStep = prefix + "-after";
+                var after = await Command("docker", ["volume", "inspect", volumeName], output, afterStep, budget.Token);
+                if (!VolumeAbsent(after, output, afterStep, volumeName)) throw new InvalidOperationException("Volume absence after removal was not proved.");
+                success = true;
+                outcome = "removed";
+            }
+        }
+        catch (Exception error) { detail = error.Message; }
+        var receipt = new { runId = 4165, daemonId, volumeName, success, outcome, detail, finished = DateTimeOffset.UtcNow };
+        Save(output, prefix + ".receipt.json", receipt);
+        Console.WriteLine(JsonSerializer.Serialize(receipt));
+        return success;
+    }
+
+    static bool VolumeAbsent(CommandResult result, string output, string step, string name)
+    {
+        var error = Read(output, step, "stderr");
+        return result.ExitCode is not (null or 0) && !result.TimedOut && result.Error == null
+            && error.Contains("no such volume", StringComparison.OrdinalIgnoreCase) && error.Contains(name, StringComparison.Ordinal);
+    }
+
     static async Task<CommandResult> Command(string program, string[] arguments, string output, string step, CancellationToken cancellation, string? input = null)
     {
         var started = DateTimeOffset.UtcNow;