diff --git a/README.md b/README.md index 027f22b..cb0d805 100644 --- a/README.md +++ b/README.md @@ -175,7 +175,7 @@ The Ubuntu managed-test job does not compile the Swift helpers or execute Apple The separate manual `.gitea/workflows/macos-native-full.yaml` retains the same Full flow as a diagnostic entry point. CI validates source; it does not publish or deploy the workstation application. -The separate manual [native Recovery diagnostic](docs/macos-native-diagnostic.md) probes macOS startup and disk readiness before qualifying the prepared Full flow. It neither installs macOS nor runs application tests. A green Recovery diagnostic alone does not verify macOS build/test CI support; each Full run repeats Recovery readiness for its own guest and disk. +The separate manual [native Recovery diagnostic](docs/macos-native-diagnostic.md) can isolate macOS startup and disk readiness. It neither installs macOS nor runs application tests. A green Recovery diagnostic alone does not verify macOS build/test CI support; each Full run requires fresh Recovery readiness for its own guest and disk before permitting installation. ## Operations And Limitations diff --git a/docs/macos-native-diagnostic.md b/docs/macos-native-diagnostic.md index c992931..3a2eaa2 100644 --- a/docs/macos-native-diagnostic.md +++ b/docs/macos-native-diagnostic.md @@ -8,7 +8,11 @@ The existing Intel Celeron 1037U has neither AVX nor AVX2. KVM run 4187 at `720a [CryptexFixup 1.0.5](https://github.com/acidanthera/CryptexFixup/blob/1.0.5/kern_start.cpp) selects the installed/updated Rosetta Cryptex and patches APFS hash checking; it does not replace the running Recovery cache or emulate instructions. macOS 13 is outside the [.NET 10 supported-OS policy](https://github.com/dotnet/core/blob/main/release-notes/10.0/supported-os.md). This candidate therefore uses macOS 14 and software CPU emulation without Cryptex. It changes the compatibility profile, not one isolated causal variable; actual success must be measured. -Earlier TCG run 4159 observed guest AVX2. Runs 4161/4163 measured slow native startup and reached the 40-minute host limit before readiness. They predated the UDIF CRC repair at `94a70b2`, reuse of successful sw_vers output and capturing the large Recovery hash only once. They do not qualify this candidate. Host/workflow limits are 90/95 minutes; a readiness pass does not establish that full installation/build/tests fit the pipeline. +Earlier TCG run 4159 observed guest AVX2 with the upstream-selected Skylake model. Runs 4161/4163 measured slow native startup and reached the 40-minute host limit before readiness. They predated the UDIF CRC repair at `94a70b2`, reuse of successful sw_vers output and capturing the large Recovery hash only once. They do not qualify this candidate. Host/workflow limits for the read-only mode are 90/95 minutes; a readiness pass does not establish that full installation/build/tests fit the pipeline. + +Run 4188 with Haswell recorded a boot loop; first-reset run 4189 retained repeated supervisor instruction-fetch pagefaults at RIP/CR2 `0x24b0` before native readiness. Run 4190 at `3aaab45` restored `Skylake-Client-v4` and the upstream TCG `-spec-ctrl` mask. The actual AVX/AVX2 ROM passed (exit 33; AVX2-disabled control exit 0), and macOS 14's Darwin 23.6.0 kernel identified the Skylake CPU. It retained one kernel handoff, a running VM and later userspace execution without the earlier reset, but reached the 20-minute diagnostic deadline without the readiness hook. These observations do not identify Haswell as the original cause or qualify native tests. + +Run 4190 used synchronous kernel serial output and QEMU interrupt/register tracing to preserve the failure context. Its kernel explicitly warned that synchronous output impacts performance. The current full candidate uses normal upstream boot arguments and only the existing iothread QEMU argument; it retains actual CPU/staging receipts independently of Docker log rotation. Its existing fresh-readiness gate precedes every installation permit. This allows one bounded full qualification to test boot performance and, only after readiness, installation/build/tests without duplicating the guest startup. No remote full result has qualified this candidate yet. ## Entry points and dependencies @@ -27,7 +31,7 @@ The native diagnostic workflow is manual only. Temporary diagnostic branches are The existing daemon must be Linux/x64 with two CPUs and 6 GiB memory; the runner must have 5 GiB available memory and the Docker filesystem 8 GiB free. These checks do not reconfigure resources. Dockur commit `16a5b470cdd601bae8b05b02d748d7edfb36c12e`, both imported QEMU image digests and original source seams remain pinned. -Actual `Haswell-noTSX` CPU flags under TCG use `enforce=on` to reject unsupported requests. The CPU preflight uses that same composed flag list and QEMU binary before Recovery download/boot. `tools/ci/macos-tcg-cpu-preflight.asm` enables long mode/YMM state, executes AVX and AVX2 integer arithmetic, and checks an Int32 from the upper 128-bit lane. Only the correct result reaches [QEMU debug-exit](https://github.com/qemu/qemu/blob/v11.1.1/hw/misc/debugexit.c) code 33. No disks/network attach; failure/timeout fails preflight. This tests that instruction chain, not the complete ISA or macOS. +Actual `Skylake-Client-v4` CPU flags under TCG use `enforce=on` to reject unsupported requests. The CPU preflight uses that same composed flag list and QEMU binary before Recovery download/boot. `tools/ci/macos-tcg-cpu-preflight.asm` enables long mode/YMM state, executes AVX and AVX2 integer arithmetic, and checks an Int32 from the upper 128-bit lane. Only the correct result reaches [QEMU debug-exit](https://github.com/qemu/qemu/blob/v11.1.1/hw/misc/debugexit.c) code 33. No disks/network attach; failure/timeout fails preflight. This tests that instruction chain, not the complete ISA or macOS. The locally assembled NASM 2.16.03 ROM is 65,536 bytes, SHA256 `c32746122cc68f3ed642aa46c21b677f803c58f0d4ff665841723fcc5625f549`. Assembly/static review does not prove remote execution. @@ -41,7 +45,7 @@ Required commands retain 45 seconds, UID 180 seconds and the single disk query 1 The container retains 6 GiB memory/swap, two-CPU limit, 512 MiB shared memory and a 4-GiB/two-vCPU guest. One fresh anonymous /storage volume holds the sparse 64-GiB target. Inspection rejects devices, capabilities, binds, ports, host networking and privileged mode. KVM is disabled with no /dev/kvm mapping; guest networking stays slirp. -Evidence retains run/source/profile identity, CPU preflight, original/patched Recovery identity, container/QEMU state, native proof/result and cleanup. Optional bounded before/during/after pressure snapshots record host/cgroup counters. The /storage/14/setup.dmg hash is captured once after staging; successful evidence survives later capture failure. Screenshots/pressure observations pass no gate. +Evidence retains run/source/profile identity, CPU preflight, original/patched Recovery identity, container/QEMU state, native proof/result and cleanup. Sparse kernel-handoff lines are retained separately; two handoffs before readiness/installation permission fail early. After permission, normal installer reboots remain allowed. Optional bounded before/during/after pressure snapshots record host/cgroup counters. The /storage/14/setup.dmg hash is captured once after staging; successful evidence survives later capture failure. Screenshots/pressure observations pass no gate. Both cleanup paths verify exact token/label/ID before removing only the owned container, anonymous volume and image. No pruning, host changes, original checkout changes or Meeting Assistant restart occurs. Artifacts remain seven days. Full CI remains unverified until an installed supported guest builds/signs fresh helpers and passes all 577 tests, including the five native macOS tests, with zero skips. @@ -61,4 +65,4 @@ The installer provisions the owned guest and returns into the prepared firstboot `tools/ci/MacOsNativeGuest.cs` restores/builds/tests the source inside the installed guest. Success requires four fresh x86_64 Mach-O helpers, a valid audio-app signature and a fresh source-bound TRX containing exactly 577 distinct passing tests, zero failures/skips and all five named native macOS tests. Archive, SDK, build, signature and TRX receipts are retained. The required PR job follows the existing Wine and portable jobs; all jobs still select `ubuntu-latest`. -The workflow has 180 minutes; the controller reserves cleanup time with a shared 172-minute total deadline. Recovery, installation, CLT and test caps are 90/80/30/25 minutes under that same total, not additive promises. Actual supported-guest installation/performance and remote test success remain unqualified. Do not start this prepared installer until the separate prerequisite diagnostic passes. CI does not deploy or restart the workstation application. +The workflow has 180 minutes; the controller reserves cleanup time with a shared 172-minute total deadline. Recovery, installation, CLT and test caps are 90/80/30/25 minutes under that same total, not additive promises. Actual supported-guest installation/performance and remote test success remain unqualified. The full run's own mandatory fresh-readiness and owned-disk gates prevent installation until that guest passes its prerequisites. CI does not deploy or restart the workstation application. diff --git a/tools/ci/MacOsNativeDiagnostic.cs b/tools/ci/MacOsNativeDiagnostic.cs index 9489036..8c55858 100644 --- a/tools/ci/MacOsNativeDiagnostic.cs +++ b/tools/ci/MacOsNativeDiagnostic.cs @@ -14,9 +14,9 @@ return await NativeDiagnostic.Execute(args); static class NativeDiagnostic { const string DockurCommit = "16a5b470cdd601bae8b05b02d748d7edfb36c12e"; - const string Profile = "tcg-haswell-sonoma"; - const string CpuModel = "Haswell-noTSX"; - const string CpuFlags = "Haswell-noTSX,l3-cache=on,+hypervisor,vendor=GenuineIntel,vmx=off,vmware-cpuid-freq=on,-pdpe1gb,-pcid,-invpcid,-tsc-deadline,-xsavec,-xsaves,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+bmi1,+bmi2,+smep,+xsave,+xsaveopt,+xgetbv1,+movbe,+rdrand,enforce=on"; + const string Profile = "tcg-skylake-sonoma"; + const string CpuModel = "Skylake-Client-v4"; + const string CpuFlags = "Skylake-Client-v4,l3-cache=on,+hypervisor,vendor=GenuineIntel,vmx=off,vmware-cpuid-freq=on,-pdpe1gb,-spec-ctrl,-pcid,-invpcid,-tsc-deadline,-xsavec,-xsaves,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+bmi1,+bmi2,+smep,+xsave,+xsaveopt,+xgetbv1,+movbe,+rdrand,enforce=on"; const string OpenCoreTemplateHash = "287328995d4198f1b05166f087d85bf7ef66bedafe150d17ad112ac8de60051d"; const string UdifChecksumBindingHash = "6109d04619e800c483fdac363d593cd1cd69f34131d2521417334e11d41c8bfa"; const string OwnerLabel = "org.meeting-assistant.native-diagnostic"; @@ -69,6 +69,7 @@ static class NativeDiagnostic if (args.Contains("--validate")) { ValidateContracts(); + ValidateBootProgress(); if (full) ValidateFullContracts(); if (Option(args, "--source") is { } source) { @@ -88,6 +89,7 @@ static class NativeDiagnostic resultNegativeCases = 6, containerNegativeCases = 11, recoveryPositiveCases = 4, recoveryNegativeCases = 12, independentFixtureCrc32Readback = true, resourceSnapshotRetention = true, cpuProfileSourceContractsVerified = true, preflightGateFixtureCases = 8, qemuRuntimePreflightExecuted = false, templateIsoDownloaded = false, + bootProgressParserFixtureCases = 6, recoveryRepeatGuardBeforePermit = true, fullResultContractsVerified = full, fullBootstrapFixtureCases = full ? 12 : 0, installerGuardFixtureCases = full ? 10 : 0, firstbootEvidenceFixtureCases = full ? 4 : 0, ownedDiskSerialFixtureCases = full ? 6 : 0, sourceModified = false, dockerExecuted = false, guestExecuted = false, completedUtc = DateTimeOffset.UtcNow @@ -162,7 +164,7 @@ static class NativeDiagnostic AssertContainer(File.ReadAllText(Path.Combine(output, "container-created.stdout.log")), token); await Command("docker", ["start", id], output, "docker-start", deadline.Token); await CapturePressure(id, output, "before", deadline.Token); - Console.WriteLine(full ? "The owned unprivileged TCG/Haswell macOS 14 guest is starting. Installation requires fresh native readiness and an owned-disk permit; success requires all 577 tests with zero skips." : "The owned unprivileged TCG/Haswell macOS 14 guest is starting. Success requires native macOS 14+/x86_64 and a writable 64-GiB disk; no installer will run."); + Console.WriteLine(full ? "The owned unprivileged TCG/Skylake macOS 14 guest is starting. Installation requires fresh native readiness and an owned-disk permit; success requires all 577 tests with zero skips." : "The owned unprivileged TCG/Skylake macOS 14 guest is starting. Success requires native macOS 14+/x86_64 and a writable 64-GiB disk; no installer will run."); var phaseStarted = Stopwatch.StartNew(); var phase = "recovery"; var phaseBudget = TimeSpan.FromMinutes(90); @@ -173,6 +175,7 @@ static class NativeDiagnostic { deadline.Token.ThrowIfCancellationRequested(); await CaptureGuest(id, output, deadline.Token, full); + if (!permitted) CheckRecoveryBootProgress(output); if (full && phaseStarted.Elapsed > phaseBudget) throw new InvalidOperationException("The bounded native " + phase + " phase exceeded " + phaseBudget.TotalMinutes + " minutes."); var proofPath = Path.Combine(output, "guest-proof.log"); @@ -283,10 +286,10 @@ static class NativeDiagnostic throw new InvalidOperationException("Diagnostic validator accepted an invalid/stale result."); } var boundary = """ - [{"Config":{"Labels":{"org.meeting-assistant.native-diagnostic":"validation"},"Env":["KVM=N","CPU_MODEL=Haswell-noTSX","VERSION=14"]},"HostConfig":{"Privileged":false,"NetworkMode":"default","Memory":6442450944,"MemorySwap":6442450944,"NanoCpus":2000000000,"ShmSize":536870912,"CapAdd":null,"DeviceRequests":null,"Binds":null,"PortBindings":{},"DeviceCgroupRules":null,"Tmpfs":null,"Devices":[]},"Mounts":[{"Type":"volume","Destination":"/storage","RW":true}]}] + [{"Config":{"Labels":{"org.meeting-assistant.native-diagnostic":"validation"},"Env":["KVM=N","CPU_MODEL=Skylake-Client-v4","VERSION=14"]},"HostConfig":{"Privileged":false,"NetworkMode":"default","Memory":6442450944,"MemorySwap":6442450944,"NanoCpus":2000000000,"ShmSize":536870912,"CapAdd":null,"DeviceRequests":null,"Binds":null,"PortBindings":{},"DeviceCgroupRules":null,"Tmpfs":null,"Devices":[]},"Mounts":[{"Type":"volume","Destination":"/storage","RW":true}]}] """; AssertContainer(boundary, "validation"); - foreach (var invalid in new[] { boundary.Replace("\"Privileged\":false", "\"Privileged\":true"), boundary.Replace("\"Devices\":[]", "\"Devices\":[{\"PathOnHost\":\"/dev/kvm\",\"PathInContainer\":\"/dev/kvm\",\"CgroupPermissions\":\"rw\"}]"), boundary.Replace("KVM=N", "KVM=Y"), boundary.Replace("CPU_MODEL=Haswell-noTSX", "CPU_MODEL=host"), boundary.Replace("VERSION=14", "VERSION=13"), boundary.Replace("6442450944", "8589934592"), boundary.Replace("\"NetworkMode\":\"default\"", "\"NetworkMode\":\"host\""), boundary.Replace("\"CapAdd\":null", "\"CapAdd\":[\"NET_ADMIN\"]"), boundary.Replace("\"Type\":\"volume\"", "\"Type\":\"bind\""), boundary.Replace("/storage", "/host"), boundary.Replace("\"NanoCpus\":2000000000", "\"NanoCpus\":4000000000") }) + foreach (var invalid in new[] { boundary.Replace("\"Privileged\":false", "\"Privileged\":true"), boundary.Replace("\"Devices\":[]", "\"Devices\":[{\"PathOnHost\":\"/dev/kvm\",\"PathInContainer\":\"/dev/kvm\",\"CgroupPermissions\":\"rw\"}]"), boundary.Replace("KVM=N", "KVM=Y"), boundary.Replace("CPU_MODEL=Skylake-Client-v4", "CPU_MODEL=host"), boundary.Replace("VERSION=14", "VERSION=13"), boundary.Replace("6442450944", "8589934592"), boundary.Replace("\"NetworkMode\":\"default\"", "\"NetworkMode\":\"host\""), boundary.Replace("\"CapAdd\":null", "\"CapAdd\":[\"NET_ADMIN\"]"), boundary.Replace("\"Type\":\"volume\"", "\"Type\":\"bind\""), boundary.Replace("/storage", "/host"), boundary.Replace("\"NanoCpus\":2000000000", "\"NanoCpus\":4000000000") }) { try { AssertContainer(invalid, "validation"); } catch (InvalidOperationException) { continue; } throw new InvalidOperationException("Diagnostic validator accepted an excessive/wrong-profile container boundary."); @@ -350,13 +353,17 @@ static class NativeDiagnostic var cpu = File.ReadAllText(cpuPath); if (Hash(Encoding.UTF8.GetBytes(cpu)) != "0f3e4b4e1c3e17743d3a8d27b77a76424ceebb576b269283d612c489bc70993e") throw new InvalidOperationException("Pinned CPU composition script hash mismatch."); cpu = ReplaceOnce(cpu, ",+movbe,+rdrand,check\"", ",+movbe,+rdrand,enforce=on\""); + // Explicit CPU_MODEL bypasses upstream selection, so restore its TCG mitigation mask. + cpu = ReplaceOnce(cpu, " DEFAULT_FLAGS+=\",-pcid,-invpcid,-tsc-deadline,-xsavec,-xsaves\"", " DEFAULT_FLAGS+=\",-spec-ctrl,-pcid,-invpcid,-tsc-deadline,-xsavec,-xsaves\""); var preflight = File.ReadAllText(Path.Combine("tools", "ci", "macos-tcg-cpu-preflight.asm")); var entryPath = Path.Combine(source, "src/entry.sh"); var entry = ReplaceOnce(File.ReadAllText(entryPath), "set -Eeuo pipefail\n", "set -Eeuo pipefail\n\n# Diagnostic budget: inspect existing Docker storage before Recovery download/boot.\ndf -Pk /storage\nfree_kib=$(df -Pk /storage | awk 'NR==2 {print $4}')\n[[ \"$free_kib\" =~ ^[0-9]+$ ]] && (( free_kib >= 8 * 1024 * 1024 )) || { echo 'Existing Docker storage has less than the 8-GiB diagnostic budget.' >&2; exit 1; }\n"); entry = ReplaceOnce(entry, ". cpu.sh # Configure CPU model\n", ""); entry = ReplaceOnce(entry, ". proc.sh # Initialize processor\n", ""); entry = ReplaceOnce(entry, ". init.sh # Initialize system\n", ". init.sh # Initialize system\n. cpu.sh # Compose the exact guest CPU before any Apple download\n. proc.sh # Compose the actual accelerator/CPU_FLAGS once\n" + TcgPreflight + "\n"); - entry = ReplaceOnce(entry, "trap - ERR\n", "[[ \"$KVM_OPTS\" == ' -accel tcg,thread=multi' && \"$CPU_FLAGS\" == '" + CpuFlags + "' && \"$CPU_OPTS\" == \"-cpu $CPU_FLAGS -smp $SMP\" ]] || { error 'Supported profile refuses a CPU/accelerator fallback.'; exit 1; }\ninfo '[supported-profile] accelerator=tcg cpu=Haswell-noTSX recovery=14; AVX/AVX2 preflight passed; native guest gates still pending'\n\ntrap - ERR\n"); + entry = ReplaceOnce(entry, "trap - ERR\n", "[[ \"$KVM_OPTS\" == ' -accel tcg,thread=multi' && \"$CPU_FLAGS\" == '" + CpuFlags + "' && \"$CPU_OPTS\" == \"-cpu $CPU_FLAGS -smp $SMP\" ]] || { error 'Supported profile refuses a CPU/accelerator fallback.'; exit 1; }\ninfo '[supported-profile] accelerator=tcg cpu=Skylake-Client-v4 recovery=14; AVX/AVX2 preflight passed; native guest gates still pending'\n\nprintf '%s\\n' '[supported-profile] accelerator=tcg cpu=Skylake-Client-v4 recovery=14; AVX/AVX2 preflight passed; native guest gates still pending' >> \"$QEMU_DIR/native-stage.log\"\ntrap - ERR\n"); + // Retain sparse boot markers without enabling verbose kernel/exception output. + entry = ReplaceOnce(entry, " -e 's/failed to load Boot/skipped Boot/g' \\\n", " -e 's/failed to load Boot/skipped Boot/g' \\\n -e '/^#\\[EB|LOG:HANDOFF TO XNU\\] /w /run/shm/kernel-handoffs.log' \\\n"); var hookPath = Path.Combine("tools", "ci", "macos-native-readiness.sh"); var hook = ReplaceOnce(File.ReadAllText(hookPath), "@@PROOF_TOKEN@@", token); var wrapper = File.ReadAllText(Path.Combine("tools", "ci", full ? "macos-native-full-bootstrap.sh" : "macos-native-bootstrap.sh")); @@ -452,7 +459,7 @@ static class NativeDiagnostic if (!add.Elements("dict").Select(dict => PlistValue(dict, "BundlePath").Value).SequenceEqual(expected) || add.Elements("dict").Any(dict => PlistValue(dict, "Enabled").Name != "true")) throw new InvalidOperationException("Pinned Kernel.Add order/enabled contract mismatch."); boot = ReplaceOnce(boot, " PLIST=\"/assets/config.plist\"\n", " [ ! -e /custom.plist ] || { error 'Supported profile refuses an unverified custom OpenCore config!'; exit 12; }\n PLIST=\"/assets/config.plist\"\n"); boot = ReplaceOnce(boot, " if [ -s \"$target\" ] && [ \"$previous\" = \"$current\" ]; then\n IMG=\"$target\"\n return 0\n fi\n", " # This owned compatibility probe always rebuilds; never trust a cached boot.img.\n"); - boot = ReplaceOnce(boot, " echo \"VMHIDE=$vmhide\"\n", " echo \"VMHIDE=$vmhide\"\n echo \"PROFILE=tcg-haswell-sonoma\"\n"); + boot = ReplaceOnce(boot, " echo \"VMHIDE=$vmhide\"\n", " echo \"VMHIDE=$vmhide\"\n echo \"PROFILE=tcg-skylake-sonoma\"\n"); return boot; } @@ -465,7 +472,7 @@ static class NativeDiagnostic static readonly string TcgPreflight = """ # Realize this exact TCG model and execute AVX/AVX2 before Apple downloads. - disabled "$KVM" && [[ "$ARCH" == amd64 && "$CPU_MODEL" == Haswell-noTSX && "$VERSION" == 14 && "$CPU_FLAGS" == '@@CPU_FLAGS@@' ]] || { error 'Supported probe requires the exact TCG/Haswell/macOS 14 profile.'; exit 1; } + disabled "$KVM" && [[ "$ARCH" == amd64 && "$CPU_MODEL" == Skylake-Client-v4 && "$VERSION" == 14 && "$CPU_FLAGS" == '@@CPU_FLAGS@@' ]] || { error 'Supported probe requires the exact TCG/Skylake/macOS 14 profile.'; exit 1; } [[ "$(qemu-system-x86_64 --version | head -n 1)" == 'QEMU emulator version 11.1.1 (Reims 11.1.3)' ]] || { error 'Pinned QEMU runtime version mismatch.'; exit 1; } printf '%s %s\n' c32746122cc68f3ed642aa46c21b677f803c58f0d4ff665841723fcc5625f549 /assets/ci-cpu-preflight.bin | sha256sum -c - || { error 'Compiled AVX/AVX2 preflight ROM hash mismatch.'; exit 1; } probeTcgInstructions() { @@ -481,6 +488,7 @@ static class NativeDiagnostic cat "$QEMU_DIR/cpu-preflight-negative.log" (( negative != 33 )) || { error 'AVX2-disabled negative control unexpectedly passed.'; exit 1; } info "[cpu-preflight] positive=$positive negative=$negative cpu=$CPU_FLAGS; actual AVX/AVX2 executed before Apple download" + printf '[cpu-preflight] positive=%s negative=%s cpu=%s; actual AVX/AVX2 executed before Apple download\n' "$positive" "$negative" "$CPU_FLAGS" > "$QEMU_DIR/native-stage.log" """.Replace("@@CPU_FLAGS@@", CpuFlags, StringComparison.Ordinal); static async Task ValidateTcgPreflight(string output, CancellationToken cancellation) @@ -518,7 +526,7 @@ static class NativeDiagnostic printf '[fixture-command] %s\n' "$*" case "$*" in *,-avx2*) return @@NEGATIVE@@ ;; *) return @@POSITIVE@@ ;; esac } - KVM=N; ARCH=amd64; CPU_MODEL=Haswell-noTSX; VERSION='@@VERSION@@' + KVM=N; ARCH=amd64; CPU_MODEL=Skylake-Client-v4; VERSION='@@VERSION@@' CPU_FLAGS='@@FLAGS@@'; QEMU_DIR='@@WORK@@' """.Replace("@@HASH_EXIT@@", item.Item6 ? "0" : "1", StringComparison.Ordinal) .Replace("@@NEGATIVE@@", item.Item3.ToString(), StringComparison.Ordinal) @@ -969,7 +977,7 @@ static class NativeDiagnostic var drive = await Command("docker", ["exec", id, "qemu-img", "info", "--force-share", "--output=json", "/storage/14/data.img"], output, "owned-raw-disk", cancellation); using (var document = JsonDocument.Parse(drive.Output)) if (document.RootElement.GetProperty("format").GetString() != "raw" || document.RootElement.GetProperty("virtual-size").GetInt64() != GuestDiskBytes) throw new InvalidOperationException("Owned VM raw-disk capacity/format mismatch."); - var attachment = await Command("docker", ["exec", id, "sh", "-c", "qemu_pid=$(cat /dev/shm/qemu.pid); tr '\\0' '\\n' < /proc/\"$qemu_pid\"/cmdline | sed -n '/^file=\\/storage\\/13\\/data\\.img,/p; /^ide-hd,drive=data3,/p; /^local,id=installstatefs,/p'"], output, "owned-disk-attachment", cancellation); + var attachment = await Command("docker", ["exec", id, "sh", "-c", "qemu_pid=$(cat /dev/shm/qemu.pid); tr '\\0' '\\n' < /proc/\"$qemu_pid\"/cmdline | sed -n '/^file=\\/storage\\/14\\/data\\.img,/p; /^ide-hd,drive=data3,/p; /^local,id=installstatefs,/p'"], output, "owned-disk-attachment", cancellation); var lines = attachment.Output.Split('\n', StringSplitOptions.RemoveEmptyEntries); if (lines.Length != 3 || !lines.Any(line => line.StartsWith("file=/storage/14/data.img,id=data3,format=raw,", StringComparison.Ordinal) && !line.Contains("readonly=on", StringComparison.Ordinal)) || !lines.Any(line => line.StartsWith("ide-hd,drive=data3,", StringComparison.Ordinal) && line.Contains("serial=" + DiskSerial(token), StringComparison.Ordinal)) || !lines.Contains("local,id=installstatefs,path=" + FullState + ",security_model=none")) throw new InvalidOperationException("QEMU did not attach the exact owned raw disk/serial and persistent state share."); var owner = await Command("docker", ["exec", id, "cat", FullState + "/run.owner"], output, "full-share-owner", cancellation); @@ -1105,14 +1113,16 @@ static class NativeDiagnostic || new[] { "KVM=N", "CPU_MODEL=" + CpuModel, "VERSION=14" }.Any(expected => environment.Count(value => value is not null && value.StartsWith(expected.Split('=')[0] + "=", StringComparison.Ordinal)) != 1 || !environment.Contains(expected))) - throw new InvalidOperationException("Created container exceeds the owned unprivileged TCG/Haswell/macOS 14 boundary."); + throw new InvalidOperationException("Created container exceeds the owned unprivileged TCG/Skylake/macOS 14 boundary."); } static async Task CaptureGuest(string id, string output, CancellationToken cancellation, bool full = false, bool final = false, string? token = null) { if (final && token is not null) await CaptureMonitor(id, output, token, cancellation); var logs = await Command("docker", ["logs", "--tail", "3000", id], output, "container", cancellation, requireSuccess: false); - ReportCpuPreflight(output, logs.Output); + var stage = await Command("docker", ["exec", id, "cat", "/run/shm/native-stage.log"], output, "capture-native-stage", cancellation, requireSuccess: false); + ReportCpuPreflight(output, stage.ExitCode == 0 ? stage.Output : logs.Output); + await Command("docker", ["exec", id, "head", "-c", "4096", "/run/shm/kernel-handoffs.log"], output, "capture-kernel-handoffs", cancellation, requireSuccess: false, retainSuccessful: true); var files = new List<(string, string)> { ("proof.log", "guest-proof.log"), ("result.json", "guest-result.json") }; if (full) files.AddRange([("guest-phase.json", "guest-phase.json"), ("full-result.json", "full-result.json"), ("firstboot.log", "firstboot.log"), ("unattended-firstboot.log", "unattended-firstboot.log"), ("install.log", "install.log"), ("apple.log", "apple.log"), ("disk-ownership-ioreg.log", "disk-ownership-ioreg.log"), ("installed-root.plist", "installed-root.plist"), ("apfs-containers.plist", "apfs-containers.plist"), ("physical-store.plist", "physical-store.plist"), ("clt-catalog.log", "clt-catalog.log"), ("clt-install.log", "clt-install.log"), ("clt-sdk.log", "clt-sdk.log")]); foreach (var file in files) @@ -1132,7 +1142,7 @@ static class NativeDiagnostic } // The immutable Recovery image is complete only after this staging marker. // Hash it once instead of rereading the image on every twenty-second poll. - if (logs.Output.Contains("[supported-profile] accelerator=tcg", StringComparison.Ordinal) + if ((logs.Output + stage.Output).Contains("[supported-profile] accelerator=tcg", StringComparison.Ordinal) && !File.Exists(Path.Combine(output, "guest-container-resources.last-success.json"))) await Command("docker", ["exec", id, "sh", "-c", "printf '[qemu]\n'; qemu-system-x86_64 --version | head -n 1; printf '[Recovery hash]\n'; test -f /storage/14/setup.dmg && sha256sum /storage/14/setup.dmg || exit 1; printf '[resources]\n'; df -Pk /storage; cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max 2>/dev/null || true"], output, "guest-container-resources", cancellation, requireSuccess: false, retainSuccessful: true); } @@ -1154,6 +1164,23 @@ static class NativeDiagnostic } } + static int KernelHandoffs(string logs) => logs.Split('\n').Count(line => line.Trim().StartsWith("#[EB|LOG:HANDOFF TO XNU] ", StringComparison.Ordinal)); + + static void ValidateBootProgress() + { + const string handoff = "#[EB|LOG:HANDOFF TO XNU] _\r\n"; + foreach (var (logs, count) in new[] { ("", 0), ("BdsDxe: starting Boot0002\n", 0), (handoff, 1), (handoff + handoff, 2), ("source says \"" + handoff, 0), ("#[EB|LOG:HANDOFF TO XNU-ish] _\n", 0) }) + if (KernelHandoffs(logs) != count) throw new InvalidOperationException("Recovery boot-progress parser accepted missing, quoted or malformed markers."); + } + + static void CheckRecoveryBootProgress(string output) + { + var retained = Path.Combine(output, "capture-kernel-handoffs.last-success.stdout.log"); + var count = KernelHandoffs(File.ReadAllText(File.Exists(retained) ? retained : Path.Combine(output, "container.stdout.log"))); + Save(Path.Combine(output, "recovery-boot-progress.json"), new { kernelHandoffs = count, unexpectedRepeat = count >= 2, capturedUtc = DateTimeOffset.UtcNow }); + if (count >= 2) throw new InvalidOperationException("Recovery returned to kernel boot before readiness; repeated handoff detected. This does not identify the reset cause."); + } + static async Task CapturePressure(string id, string output, string phase, CancellationToken cancellation) { using var snapshotDeadline = CancellationTokenSource.CreateLinkedTokenSource(cancellation);