feat: add macOS meeting audio capture

2026-08-06 08:16:54 +02:00
parent b3aeef24eb
commit 59ba70ca11
15 changed files with 1282 additions and 4 deletions
@@ -0,0 +1,49 @@
+## Context
+
+The recording pipeline already depends on `IMeetingAudioSource` and composes independent microphone and system streams through `CompositeMeetingAudioSource`. Windows supplies those streams with NAudio behind the `WINDOWS` compilation boundary. The portable target currently supplies an unavailable source on every non-Windows host.
+
+.NET does not ship ScreenCaptureKit or AVFoundation bindings. A small native process is therefore the narrowest reliable adapter between Apple's capture frameworks and the existing managed PCM stream contract.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Capture the default macOS microphone and computer output without a virtual audio device.
+- Emit the configured sample rate/channel count as signed 16-bit PCM.
+- Reuse the existing managed alignment, AEC, gain, mixing, WAV, and transcription path.
+- Keep Windows source files, registrations, packages, and runtime behavior unchanged.
+- Fail with actionable privacy-permission or helper-packaging diagnostics.
+
+**Non-Goals:**
+
+- Add a macOS tray icon, global hotkeys, Outlook metadata, notifications, or screenshots.
+- Add runtime macOS microphone selection in this change.
+- Replace or refactor the Windows NAudio implementation.
+- Commit long-lived user meeting audio; the proof WAV is a short implementation artifact only.
+
+## Decisions
+
+### Reuse the existing composite source
+
+macOS registers separate microphone and system `IMeetingAudioSource` adapters and feeds them to `CompositeMeetingAudioSource`. This preserves the established audio semantics and keeps platform code limited to capture and PCM conversion.
+
+### Use one native helper with two modes
+
+The bundled Swift helper accepts `microphone` or `system`, plus sample rate and channel count, and writes headerless signed 16-bit PCM to standard output. AVFoundation supplies microphone buffers. ScreenCaptureKit supplies system-audio buffers. AVAudioConverter normalizes both sources before managed code reads them.
+
+The managed adapter owns process lifetime, reads PCM chunks asynchronously, terminates the helper when capture is cancelled, and surfaces native diagnostics when the helper exits unexpectedly.
+
+### Build the helper only on macOS
+
+The Swift source is portable content, but compilation is conditioned on the build host being macOS. macOS build and publish output receives the executable under `Native`. Windows compilation remains within its existing `WINDOWS` branch and does not compile or invoke Swift code.
+
+### Keep unsupported-host behavior
+
+The portable target chooses macOS capture with a runtime OS check. Linux and other hosts continue resolving `UnavailableMeetingAudioSource` with an updated host-neutral error.
+
+## Risks / Trade-offs
+
+- [macOS privacy controls can initially deny or pause capture] → Request microphone authorization and report explicit Screen Recording/System Audio or Microphone guidance on stderr.
+- [A helper process adds a packaging boundary] → Make the build fail on macOS if Swift compilation fails and verify the helper exists in build and publish output.
+- [Native audio buffer formats vary] → Convert the CoreMedia/AVFoundation buffers with `AVAudioConverter` to the exact requested PCM format before writing.
+- [Windows regressions from shared startup edits] → Leave the `#if WINDOWS` branch intact and build the Windows target in addition to portable tests.
@@ -0,0 +1,52 @@
+## Implementation Evidence
+
+Date: 2026-07-21
+Host: macOS arm64
+
+### Live audio verification
+
+- The bundled Swift helper compiled as an arm64 Mach-O executable and was launched directly from the portable application build output.
+- AVFoundation microphone capture emitted 546,920 bytes of signed 16-bit, 16 kHz mono PCM during the final 17-second proof window. The selected default input was `Jabra Link 390`; its captured samples were all zero, consistent with a muted/silent input rather than a capture-process failure.
+- ScreenCaptureKit system capture emitted 551,680 bytes of signed 16-bit, 16 kHz mono PCM while the repository's known `sample-16khz-mono.wav` fixture was played 12 times through the default system output.
+- The captured streams were aligned to their shared duration, summed with 16-bit clamping, and written to `docs/evidence/macos-meeting-audio-proof.wav`.
+- `afinfo` reports the proof as 17.091250 seconds, mono, 16000 Hz, signed 16-bit PCM, with 546,920 audio bytes.
+- Signal inspection found 48,133 non-zero samples and a peak amplitude of 12,000 in the proof WAV.
+- SHA-256: `f38285a5386067f0c2637698f2e3c720ad30b65cebaaefe8d00454c78d8c2d05`.
+
+### Behavior and packaging verification
+
+- Public recording endpoint behavior:
+  - Test: `RecordingEndpointsCaptureMixedMacOsAudioAndStopNativeProcesses` calls `/recording/start`, observes known microphone/system samples combined into one `12000` PCM sample at the speech-pipeline boundary, calls `/recording/stop`, and verifies both native capture processes were terminated.
+  - The test uses deterministic process-boundary audio because automated test hosts cannot depend on live devices or macOS privacy state.
+- Focused macOS-source, public endpoint, and existing mixer tests:
+  - Command: `dotnet test MeetingAssistant.Tests/MeetingAssistant.Tests.csproj --filter 'FullyQualifiedName~MacOsMeetingAudioSourceTests|FullyQualifiedName~AudioMixingTests' --no-restore --nologo`
+  - Result: passed, 11/11.
+- Portable application build:
+  - Command: `dotnet build MeetingAssistant/MeetingAssistant.csproj -f net10.0 --no-restore --nologo`
+  - Result: succeeded with 0 warnings and 0 errors; the Swift helper was compiled into the application output.
+- macOS publish:
+  - Command: `dotnet publish MeetingAssistant/MeetingAssistant.csproj -f net10.0 -r osx-arm64 --self-contained false -p:EnableWindowsTargeting=true -o tmp/macos-publish-proof --nologo`
+  - Result: succeeded; `Native/macos-meeting-audio-capture` is an executable arm64 Mach-O file in publish output.
+- OpenSpec validation:
+  - Command: `openspec validate add-macos-meeting-audio-capture --strict`
+  - Result: valid.
+
+### Windows isolation verification
+
+- The existing `#if WINDOWS` registration continues to use `MicrophoneAudioSource`, `SystemAudioSource`, `AdaptiveFilterAcousticEchoCancellerFactory`, and `CompositeMeetingAudioSource`. macOS selection exists only inside the portable `#else` branch.
+- Windows C# compilation:
+  - Command: `dotnet msbuild MeetingAssistant/MeetingAssistant.csproj -t:Compile -p:TargetFramework=net10.0-windows10.0.19041.0 -p:EnableWindowsTargeting=true -nologo -v:minimal`
+  - Result: succeeded.
+- A full Windows-target build cannot finish on macOS because the Windows App SDK attempts to execute `MakePri.exe` and returns `Exec format error`. This occurs after managed compilation and is the repository's known cross-host limitation; Windows CI or a Windows host remains the full binary verification environment.
+
+### Full portable test result
+
+- Command: `dotnet test MeetingAssistant.Tests/MeetingAssistant.Tests.csproj --no-restore --nologo`
+- Result: 425 passed and 10 failed in the combined run. One unrelated inactivity-timer test was then rerun in isolation and passed (1/1), leaving the effective result at 426 passing tests and the same 9 unrelated baseline failures.
+- The 9 failures are the same pre-existing macOS baseline recorded by the active `detect-host-operating-system` change: Windows/GDI+ image rendering, Windows-path expectations, and Windows user-environment behavior. None exercises macOS audio capture or the shared mixer.
+
+### Operational verification boundary
+
+The live helpers were verified while the macOS display session was awake and produced the committed proof signal. The public endpoint path was verified deterministically in-process through the real registration, macOS source adapters, shared mixer, coordinator, and start/stop endpoints. A later attempt to repeat live ScreenCaptureKit capture under the transient `vstest`/console proof host could not enumerate a display after the built-in display entered the asleep state; ScreenCaptureKit returned `ScreenCaptureKit could not find a display for system-audio capture`. The production port-5090 service was deliberately not restarted or reconfigured with fake ASR dependencies. These are the reasons live native input and the public application path were verified in two complementary passes rather than one production meeting run.
+
+The live Meeting Assistant service was not restarted, and no active recording or meeting artifacts were modified during verification.
@@ -0,0 +1,30 @@
+## Why
+
+The portable Meeting Assistant build runs on macOS, but starting a recording resolves `UnavailableMeetingAudioSource` and fails before any audio reaches transcription. macOS users need the same microphone-plus-computer-output capture contract without changing the established Windows NAudio implementation.
+
+## What Changes
+
+- Add native macOS microphone capture through AVFoundation.
+- Add native macOS computer-output capture through ScreenCaptureKit.
+- Stream both sources as 16-bit PCM into the existing alignment, echo-cancellation, gain, mixing, recording, and transcription pipeline.
+- Select the macOS sources only when the portable build is running on macOS; retain the existing compile-time Windows registrations unchanged.
+- Package a small Swift capture helper in macOS build and publish output.
+- Produce a short mixed WAV from live microphone and system PCM emitted by the implemented native capture helper as operational evidence.
+- Verify the public recording endpoints, managed macOS adapters, mixer, and stop behavior together with deterministic process-boundary audio.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `meeting-recording`: Extend microphone and computer-output recording to macOS while preserving Windows behavior.
+
+## Impact
+
+- Portable application dependency registration.
+- A macOS-native helper executable built from Swift source.
+- macOS Screen Recording/System Audio and Microphone privacy permissions.
+- Recording behavior tests, build verification, and runtime documentation.
@@ -0,0 +1,55 @@
+## MODIFIED Requirements
+
+### Requirement: Recording mode captures microphone and computer output
+Meeting Assistant SHALL capture microphone input and computer output and combine them into one audio stream for transcription on supported Windows and macOS hosts.
+
+Meeting Assistant SHALL capture audio as 16 kHz mono PCM chunks for the existing recording and transcription pipeline.
+
+Meeting Assistant SHALL capture microphone and system loopback as separate input streams before producing the final mono chunks.
+
+On Windows, Meeting Assistant SHALL retain the existing NAudio microphone and WASAPI loopback implementation.
+
+On macOS, Meeting Assistant SHALL capture the default microphone through AVFoundation and computer output through ScreenCaptureKit without requiring a virtual audio device.
+
+The macOS capture adapter SHALL convert both native sources to signed 16-bit PCM using the active run's configured sample rate and channel count before passing chunks to the existing managed mixing pipeline.
+
+The macOS capture adapter SHALL stop its native capture process when the recording capture token is cancelled.
+
+When macOS privacy permission is missing or denied, Meeting Assistant SHALL fail capture with an actionable message identifying the required Microphone or Screen Recording/System Audio permission.
+
+Meeting Assistant SHALL clean the microphone stream with a local acoustic echo cancellation stage that uses system loopback as the far-end reference.
+
+Meeting Assistant SHALL produce final mono chunks by adding the cleaned microphone samples and system samples.
+
+Meeting Assistant SHALL align microphone and system samples through per-source buffers before producing final chunks and SHALL apply the existing configured gains and clamping behavior.
+
+Meeting Assistant SHALL write only the mixed stream to the temporary WAV used by transcription and finalization.
+
+#### Scenario: macOS records both meeting-audio sources
+- **GIVEN** the portable build is running on macOS
+- **AND** Microphone and Screen Recording/System Audio permissions are granted
+- **WHEN** a meeting recording starts
+- **THEN** Meeting Assistant captures microphone audio through AVFoundation
+- **AND** captures computer output through ScreenCaptureKit
+- **AND** passes both signed 16-bit PCM streams through the existing mixer
+
+#### Scenario: macOS capture uses run audio format
+- **GIVEN** an active macOS recording configures a sample rate and channel count
+- **WHEN** the native capture helpers start
+- **THEN** both helpers emit signed 16-bit PCM with that sample rate and channel count
+
+#### Scenario: macOS recording stops native capture
+- **GIVEN** macOS microphone and system capture are active
+- **WHEN** Meeting Assistant stops capture
+- **THEN** it terminates both native capture processes
+- **AND** PCM already delivered to the managed pipeline before stop remains available to the existing recording pipeline
+
+#### Scenario: macOS privacy permission is unavailable
+- **WHEN** macOS denies microphone or Screen Recording/System Audio capture permission
+- **THEN** Meeting Assistant reports which macOS privacy permission is required
+
+#### Scenario: Windows audio capture remains isolated
+- **GIVEN** Meeting Assistant is compiled for the Windows target
+- **WHEN** the application registers and starts meeting audio capture
+- **THEN** it uses the existing NAudio microphone and WASAPI loopback sources
+- **AND** does not invoke or require the macOS native helper
@@ -0,0 +1,13 @@
+## 1. macOS Capture Behavior
+
+- [x] 1.1 Specify macOS microphone and computer-output capture behavior and Windows isolation.
+- [x] 1.2 Add failing behavior tests for macOS PCM streaming, platform registration, and the public recording endpoints.
+- [x] 1.3 Implement the managed macOS audio-source and process-lifetime adapter.
+- [x] 1.4 Implement and package the AVFoundation/ScreenCaptureKit Swift helper.
+- [x] 1.5 Register the macOS composite source without changing the Windows registration path.
+
+## 2. Verification and Documentation
+
+- [x] 2.1 Document macOS permissions, build packaging, and recording behavior.
+- [x] 2.2 Capture and inspect a short example mixed WAV through the new implementation.
+- [x] 2.3 Run focused tests, portable tests/build, Windows-target regression compilation, strict OpenSpec validation, and refactoring/code review.