Public Access
feat(macos): add native desktop controls and hotkey integration
- Introduced MacOsDesktopControlManifest to manage desktop control configurations. - Implemented MacOsDesktopControlService to handle the lifecycle of the macOS helper process. - Created MacOsWorkflowRulesEditorWindowService for managing workflow rules through a native window. - Developed a Swift helper for macOS that integrates with AppKit and Carbon for menu-bar controls and global hotkeys. - Added support for recording lifecycle actions via HTTP endpoints, maintaining consistency with existing Windows functionality. - Enhanced user experience by allowing macOS users to start/stop recordings and access settings through a menu-bar interface. - Documented design decisions, implementation evidence, and verification tasks for the new macOS features.
This commit is contained in:
dh
@@ -0,0 +1,23 @@
|
||||
## Context
|
||||
|
||||
The application already exposes local HTTP endpoints for recording lifecycle actions. Windows desktop integrations invoke the coordinator in-process, while macOS currently has no desktop process capable of participating in AppKit or Carbon event loops.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Native helper owns macOS desktop APIs
|
||||
|
||||
A bundled Swift helper owns the AppKit status item and Carbon global-hotkey registrations. The .NET host starts the helper only on macOS and terminates it during shutdown.
|
||||
|
||||
### Existing HTTP endpoints remain the control seam
|
||||
|
||||
The helper invokes loopback endpoints for recording and diagnostic actions. It does not duplicate recording state or lifecycle policy. It polls the status endpoint to keep the menu-bar presentation current.
|
||||
|
||||
### Configuration is passed as an explicit manifest
|
||||
|
||||
The .NET host derives a manifest from configured launch profiles. Shortcut strings are passed unchanged so macOS uses the same bindings displayed and registered on Windows.
|
||||
|
||||
## Verification
|
||||
|
||||
- Behavior tests prove the manifest preserves configured shortcuts and maps actions to the existing endpoints.
|
||||
- The Swift helper supports a describe/self-test mode for deterministic packaging validation.
|
||||
- Runtime verification launches the application, starts a new recording, and captures the visible macOS menu-bar recording state.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Implementation Evidence
|
||||
|
||||
## Automated
|
||||
|
||||
- `dotnet test MeetingAssistant.Tests/MeetingAssistant.Tests.csproj --filter FullyQualifiedName~MacOsDesktopControlManifestTests --no-restore`
|
||||
- Passed: 1, Failed: 0.
|
||||
- `dotnet build MeetingAssistant/MeetingAssistant.csproj -f net10.0 --no-restore`
|
||||
- Passed and produced `Native/macos-desktop-controls` as an arm64 Mach-O executable.
|
||||
- `openspec validate add-macos-desktop-controls --strict`
|
||||
- Passed.
|
||||
- Full portable test project:
|
||||
- Passed: 427.
|
||||
- Failed: 9 existing macOS/environment-dependent tests: Windows path expectations, unavailable GDI+, and user-environment key resolution.
|
||||
- Whole solution:
|
||||
- Cannot build the Windows target on macOS without enabling cross-Windows targeting (`NETSDK1100`).
|
||||
|
||||
## Runtime
|
||||
|
||||
- Installed the rebuilt application assembly and both native helpers into the existing local LaunchAgent application directory.
|
||||
- LaunchAgent restarted successfully.
|
||||
- `/health` returned `ok`.
|
||||
- `macos-desktop-controls` ran as a child companion process.
|
||||
- The helper manifest contained six configured hotkeys, including the unchanged defaults `Ctrl+Alt+M`, `Ctrl+Alt+L`, `Ctrl+Alt+Z`, and `Ctrl+Alt+S`.
|
||||
- A fresh default-profile recording request created the `20260724-2015` meeting note and linked artifacts.
|
||||
- Screenshot evidence: `tmp/macos-desktop-recording-verification.png`.
|
||||
|
||||
## Live desktop-control proof (2026-07-24)
|
||||
|
||||
- The installed LaunchAgent ran:
|
||||
- `.NET host` PID `23023`
|
||||
- native `macos-desktop-controls` PID `23097`
|
||||
- The native helper manifest exposed:
|
||||
- `Ctrl+Alt+M` -> `/profiles/default/recording/toggle`
|
||||
- `Ctrl+Alt+L` -> `/profiles/english/recording/toggle`
|
||||
- `Ctrl+Alt+Z` -> `/profiles/default/recording/abort`
|
||||
- `Ctrl+Alt+S` -> `/profiles/default/meetings/screenshot/capture`
|
||||
- Accessibility inspection of the real AppKit menu returned:
|
||||
- `Meeting Assistant — idle`
|
||||
- `Open agent`
|
||||
- `Start meeting recording (default)`
|
||||
- `Start meeting recording (english)`
|
||||
- `Exit`
|
||||
- Visible menu screenshot:
|
||||
- `tmp/proof-macos-menu-idle.png`
|
||||
- The menu visibly displays the macOS equivalents `⌃⌥M` and `⌃⌥L`.
|
||||
- Clicking `Start meeting recording (default)` through the real menu produced the sampled status transition `false,true,false`, with `launchProfile: default`, and created:
|
||||
- `Meetings/Notes/20260724-2022-note.md`
|
||||
- `Meetings/Transcripts/20260724-2022-transcript.md`
|
||||
- `Meetings/Assistant Context/20260724-2022-context.md`
|
||||
- Posting the HID-level `Ctrl+Alt+M` combination produced the sampled status transition `false,true,false`, with `launchProfile: default`.
|
||||
- Posting the HID-level `Ctrl+Alt+L` combination produced an active sampled state with `launchProfile: english`.
|
||||
- Raw sampled status evidence:
|
||||
- `tmp/proof-menu-recording-status.jsonl`
|
||||
- `tmp/proof-hotkey-recording-status.jsonl`
|
||||
- `tmp/proof-english-hotkey-status.jsonl`
|
||||
|
||||
## Runtime limitation
|
||||
|
||||
The new recording did not remain active because the local audio capture process stopped shortly after artifact creation. The desktop-control helper, menu-bar item, endpoint routing, and meeting creation were verified; sustained audio capture was not.
|
||||
|
||||
## Open agent regression and runtime proof
|
||||
|
||||
- Original deterministic regression:
|
||||
- `MacOsApplicationRegistersARealInteractiveAgentWindow`
|
||||
- Red result: portable registration resolved exactly `NoopWorkflowRulesEditorWindowService`.
|
||||
- Fix:
|
||||
- macOS now registers `MacOsWorkflowRulesEditorWindowService`.
|
||||
- `Open agent` in the native helper creates an AppKit `NSWindow` titled `Meeting Summary Agent`.
|
||||
- The window hosts a WebKit chat page backed by the existing `WorkflowRulesEditorChatViewModel` and agent pipeline.
|
||||
- Green regression:
|
||||
- Passed: 1, Failed: 0.
|
||||
- Live installed runtime:
|
||||
- Clicking the actual menu item created an on-screen CoreGraphics window:
|
||||
- owner: `macos-desktop-controls`
|
||||
- title: `Meeting Summary Agent`
|
||||
- layer: `0`
|
||||
- bounds: `756x714`
|
||||
- `GET /diagnostics/settings-and-logs` returned the interactive agent HTML page.
|
||||
|
||||
## Open agent prompt execution proof (2026-07-25)
|
||||
|
||||
- Original live failure:
|
||||
- `Meeting Summary Agent failed: No workflow rules editor API key configured.`
|
||||
- Root cause:
|
||||
- The LaunchAgent did not inherit the interactive shell's model API key.
|
||||
- Its configured `127.0.0.1:4021` model endpoint had no running service.
|
||||
- Runtime configuration:
|
||||
- The protected LaunchAgent environment file now supplies the local LiteLLM gateway key.
|
||||
- `MeetingAssistant__WorkflowRulesEditor__Endpoint` targets the running local gateway on port `4000`.
|
||||
- The local gateway exposes the Responses-compatible `chatgpt-gpt-5.5` model alias.
|
||||
- Event-stream compatibility regression:
|
||||
- `ClientReconstructsResponseFromServerSentEvents` failed before the fix with `JsonReaderException`.
|
||||
- The client now reconstructs completed response items from LiteLLM server-sent events.
|
||||
- All 11 `LiteLlmResponsesChatClientTests` pass.
|
||||
- Live installed proof:
|
||||
- `POST /diagnostics/settings-and-logs/chat` with `Reply with exactly: key-ok`
|
||||
returned `{"role":"agent","content":"key-ok"}`.
|
||||
- `/health` returned successfully after deployment.
|
||||
@@ -0,0 +1,24 @@
|
||||
## Why
|
||||
|
||||
Meeting Assistant can record audio on macOS, but the portable build does not register the configured global shortcuts or provide the menu-bar controls used for the normal Windows desktop workflow. macOS users must call HTTP endpoints manually.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a native macOS menu-bar companion that starts and stops with Meeting Assistant.
|
||||
- Register the same configured recording, abort, and screenshot shortcuts used on Windows.
|
||||
- Expose recording/profile controls, `Open agent`, and exit from the macOS menu bar.
|
||||
- Keep recording lifecycle behavior behind the existing local HTTP/service surface.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `meeting-recording`: configured global recording controls work on macOS.
|
||||
- `meeting-session`: the desktop control menu is available from the macOS menu bar.
|
||||
|
||||
## Impact
|
||||
|
||||
- `MeetingAssistant/Native/MacOsDesktopControls`
|
||||
- portable macOS build/publish output
|
||||
- startup service registration
|
||||
- desktop-control behavior tests and macOS runtime documentation
|
||||
@@ -0,0 +1,20 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Recording mode is controlled by a configurable hotkey
|
||||
Meeting Assistant SHALL use normal .NET configuration to define global hotkeys that work on Windows and macOS and toggle recording/transcription mode.
|
||||
|
||||
Meeting Assistant SHALL preserve the configured shortcut strings across Windows and macOS, including the defaults `Ctrl+Alt+M`, `Ctrl+Alt+L`, `Ctrl+Alt+Z`, and `Ctrl+Alt+S`.
|
||||
|
||||
Meeting Assistant SHALL expose a configurable abort/discard hotkey for an active recording on Windows and macOS.
|
||||
|
||||
#### Scenario: macOS uses the configured Windows-equivalent shortcuts
|
||||
- **GIVEN** the default and English profiles use `Ctrl+Alt+M` and `Ctrl+Alt+L`
|
||||
- **AND** abort and screenshot use `Ctrl+Alt+Z` and `Ctrl+Alt+S`
|
||||
- **WHEN** Meeting Assistant starts on macOS
|
||||
- **THEN** it registers those same global shortcut combinations
|
||||
- **AND** routes them to toggle, abort, and screenshot actions through the local service surface
|
||||
|
||||
#### Scenario: macOS recording hotkey starts recording
|
||||
- **GIVEN** Meeting Assistant is idle on macOS
|
||||
- **WHEN** the user presses the configured recording hotkey
|
||||
- **THEN** Meeting Assistant starts recording/transcription mode
|
||||
@@ -0,0 +1,28 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Workflow rules and speaker identities can be edited through a tray-launched assistant
|
||||
Meeting Assistant SHALL expose desktop controls from the Windows tray icon and the macOS menu bar.
|
||||
|
||||
The macOS menu-bar control SHALL start with Meeting Assistant, show current recording state, expose each configured launch profile with its configured hotkey, expose stop and abort controls while recording, expose `Open agent`, and expose exit.
|
||||
|
||||
Selecting `Open agent` on macOS SHALL open a native window titled `Meeting Summary Agent` containing the interactive settings-and-logs chat surface.
|
||||
|
||||
#### Scenario: macOS menu bar shows idle recording controls
|
||||
- **GIVEN** Meeting Assistant is idle on macOS
|
||||
- **WHEN** the user opens the menu-bar control
|
||||
- **THEN** it shows `Open agent`
|
||||
- **AND** shows a start action for every configured launch profile
|
||||
- **AND** displays each profile's configured toggle hotkey
|
||||
- **AND** shows an exit action
|
||||
|
||||
#### Scenario: macOS menu bar shows active recording controls
|
||||
- **GIVEN** a meeting is recording on macOS
|
||||
- **WHEN** the user opens the menu-bar control
|
||||
- **THEN** it shows the active recording state
|
||||
- **AND** exposes stop and abort actions
|
||||
|
||||
#### Scenario: macOS menu opens the interactive agent
|
||||
- **GIVEN** Meeting Assistant is running on macOS
|
||||
- **WHEN** the user selects `Open agent` from the menu-bar control
|
||||
- **THEN** a `Meeting Summary Agent` window opens
|
||||
- **AND** the user can submit a chat message and receive the interactive agent response
|
||||
@@ -0,0 +1,18 @@
|
||||
## 1. Contract and behavior
|
||||
|
||||
- [x] 1.1 Define macOS desktop-control requirements and design.
|
||||
- [x] 1.2 Add a failing behavior test for configured shortcut and endpoint preservation.
|
||||
- [x] 1.3 Implement the desktop-control manifest and make the behavior test pass.
|
||||
|
||||
## 2. Native macOS integration
|
||||
|
||||
- [x] 2.1 Add and package the native macOS menu-bar/global-hotkey helper.
|
||||
- [x] 2.2 Start and stop the helper with the .NET application on macOS.
|
||||
- [x] 2.3 Add deterministic helper packaging/self-test coverage.
|
||||
|
||||
## 3. Verification and closeout
|
||||
|
||||
- [x] 3.1 Run narrow and full tests.
|
||||
- [x] 3.2 Run strict OpenSpec validation.
|
||||
- [x] 3.3 Launch Meeting Assistant, start a new recording, and save screenshot evidence.
|
||||
- [x] 3.4 Perform required DRY, SOLID, and KISS refactoring reviews and re-run verification.
|
||||
@@ -375,3 +375,43 @@ After repairing a meeting or summary, the instructions SHALL direct the agent to
|
||||
- **WHEN** the user asks the interactive agent to fix that meeting or its summary
|
||||
- **THEN** the agent instructions direct it to inspect the matching assistant context for relevant meeting-specific memory
|
||||
- **AND** direct it to append its fixes and conclusions to that assistant context
|
||||
|
||||
### Requirement: Interactive agent uses detected host operating-system context
|
||||
Meeting Assistant SHALL detect the host operating system when the application starts and SHALL make that immutable runtime context available to the interactive settings/logs assistant.
|
||||
|
||||
When the detected host is Windows, the interactive assistant instructions SHALL identify Windows as the runtime platform and SHALL direct the assistant to use Windows-specific commands, paths, services, environment conventions, and desktop concepts instead of macOS concepts.
|
||||
|
||||
When the detected host is macOS, the interactive assistant instructions SHALL identify macOS as the runtime platform and SHALL direct the assistant to use macOS-specific commands, POSIX paths, launchd services, environment conventions, and desktop concepts instead of Windows concepts.
|
||||
|
||||
Platform-aware instructions SHALL remain present when the interactive assistant uses a configured custom initial prompt.
|
||||
|
||||
The instructions SHALL distinguish platform-aware operational guidance from feature availability and SHALL NOT imply that detecting macOS provides Windows-only or otherwise unavailable integrations.
|
||||
|
||||
For other detected hosts, Meeting Assistant SHALL identify the platform as unsupported for platform-specific guidance and SHALL direct the assistant to prefer portable tools and concepts rather than assuming Windows or macOS.
|
||||
|
||||
#### Scenario: Windows host receives Windows operational guidance
|
||||
- **GIVEN** Meeting Assistant detected Windows during startup
|
||||
- **WHEN** it builds the interactive settings/logs assistant instructions
|
||||
- **THEN** the instructions identify Windows as the runtime platform
|
||||
- **AND** direct the assistant to use Windows commands and concepts
|
||||
- **AND** direct the assistant not to substitute macOS commands or concepts
|
||||
|
||||
#### Scenario: macOS host receives macOS operational guidance
|
||||
- **GIVEN** Meeting Assistant detected macOS during startup
|
||||
- **WHEN** it builds the interactive settings/logs assistant instructions
|
||||
- **THEN** the instructions identify macOS as the runtime platform
|
||||
- **AND** direct the assistant to use macOS commands and concepts
|
||||
- **AND** direct the assistant not to substitute Windows commands or concepts
|
||||
|
||||
#### Scenario: Custom prompt retains detected platform context
|
||||
- **GIVEN** Meeting Assistant detected macOS during startup
|
||||
- **AND** a custom interactive-assistant initial prompt is configured
|
||||
- **WHEN** it builds the interactive assistant instructions
|
||||
- **THEN** the instructions contain both the custom prompt and macOS runtime guidance
|
||||
|
||||
#### Scenario: Unsupported host uses neutral guidance
|
||||
- **GIVEN** Meeting Assistant runs on a host other than Windows or macOS
|
||||
- **WHEN** it builds the interactive settings/logs assistant instructions
|
||||
- **THEN** the instructions do not label the host as Windows or macOS
|
||||
- **AND** direct the assistant to prefer portable tools and concepts
|
||||
|
||||
|
||||
Reference in New Issue
Block a user