Add resilient Azure offline transcription backlog

2026-06-15 17:26:34 +02:00
parent b22754ce5d
commit b774ccc375
35 changed files with 2456 additions and 198 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-06-12
@@ -0,0 +1,18 @@
+# Design
+
+## Tray Menu
+
+`MeetingTaskbarMenuBuilder` will add a stable `Exit` action after the existing recording controls. Keeping this in the menu builder lets tests assert the visible menu behavior without depending on the Windows tray implementation.
+
+## Exit Confirmation
+
+The Windows tray service will handle `Exit` by checking the current `RecordingStatus.State`:
+
+- `Idle` exits immediately.
+- `Recording` or `Summarizing` prompts for confirmation before exiting.
+
+The existing `Summarizing` state represents stopped-run finalization, including transcript drain, speaker recognition, and summary generation. `Recording` represents active capture plus live transcription. If the user confirms, the tray service asks `IHostApplicationLifetime` to stop the application.
+
+## Windows UI Boundary
+
+The confirmation dialog is only needed in the Windows tray implementation. A small method local to `UnoTaskbarIconService` can use WPF `MessageBox` so tests can continue covering platform-independent menu construction through `MeetingTaskbarMenuBuilder`.
@@ -0,0 +1,21 @@
+# Add Tray Exit
+
+## Summary
+
+Add an Exit action to the Windows taskbar icon menu so Meeting Assistant can be shut down from the tray. If Meeting Assistant is recording or still processing a stopped meeting's transcript or summary, Exit should ask for confirmation before stopping the application.
+
+## Motivation
+
+The tray icon is the normal local control surface for Meeting Assistant, but it currently cannot close the application. Because the app may be doing live transcription or post-recording summary work after capture stops, an accidental exit can interrupt useful meeting processing.
+
+## Scope
+
+- Add an Exit item to the tray context menu in all recording states.
+- Stop the application when Exit is selected and Meeting Assistant is idle.
+- Show a confirmation dialog before exiting while recording, transcribing, speaker recognition, or summarization is still in progress.
+
+## Out Of Scope
+
+- Changing shutdown semantics for the recording coordinator.
+- Adding background resume of interrupted transcription or summary work.
+- Restarting or otherwise managing the local service process.
@@ -0,0 +1,58 @@
+## MODIFIED Requirements
+
+### Requirement: Windows taskbar icon controls recording
+Meeting Assistant SHALL show a Windows taskbar notification icon when running on Windows.
+
+The taskbar icon SHALL indicate whether the newest meeting process is idle, actively recording, or post-recording processing/summarizing.
+
+When a new meeting is actively recording while an older stopped meeting is still transcribing, recognizing speakers, or summarizing, the taskbar icon SHALL show the new active recording state.
+
+The taskbar icon right-click menu SHALL expose recording controls based on the current state and configured launch profiles.
+
+The taskbar icon right-click menu SHALL expose an Exit action in every recording state.
+
+When Meeting Assistant is idle or only processing older stopped meetings, the menu SHALL allow starting a meeting recording for each configured launch profile.
+
+When a meeting is actively recording, the menu SHALL allow stopping the recording and continuing transcription/summary generation.
+
+When a meeting is actively recording, the menu SHALL allow canceling the recording and discarding that run's artifacts.
+
+When a meeting is actively recording, the menu SHALL allow switching to each configured launch profile other than the current active profile.
+
+Selecting Exit while Meeting Assistant is idle SHALL stop the application without an additional confirmation prompt.
+
+Selecting Exit while Meeting Assistant is recording, transcribing, recognizing speakers, or summarizing SHALL show a confirmation dialog before stopping the application.
+
+#### Scenario: Idle tray menu can start configured profiles
+- **GIVEN** launch profiles `default` and `english` are configured
+- **AND** no meeting recording is active
+- **WHEN** the taskbar menu is opened
+- **THEN** it offers start recording actions for `default` and `english`
+
+#### Scenario: Recording tray menu exposes stop, cancel, and profile switches
+- **GIVEN** launch profiles `default` and `english` are configured
+- **AND** a meeting is actively recording with profile `default`
+- **WHEN** the taskbar menu is opened
+- **THEN** it offers stop and cancel actions
+- **AND** it offers switching to `english`
+- **AND** it does not offer switching to `default`
+
+#### Scenario: Active recording has priority over older summarizing runs
+- **GIVEN** an older meeting is still summarizing
+- **WHEN** a newer meeting is actively recording
+- **THEN** the taskbar icon indicates recording
+
+#### Scenario: Tray menu always exposes Exit
+- **GIVEN** Meeting Assistant is running
+- **WHEN** the taskbar menu is opened
+- **THEN** it offers an Exit action
+
+#### Scenario: Idle Exit stops immediately
+- **GIVEN** no recording, transcription, speaker recognition, or summary work is running
+- **WHEN** the user selects Exit from the taskbar menu
+- **THEN** Meeting Assistant stops the application without an additional confirmation prompt
+
+#### Scenario: In-progress Exit asks for confirmation
+- **GIVEN** Meeting Assistant is recording, transcribing, recognizing speakers, or summarizing
+- **WHEN** the user selects Exit from the taskbar menu
+- **THEN** Meeting Assistant asks for confirmation before stopping the application
@@ -0,0 +1,7 @@
+# Tasks
+
+- [x] Add requirement scenarios for tray Exit and in-progress confirmation.
+- [x] Add a failing behavior test proving the tray menu exposes Exit.
+- [x] Add the Exit tray menu item.
+- [x] Implement Windows tray Exit handling with confirmation for non-idle states.
+- [x] Run focused taskbar tests and `openspec validate add-tray-exit --strict`.
@@ -0,0 +1,27 @@
+# Design
+
+## Current Azure Input Path
+
+The live Azure Speech path does not use MSAL and does not record audio independently. `AzureSpeechStreamingTranscriptionProvider` receives `AudioChunk` values from `StreamingSpeechRecognitionPipeline`, writes them into a Speech SDK `PushAudioInputStream`, and emits SDK transcript events back as `TranscriptionSegment` values. The recording coordinator separately writes the same mixed chunks to the temporary WAV.
+
+## Transcript Markers
+
+Add lightweight transcript marker semantics to `TranscriptionSegment`:
+
+- a marker segment writes its text directly, for example `<Reconnecting... 1/5>`;
+- a later real transcript segment can identify the marker it replaces;
+- the coordinator rewrites that exact marker line using the existing transcript-line reference path.
+
+This keeps markers independent from markdown string matching and prevents a later append from being lost when the marker is replaced.
+
+## Azure Reconnect Loop
+
+When Azure Speech reports a transient connection cancellation, the provider should stop the current Speech SDK session, leave the upstream audio channel unconsumed while disconnected so it naturally buffers, emit reconnect markers, then create a new SDK session and continue reading the buffered audio. Once the next real transcript segment arrives, it replaces the latest reconnect marker.
+
+After configured reconnect attempts are exhausted, Azure should emit a longer disconnect marker that explains recording can continue and transcription will drain when Azure reconnects. The provider should continue retrying.
+
+## Durable Offline Backlog
+
+If a stopped Azure Speech meeting cannot finish draining before `Recording:StopProcessingTimeout`, the coordinator queues the completed temporary WAV and artifact paths in a persisted offline backlog, releases the active recording slot, and keeps the WAV from startup cleanup. This lets the user start more meetings while Azure or the network is still unavailable.
+
+The backlog processor retries queued items in the background. Each item creates a fresh speech recognition pipeline for the original launch profile, streams the queued WAV into that pipeline, rewrites the original transcript file with the finished lines, updates meeting-note and transcript metadata, runs transcript-line workflow transformations, transitions the assistant context through summarizing to finished/error, runs the normal summary pipeline, then removes the backlog item and deletes the temporary WAV. Failed processing leaves the backlog item and WAV in place for a later retry.
@@ -0,0 +1,25 @@
+# Azure Speech Offline Resilience
+
+## Summary
+
+Make Azure Speech transcription resilient to transient and longer network loss by keeping meeting audio capture alive, surfacing reconnect/disconnect state in the transcript note, and draining buffered audio once Azure Speech can be reached again.
+
+## Motivation
+
+When IPv4 connectivity was lost while IPv6 still worked, Azure Speech stopped producing transcript lines but recovered after connectivity returned. The app continued running, which points to Azure Speech connectivity rather than local audio capture. Users need visible status and continued recording instead of silent transcript stalls.
+
+## Scope
+
+- Confirm Azure Speech live transcription consumes only Meeting Assistant's audio channel.
+- Add transcript marker support for reconnect/disconnect status lines.
+- Add Azure Speech transient reconnect markers like `<Reconnecting... n/m>`.
+- Replace a reconnect marker with the next real transcript line when Azure resumes.
+- Keep audio capture and buffering alive while Azure Speech is unavailable.
+- Reset live speaker identity assumptions after Azure reconnects because speaker IDs may change across SDK sessions.
+- Queue stopped Azure meetings durably when the SDK cannot drain before the stop timeout, then replay the recorded WAV and complete transcription/summary after connectivity returns.
+
+## Out Of Scope
+
+- Replacing Azure Speech with local models.
+- Changing the temporary mixed WAV format.
+- Replacing the summary or workflow engines used after an offline replay.
@@ -0,0 +1,78 @@
+## MODIFIED Requirements
+
+### Requirement: Azure Speech can provide streaming transcription
+Meeting Assistant SHALL provide an Azure Speech speech recognition pipeline that streams captured PCM audio to Azure AI Speech through the Azure Speech SDK conversation transcription API.
+
+The Azure Speech adapter SHALL use configurable endpoint, region, language, key, and key environment variable settings.
+
+The Azure Speech adapter SHALL support configurable diarization of intermediate conversation transcription results.
+
+The Azure Speech adapter SHALL bound recognizer shutdown after the input audio stream is closed.
+
+When Azure Speech is the configured speech recognition pipeline, Meeting Assistant SHALL emit live transcript segments from Azure conversation transcription events with Azure speaker IDs when available.
+
+When Azure Speech is the configured speech recognition pipeline, Meeting Assistant SHALL use the live Azure conversation transcription segments as the finished transcript without running pyannote finalization.
+
+When Azure Speech reports a transient connectivity interruption, Meeting Assistant SHALL keep accepting captured audio into the active in-process pipeline buffer.
+
+When Azure Speech is reconnecting, Meeting Assistant SHALL write a transcript marker in the form `<Reconnecting... n/m>`.
+
+When Azure Speech emits the next real transcript segment after reconnecting, Meeting Assistant SHALL replace the latest reconnect marker with that transcript segment.
+
+When Azure Speech cannot reconnect after the configured immediate retry attempts, Meeting Assistant SHALL write a transcript marker explaining that Azure Speech is disconnected, recording can continue, and transcription/summarization will continue after Azure reconnects.
+
+After Azure Speech reconnects through a new SDK session, Meeting Assistant SHALL clear live speaker-label assumptions for future Azure speaker labels.
+
+When Azure Speech is still unavailable after recording stops and transcription cannot drain before the configured stop-processing timeout, Meeting Assistant SHALL persist the stopped meeting as a durable transcription backlog item that references the completed mixed WAV and meeting artifacts.
+
+When a stopped meeting is persisted to the durable transcription backlog, Meeting Assistant SHALL release the active recording slot so another meeting can be recorded while the stopped meeting waits for Azure Speech to become available.
+
+When Azure Speech becomes available again, Meeting Assistant SHALL retry durable backlog items, rewrite the transcript from the recorded WAV, run the normal post-transcription meeting completion and summary flow, and remove the backlog item after successful completion.
+
+When Meeting Assistant starts, it SHALL preserve WAV files that are referenced by durable transcription backlog items instead of deleting them as stale temporary recordings.
+
+#### Scenario: Azure Speech pipeline is configured
+- **WHEN** the configured provider is `azure-speech`
+- **THEN** Meeting Assistant streams captured PCM chunks to Azure AI Speech conversation transcription through the configured speech recognition pipeline without changing recording control or transcript persistence code
+
+#### Scenario: Azure Speech returns speaker IDs
+- **WHEN** Azure Speech conversation transcription returns transcript text with speaker identity
+- **THEN** Meeting Assistant emits ordered transcript segments with those Azure speaker identities
+
+#### Scenario: Azure Speech key is resolved from environment
+- **WHEN** Azure Speech is configured with `KeyEnv`
+- **THEN** Meeting Assistant reads the Azure Speech key from that environment variable
+
+#### Scenario: Azure Speech stream shutdown is bounded
+- **WHEN** the captured audio stream has ended
+- **THEN** Meeting Assistant stops Azure Speech continuous recognition within the configured stop timeout instead of waiting indefinitely
+
+#### Scenario: Azure Speech transcript is finalized from live conversation segments
+- **WHEN** the configured provider is `azure-speech` and live Azure transcript segments exist
+- **THEN** Meeting Assistant uses the live Azure conversation transcript segments as the finished transcript
+
+#### Scenario: Azure reconnect marker is replaced by next transcript line
+- **GIVEN** Azure Speech reports a transient connectivity interruption
+- **WHEN** Meeting Assistant writes `<Reconnecting... 1/5>` and Azure later emits a transcript segment
+- **THEN** Meeting Assistant replaces the reconnect marker with the transcript segment
+- **AND** keeps later transcript lines intact
+
+#### Scenario: Azure disconnected marker keeps meeting recording alive
+- **GIVEN** Azure Speech cannot reconnect after the configured immediate retry attempts
+- **WHEN** meeting audio continues to be captured
+- **THEN** Meeting Assistant writes a transcript marker explaining that Azure Speech is disconnected
+- **AND** keeps buffering captured audio in the active process for later transcription
+
+#### Scenario: Stopped Azure meeting is queued durably while offline
+- **GIVEN** Azure Speech cannot finish transcription before the recording stop-processing timeout
+- **WHEN** the user stops the meeting
+- **THEN** Meeting Assistant persists a durable backlog item for the stopped meeting
+- **AND** keeps the completed mixed WAV referenced by that backlog item
+- **AND** returns to an idle recording state so another meeting can start
+
+#### Scenario: Durable Azure backlog resumes after connectivity returns
+- **GIVEN** a stopped Azure meeting exists in the durable transcription backlog
+- **WHEN** Azure Speech can transcribe the recorded WAV
+- **THEN** Meeting Assistant rewrites the transcript from the recorded WAV
+- **AND** runs meeting completion and summarization
+- **AND** removes the durable backlog item and its temporary WAV after successful completion
@@ -0,0 +1,11 @@
+# Tasks
+
+- [x] Inspect Azure Speech live input path and confirm whether it uses separate recording/MSAL.
+- [x] Add OpenSpec requirements for Azure reconnect/disconnect transcript markers and buffered audio.
+- [x] Add transcript marker replacement tests through the recording coordinator.
+- [x] Add Azure reconnect marker emission and SDK-session restart behavior.
+- [x] Reset live speaker assumptions after Azure reconnect markers.
+- [x] Document Azure resilience behavior.
+- [x] Run focused tests, full solution tests, and `openspec validate azure-speech-offline-resilience --strict`.
+- [x] Add a durable offline backlog for stopped Azure meetings across process restarts.
+- [x] Replay queued WAV files through a fresh speech pipeline and finish transcript metadata, meeting context state, summary generation, and backlog cleanup.
@@ -0,0 +1,32 @@
+# Design
+
+## Live Transcript Ordering
+
+`MeetingRecordingCoordinator` will treat the formatted transcript line as the durable fallback. For each live segment:
+
+1. Format and relabel the segment.
+2. Append the formatted line to the transcript store immediately.
+3. Keep the returned transcript-line reference for that exact appended line.
+4. Queue `transcript_line` workflow processing out of band so later live segments can continue to be appended.
+5. If workflow processing returns a different line, ask the transcript store to replace the referenced line.
+6. If workflow processing fails, log the failure and keep the original appended line.
+
+This preserves the observable transcript even when local automation rules are invalid, slow, or unexpectedly fail after validation.
+
+## Transcript Store Contract
+
+Add a transcript-store operation for replacing one previously written line by stable line reference. The vault-backed implementation serializes transcript-file edits per file and rewrites the referenced body line while preserving frontmatter. Existing bulk replacement remains in place for final diarization and speaker relabeling rewrites.
+
+## Workflow Logging
+
+The workflow engine already logs when a rule is applied. Extend this so logs show:
+
+- rule start with rule name and event type,
+- rule completion with whether the meeting note changed and whether the transcript line changed,
+- rule errors with rule name and event type.
+
+The engine will still throw for general workflow events so non-transcript lifecycle automation failures remain visible to callers. The recording coordinator catches transcript-line workflow failures during live transcript writes so transcript persistence continues.
+
+## Disk-Full Follow-Up
+
+The temporary WAV disk-full exception is tracked as a follow-up task. The expected one-hour file size for the configured `16 kHz / mono / 16-bit` stream is about 110 MiB, so a one-off disk-full error with ample later free space needs separate investigation rather than speculative cleanup behavior.
@@ -0,0 +1,25 @@
+# Resilient Transcript Workflow
+
+## Summary
+
+Make live transcript persistence independent from workflow-rule success. Transcript lines should be written before optional workflow transformations run, changed lines should be rewritten in place, and workflow rule activity/errors should be logged clearly enough to diagnose broken local rules.
+
+## Motivation
+
+A running meeting showed Azure Speech continuing to emit live segments while the transcript markdown stopped advancing. The current append path waits for workflow transformation before writing each line, so a failing or stalled workflow rule can make transcription appear dead even when ASR is still producing output.
+
+A separate earlier run logged a disk-full `IOException` while writing the temporary WAV, despite the expected WAV size being small. That anomaly should be tracked separately because the current evidence does not identify a safe corrective behavior beyond better diagnostics.
+
+## Scope
+
+- Write the formatted live transcript line before transcript-line workflow processing.
+- Rewrite the just-written transcript line if workflow rules change it.
+- Keep transcript writes going when transcript-line workflow rules fail.
+- Log triggered workflow rules, workflow rule completion, and workflow rule errors with event context.
+- Track follow-up investigation for anomalous temporary-recording disk-full failures.
+
+## Out Of Scope
+
+- Changing ASR providers or audio capture format.
+- Restarting or interrupting an active meeting.
+- Automatically deleting files or freeing disk space after a disk-full error.
@@ -0,0 +1,102 @@
+## MODIFIED Requirements
+
+### Requirement: Meeting automation rules support lifecycle triggers
+Meeting Assistant SHALL support rule triggers for `created`, `state_transition`, `speaker_identified`, and `transcript_line`.
+
+A `state_transition` trigger MAY filter by `from`, `to`, or both state values.
+
+A `speaker_identified` trigger MAY filter by speaker name.
+
+A `transcript_line` trigger MAY filter by speaker name.
+
+Workflow rule execution SHALL log each triggered rule with its rule name and event type.
+
+Workflow rule execution SHALL log rule failures with the rule name and event type.
+
+#### Scenario: State transition rule matches from and to
+- **GIVEN** a configured rule that triggers on a state transition from `collecting metadata` to `transcribing`
+- **WHEN** Meeting Assistant transitions that meeting from `collecting metadata` to `transcribing`
+- **THEN** it applies the rule steps
+
+#### Scenario: Speaker identified rule filters by name
+- **GIVEN** a configured rule that triggers when speaker `Ada` is identified
+- **WHEN** Meeting Assistant identifies speaker `Grace`
+- **THEN** it does not apply the rule
+- **WHEN** Meeting Assistant identifies speaker `Ada`
+- **THEN** it applies the rule steps
+
+#### Scenario: Transcript line rule rewrites masked profanity after durable append
+- **GIVEN** a configured rule that triggers on transcript line writes and sets `transcript.line` by replacing `*****` with `[redacted]`
+- **WHEN** Meeting Assistant writes a transcript line for speaker `Guest-1` containing `*****`
+- **THEN** Meeting Assistant first appends the original formatted line to the transcript file
+- **AND** rewrites that written line to contain `[redacted]`
+- **AND** the final written transcript line does not contain `*****`
+
+#### Scenario: Transcript line workflow failure keeps transcript writing
+- **GIVEN** a configured transcript line workflow rule fails while processing a transcript line
+- **WHEN** Meeting Assistant receives that live transcript segment
+- **THEN** Meeting Assistant keeps the original formatted line in the transcript file
+- **AND** logs the workflow rule failure
+- **AND** continues processing later transcript segments
+
+### Requirement: Meeting automation rules support conditions and steps
+Meeting Assistant SHALL support rule conditions using an expression engine.
+
+Rules SHALL support nested `and`, `or`, and `not` condition groups.
+
+Step values SHALL support Razor syntax against the current meeting event model.
+
+Rendered step values SHALL be treated as plain UTF-8 text for markdown artifacts and SHALL NOT persist Razor HTML entity encoding.
+
+Step values SHALL treat `@` characters inside valid email address tokens as literal text rather than Razor transitions.
+
+Meeting Assistant SHALL expose the formatted transcript line and transcript speaker to `transcript_line` rule conditions and Razor step templates.
+
+Meeting Assistant SHALL support these initial rule steps:
+
+- `add_attendee`
+- `remove_attendee`
+- `set_property`
+- `add_context`
+- `add_project`
+
+The `set_property` step SHALL support setting `transcript.line` during `transcript_line` events.
+
+#### Scenario: Nested conditions choose a matching rule
+- **GIVEN** a configured rule with nested `and`, `or`, and `not` conditions over meeting title, attendees, and event data
+- **WHEN** the condition evaluates to true
+- **THEN** Meeting Assistant applies the rule
+- **WHEN** the condition evaluates to false
+- **THEN** Meeting Assistant skips the rule
+
+#### Scenario: Templated context mentions identified speaker
+- **GIVEN** a configured `speaker_identified` rule with an `add_context` step using Razor syntax
+- **WHEN** Meeting Assistant identifies matching speaker `Ada`
+- **THEN** it appends rendered context text containing `Ada` to the assistant context note
+
+#### Scenario: Email addresses do not trigger Razor templating
+- **GIVEN** a configured rule step value containing `Support@contoso.com`
+- **WHEN** the rule runs
+- **THEN** Meeting Assistant preserves the email address as literal text
+
+#### Scenario: Email addresses can appear beside Razor templating
+- **GIVEN** a configured rule step value containing both `Support@contoso.com` and a Razor expression
+- **WHEN** the rule runs
+- **THEN** Meeting Assistant renders the Razor expression and preserves the email address as literal text
+
+#### Scenario: Razor-rendered transcript line preserves UTF-8 text
+- **GIVEN** a configured `transcript_line` rule uses Razor to replace part of a transcript line containing `heißt`, `Schimpfwörter`, and `nächstes`
+- **WHEN** the rule changes `transcript.line`
+- **THEN** the resulting transcript line contains the original UTF-8 characters
+- **AND** does not contain HTML entities such as `&#xDF;`, `&#xF6;`, or `&#xE4;`
+
+#### Scenario: Rule can clean a meeting title
+- **GIVEN** a configured state-transition rule that matches a title containing a configured marker
+- **WHEN** the rule runs
+- **THEN** Meeting Assistant can update the meeting title through `set_property`
+
+#### Scenario: Transcript line conditions can use the written line and speaker
+- **GIVEN** a configured `transcript_line` rule with conditions over `transcript.line` and `transcript.speaker`
+- **WHEN** Meeting Assistant writes a transcript line that matches both conditions
+- **THEN** Meeting Assistant applies the rule steps after the original formatted line is durably appended
+- **AND** rewrites the written line if a rule changes `transcript.line`
@@ -0,0 +1,9 @@
+# Tasks
+
+- [x] Add requirement scenarios for resilient transcript workflow writes and workflow diagnostics.
+- [x] Add a failing behavior test proving transcript text is written when transcript-line workflow rules fail.
+- [x] Append live transcript lines before workflow transformation and rewrite changed lines afterward.
+- [x] Add workflow rule completion/error logging.
+- [x] Update workflow engine documentation.
+- [x] Run focused tests and `openspec validate resilient-transcript-workflow --strict`.
+- [ ] Follow up later: investigate anomalous temporary-recording disk-full `IOException` despite small expected WAV size.