Add resilient Azure offline transcription backlog

2026-06-15 17:26:34 +02:00
parent b22754ce5d
commit b774ccc375
35 changed files with 2456 additions and 198 deletions
@@ -0,0 +1,27 @@
+# Design
+
+## Current Azure Input Path
+
+The live Azure Speech path does not use MSAL and does not record audio independently. `AzureSpeechStreamingTranscriptionProvider` receives `AudioChunk` values from `StreamingSpeechRecognitionPipeline`, writes them into a Speech SDK `PushAudioInputStream`, and emits SDK transcript events back as `TranscriptionSegment` values. The recording coordinator separately writes the same mixed chunks to the temporary WAV.
+
+## Transcript Markers
+
+Add lightweight transcript marker semantics to `TranscriptionSegment`:
+
+- a marker segment writes its text directly, for example `<Reconnecting... 1/5>`;
+- a later real transcript segment can identify the marker it replaces;
+- the coordinator rewrites that exact marker line using the existing transcript-line reference path.
+
+This keeps markers independent from markdown string matching and prevents a later append from being lost when the marker is replaced.
+
+## Azure Reconnect Loop
+
+When Azure Speech reports a transient connection cancellation, the provider should stop the current Speech SDK session, leave the upstream audio channel unconsumed while disconnected so it naturally buffers, emit reconnect markers, then create a new SDK session and continue reading the buffered audio. Once the next real transcript segment arrives, it replaces the latest reconnect marker.
+
+After configured reconnect attempts are exhausted, Azure should emit a longer disconnect marker that explains recording can continue and transcription will drain when Azure reconnects. The provider should continue retrying.
+
+## Durable Offline Backlog
+
+If a stopped Azure Speech meeting cannot finish draining before `Recording:StopProcessingTimeout`, the coordinator queues the completed temporary WAV and artifact paths in a persisted offline backlog, releases the active recording slot, and keeps the WAV from startup cleanup. This lets the user start more meetings while Azure or the network is still unavailable.
+
+The backlog processor retries queued items in the background. Each item creates a fresh speech recognition pipeline for the original launch profile, streams the queued WAV into that pipeline, rewrites the original transcript file with the finished lines, updates meeting-note and transcript metadata, runs transcript-line workflow transformations, transitions the assistant context through summarizing to finished/error, runs the normal summary pipeline, then removes the backlog item and deletes the temporary WAV. Failed processing leaves the backlog item and WAV in place for a later retry.
@@ -0,0 +1,25 @@
+# Azure Speech Offline Resilience
+
+## Summary
+
+Make Azure Speech transcription resilient to transient and longer network loss by keeping meeting audio capture alive, surfacing reconnect/disconnect state in the transcript note, and draining buffered audio once Azure Speech can be reached again.
+
+## Motivation
+
+When IPv4 connectivity was lost while IPv6 still worked, Azure Speech stopped producing transcript lines but recovered after connectivity returned. The app continued running, which points to Azure Speech connectivity rather than local audio capture. Users need visible status and continued recording instead of silent transcript stalls.
+
+## Scope
+
+- Confirm Azure Speech live transcription consumes only Meeting Assistant's audio channel.
+- Add transcript marker support for reconnect/disconnect status lines.
+- Add Azure Speech transient reconnect markers like `<Reconnecting... n/m>`.
+- Replace a reconnect marker with the next real transcript line when Azure resumes.
+- Keep audio capture and buffering alive while Azure Speech is unavailable.
+- Reset live speaker identity assumptions after Azure reconnects because speaker IDs may change across SDK sessions.
+- Queue stopped Azure meetings durably when the SDK cannot drain before the stop timeout, then replay the recorded WAV and complete transcription/summary after connectivity returns.
+
+## Out Of Scope
+
+- Replacing Azure Speech with local models.
+- Changing the temporary mixed WAV format.
+- Replacing the summary or workflow engines used after an offline replay.
@@ -0,0 +1,78 @@
+## MODIFIED Requirements
+
+### Requirement: Azure Speech can provide streaming transcription
+Meeting Assistant SHALL provide an Azure Speech speech recognition pipeline that streams captured PCM audio to Azure AI Speech through the Azure Speech SDK conversation transcription API.
+
+The Azure Speech adapter SHALL use configurable endpoint, region, language, key, and key environment variable settings.
+
+The Azure Speech adapter SHALL support configurable diarization of intermediate conversation transcription results.
+
+The Azure Speech adapter SHALL bound recognizer shutdown after the input audio stream is closed.
+
+When Azure Speech is the configured speech recognition pipeline, Meeting Assistant SHALL emit live transcript segments from Azure conversation transcription events with Azure speaker IDs when available.
+
+When Azure Speech is the configured speech recognition pipeline, Meeting Assistant SHALL use the live Azure conversation transcription segments as the finished transcript without running pyannote finalization.
+
+When Azure Speech reports a transient connectivity interruption, Meeting Assistant SHALL keep accepting captured audio into the active in-process pipeline buffer.
+
+When Azure Speech is reconnecting, Meeting Assistant SHALL write a transcript marker in the form `<Reconnecting... n/m>`.
+
+When Azure Speech emits the next real transcript segment after reconnecting, Meeting Assistant SHALL replace the latest reconnect marker with that transcript segment.
+
+When Azure Speech cannot reconnect after the configured immediate retry attempts, Meeting Assistant SHALL write a transcript marker explaining that Azure Speech is disconnected, recording can continue, and transcription/summarization will continue after Azure reconnects.
+
+After Azure Speech reconnects through a new SDK session, Meeting Assistant SHALL clear live speaker-label assumptions for future Azure speaker labels.
+
+When Azure Speech is still unavailable after recording stops and transcription cannot drain before the configured stop-processing timeout, Meeting Assistant SHALL persist the stopped meeting as a durable transcription backlog item that references the completed mixed WAV and meeting artifacts.
+
+When a stopped meeting is persisted to the durable transcription backlog, Meeting Assistant SHALL release the active recording slot so another meeting can be recorded while the stopped meeting waits for Azure Speech to become available.
+
+When Azure Speech becomes available again, Meeting Assistant SHALL retry durable backlog items, rewrite the transcript from the recorded WAV, run the normal post-transcription meeting completion and summary flow, and remove the backlog item after successful completion.
+
+When Meeting Assistant starts, it SHALL preserve WAV files that are referenced by durable transcription backlog items instead of deleting them as stale temporary recordings.
+
+#### Scenario: Azure Speech pipeline is configured
+- **WHEN** the configured provider is `azure-speech`
+- **THEN** Meeting Assistant streams captured PCM chunks to Azure AI Speech conversation transcription through the configured speech recognition pipeline without changing recording control or transcript persistence code
+
+#### Scenario: Azure Speech returns speaker IDs
+- **WHEN** Azure Speech conversation transcription returns transcript text with speaker identity
+- **THEN** Meeting Assistant emits ordered transcript segments with those Azure speaker identities
+
+#### Scenario: Azure Speech key is resolved from environment
+- **WHEN** Azure Speech is configured with `KeyEnv`
+- **THEN** Meeting Assistant reads the Azure Speech key from that environment variable
+
+#### Scenario: Azure Speech stream shutdown is bounded
+- **WHEN** the captured audio stream has ended
+- **THEN** Meeting Assistant stops Azure Speech continuous recognition within the configured stop timeout instead of waiting indefinitely
+
+#### Scenario: Azure Speech transcript is finalized from live conversation segments
+- **WHEN** the configured provider is `azure-speech` and live Azure transcript segments exist
+- **THEN** Meeting Assistant uses the live Azure conversation transcript segments as the finished transcript
+
+#### Scenario: Azure reconnect marker is replaced by next transcript line
+- **GIVEN** Azure Speech reports a transient connectivity interruption
+- **WHEN** Meeting Assistant writes `<Reconnecting... 1/5>` and Azure later emits a transcript segment
+- **THEN** Meeting Assistant replaces the reconnect marker with the transcript segment
+- **AND** keeps later transcript lines intact
+
+#### Scenario: Azure disconnected marker keeps meeting recording alive
+- **GIVEN** Azure Speech cannot reconnect after the configured immediate retry attempts
+- **WHEN** meeting audio continues to be captured
+- **THEN** Meeting Assistant writes a transcript marker explaining that Azure Speech is disconnected
+- **AND** keeps buffering captured audio in the active process for later transcription
+
+#### Scenario: Stopped Azure meeting is queued durably while offline
+- **GIVEN** Azure Speech cannot finish transcription before the recording stop-processing timeout
+- **WHEN** the user stops the meeting
+- **THEN** Meeting Assistant persists a durable backlog item for the stopped meeting
+- **AND** keeps the completed mixed WAV referenced by that backlog item
+- **AND** returns to an idle recording state so another meeting can start
+
+#### Scenario: Durable Azure backlog resumes after connectivity returns
+- **GIVEN** a stopped Azure meeting exists in the durable transcription backlog
+- **WHEN** Azure Speech can transcribe the recorded WAV
+- **THEN** Meeting Assistant rewrites the transcript from the recorded WAV
+- **AND** runs meeting completion and summarization
+- **AND** removes the durable backlog item and its temporary WAV after successful completion
@@ -0,0 +1,11 @@
+# Tasks
+
+- [x] Inspect Azure Speech live input path and confirm whether it uses separate recording/MSAL.
+- [x] Add OpenSpec requirements for Azure reconnect/disconnect transcript markers and buffered audio.
+- [x] Add transcript marker replacement tests through the recording coordinator.
+- [x] Add Azure reconnect marker emission and SDK-session restart behavior.
+- [x] Reset live speaker assumptions after Azure reconnect markers.
+- [x] Document Azure resilience behavior.
+- [x] Run focused tests, full solution tests, and `openspec validate azure-speech-offline-resilience --strict`.
+- [x] Add a durable offline backlog for stopped Azure meetings across process restarts.
+- [x] Replay queued WAV files through a fresh speech pipeline and finish transcript metadata, meeting context state, summary generation, and backlog cleanup.