Public Access
Implement meeting assistant v1
This commit is contained in:
@@ -0,0 +1,190 @@
|
||||
# meeting-transcription Specification
|
||||
|
||||
## Purpose
|
||||
TBD - created by archiving change define-meeting-assistant-v1. Update Purpose after archive.
|
||||
## Requirements
|
||||
### Requirement: Meeting Assistant transcribes meetings with speaker attribution
|
||||
Meeting Assistant SHALL transcribe meeting audio and associate transcript segments with speakers when speaker attribution is available.
|
||||
|
||||
The transcription contract SHALL remain independent of the meeting source so that local audio, in-person meetings, and future meeting-platform integrations can use the same downstream processing.
|
||||
|
||||
#### Scenario: Speaker-attributed transcript is produced
|
||||
- **WHEN** meeting audio is transcribed and speaker attribution is available
|
||||
- **THEN** the transcript contains ordered speaker-attributed segments
|
||||
|
||||
#### Scenario: Speaker attribution is unavailable
|
||||
- **WHEN** meeting audio is transcribed but speaker attribution is unavailable
|
||||
- **THEN** Meeting Assistant keeps the transcript and marks speaker identity as unknown rather than discarding the meeting
|
||||
|
||||
### Requirement: Speech recognition pipelines are streamable and configurable
|
||||
Meeting Assistant SHALL route audio through a provider-independent speech recognition pipeline contract.
|
||||
|
||||
The configured pipeline SHALL be selected through DI from the ASR backend configuration and SHALL expose operations to initialize, wait for readiness, write captured audio chunks, read live transcript segments, and read the finished transcript after capture completes.
|
||||
|
||||
Reading the finished transcript SHALL accept optional pipeline options, including a requested speaker count hint.
|
||||
|
||||
For real meeting recordings, Meeting Assistant SHALL derive the requested speaker count hint from the meeting note `attendees` frontmatter when at least two attendees are listed.
|
||||
|
||||
The configured pipeline SHALL receive audio chunks as they are captured rather than requiring the recording coordinator to finish a complete meeting audio file first.
|
||||
|
||||
#### Scenario: Streaming provider receives live audio
|
||||
- **WHEN** recording mode is active and the combined audio source emits a chunk
|
||||
- **THEN** Meeting Assistant can pass that chunk to the configured speech recognition pipeline before the audio source completes
|
||||
|
||||
#### Scenario: Pipeline readiness is separated from recording capture
|
||||
- **WHEN** recording mode starts before the configured speech recognition backend is ready
|
||||
- **THEN** Meeting Assistant starts audio capture immediately and lets the pipeline finish readiness before it emits live transcript output
|
||||
|
||||
#### Scenario: Pipeline is changed by configuration
|
||||
- **WHEN** the configured ASR backend setting is changed
|
||||
- **THEN** Meeting Assistant selects the configured speech recognition pipeline without changing recording control or transcript persistence code
|
||||
|
||||
#### Scenario: ASR backend is tested with a WAV file
|
||||
- **WHEN** the user submits a local PCM WAV file to the ASR diagnostic endpoint
|
||||
- **THEN** Meeting Assistant streams that file through the configured speech recognition pipeline and returns the emitted live transcript segments
|
||||
|
||||
#### Scenario: ASR diagnostic backend is unavailable
|
||||
- **WHEN** the configured speech recognition pipeline fails while processing the diagnostic WAV file
|
||||
- **THEN** Meeting Assistant returns a diagnostic backend failure response instead of hiding the pipeline error
|
||||
|
||||
#### Scenario: Final diarization backend is tested with a WAV file
|
||||
- **WHEN** the user submits a local PCM WAV file to the final diarization diagnostic endpoint
|
||||
- **THEN** Meeting Assistant streams the file through the configured speech recognition pipeline and returns the finished transcript segments
|
||||
|
||||
#### Scenario: Final diarization receives a speaker count hint
|
||||
- **WHEN** the user submits a local PCM WAV file and `numSpeakers` to the final diarization diagnostic endpoint
|
||||
- **THEN** Meeting Assistant passes the requested speaker count to the configured speech recognition pipeline as a finished-transcript option
|
||||
|
||||
#### Scenario: Meeting attendee count provides final speaker hint
|
||||
- **WHEN** a meeting note has two or more entries in `attendees`
|
||||
- **THEN** Meeting Assistant passes that attendee count to the configured speech recognition pipeline as the finished-transcript speaker count hint
|
||||
|
||||
#### Scenario: Empty or single-attendee meeting note has no final speaker hint
|
||||
- **WHEN** a meeting note has zero or one entry in `attendees`
|
||||
- **THEN** Meeting Assistant leaves the finished-transcript speaker count hint unset
|
||||
|
||||
### Requirement: Version 1 keeps local Whisper transcription available
|
||||
Meeting Assistant SHALL provide a local Whisper speech recognition pipeline as a fallback ASR backend for version 1.
|
||||
|
||||
The local Whisper adapter SHALL use configurable model path, language, and streaming window settings.
|
||||
|
||||
When local Whisper is the configured speech recognition pipeline, Meeting Assistant SHALL be able to run a final pyannote diarization pass over the temporary mixed audio and either assign pyannote speaker turns back to live Whisper transcript segments by timestamp overlap or build finished transcript segments from pyannote speaker turns.
|
||||
|
||||
The pyannote finalization backend SHALL allow configuration to use either pyannote's normal `speaker_diarization` annotation or pyannote's `exclusive_speaker_diarization` annotation.
|
||||
|
||||
When a speaker count hint is provided through the speech recognition pipeline options, the pyannote finalization backend SHALL pass it to pyannote as `num_speakers`.
|
||||
|
||||
The pyannote finalization backend SHALL use a reusable runtime image for Python dependencies and a persistent configured cache folder for Hugging Face models, torch artifacts, and pip artifacts so repeated finalization runs do not redownload the diarization model or reinstall the full Python dependency stack.
|
||||
|
||||
#### Scenario: Local Whisper model is configured
|
||||
- **WHEN** the configured provider is `whisper-local`
|
||||
- **THEN** Meeting Assistant uses the configured local Whisper model path to transcribe incoming audio windows
|
||||
|
||||
#### Scenario: Local Whisper transcript is finalized with pyannote speaker turns
|
||||
- **WHEN** the configured provider is `whisper-local`, live Whisper transcript segments exist, and pyannote returns speaker turns for the temporary recording
|
||||
- **THEN** Meeting Assistant rewrites the transcript with the original Whisper text and the best-overlap pyannote speaker label for each segment
|
||||
|
||||
#### Scenario: Local Whisper transcript is segmented by pyannote speaker turns
|
||||
- **WHEN** the configured provider is `whisper-local`, pyannote turn alignment is configured, live Whisper transcript segments exist, and pyannote returns speaker turns for the temporary recording
|
||||
- **THEN** Meeting Assistant rewrites the transcript as ordered pyannote speaker-turn segments with matching Whisper text assigned to each turn
|
||||
|
||||
#### Scenario: Local Whisper uses exclusive pyannote diarization
|
||||
- **WHEN** the configured provider is `whisper-local` and exclusive pyannote annotation is configured
|
||||
- **THEN** Meeting Assistant reads speaker turns from pyannote's `exclusive_speaker_diarization` output
|
||||
|
||||
#### Scenario: Local Whisper pyannote runtime cache is reused
|
||||
- **WHEN** local Whisper finalization runs pyannote more than once
|
||||
- **THEN** Meeting Assistant uses the configured reusable pyannote runtime image and persistent pyannote model cache instead of treating each run as a clean download environment
|
||||
|
||||
#### Scenario: Local Whisper pyannote diarization is unavailable
|
||||
- **WHEN** the configured provider is `whisper-local` but pyannote is disabled, has no configured token, or returns no speaker turns
|
||||
- **THEN** Meeting Assistant keeps the live Whisper transcript instead of deleting it
|
||||
|
||||
### Requirement: Azure Speech can provide streaming transcription
|
||||
Meeting Assistant SHALL provide an Azure Speech speech recognition pipeline that streams captured PCM audio to Azure AI Speech through the Azure Speech SDK conversation transcription API.
|
||||
|
||||
The Azure Speech adapter SHALL use configurable endpoint, region, language, key, and key environment variable settings.
|
||||
|
||||
The Azure Speech adapter SHALL support configurable diarization of intermediate conversation transcription results.
|
||||
|
||||
The Azure Speech adapter SHALL bound recognizer shutdown after the input audio stream is closed.
|
||||
|
||||
When Azure Speech is the configured speech recognition pipeline, Meeting Assistant SHALL emit live transcript segments from Azure conversation transcription events with Azure speaker IDs when available.
|
||||
|
||||
When Azure Speech is the configured speech recognition pipeline, Meeting Assistant SHALL use the live Azure conversation transcription segments as the finished transcript without running pyannote finalization.
|
||||
|
||||
#### Scenario: Azure Speech pipeline is configured
|
||||
- **WHEN** the configured provider is `azure-speech`
|
||||
- **THEN** Meeting Assistant streams captured PCM chunks to Azure AI Speech conversation transcription through the configured speech recognition pipeline without changing recording control or transcript persistence code
|
||||
|
||||
#### Scenario: Azure Speech returns speaker IDs
|
||||
- **WHEN** Azure Speech conversation transcription returns transcript text with speaker identity
|
||||
- **THEN** Meeting Assistant emits ordered transcript segments with those Azure speaker identities
|
||||
|
||||
#### Scenario: Azure Speech key is resolved from environment
|
||||
- **WHEN** Azure Speech is configured with `KeyEnv`
|
||||
- **THEN** Meeting Assistant reads the Azure Speech key from that environment variable
|
||||
|
||||
#### Scenario: Azure Speech stream shutdown is bounded
|
||||
- **WHEN** the captured audio stream has ended
|
||||
- **THEN** Meeting Assistant stops Azure Speech continuous recognition within the configured stop timeout instead of waiting indefinitely
|
||||
|
||||
#### Scenario: Azure Speech transcript is finalized from live conversation segments
|
||||
- **WHEN** the configured provider is `azure-speech` and live Azure transcript segments exist
|
||||
- **THEN** Meeting Assistant uses the live Azure conversation transcript segments as the finished transcript
|
||||
|
||||
### Requirement: FunASR can provide speaker-attributed streaming transcription
|
||||
Meeting Assistant SHALL provide a FunASR speech recognition pipeline that streams PCM audio to a configured FunASR WebSocket endpoint.
|
||||
|
||||
The FunASR provider SHALL preserve speaker attribution from FunASR response fields when they are present and SHALL fall back to `Unknown` when no speaker field is returned.
|
||||
|
||||
Meeting Assistant SHALL write the mixed recording audio only as a temporary processing artifact while streaming transcription is active. After capture stops and the streaming provider has drained already-captured audio, Meeting Assistant SHALL be able to run a final FunASR Python `AutoModel` pass with VAD, punctuation, and CAMPPlus speaker modeling over the temporary audio. When that final pass returns sentence-level speaker labels, Meeting Assistant SHALL rewrite the transcript markdown with the final speaker-attributed sentence segments.
|
||||
|
||||
Meeting Assistant SHALL delete temporary recording audio after the run completes, whether final diarization succeeds, fails, or produces no speaker output. On application startup, Meeting Assistant SHALL delete stale temporary recording audio left behind by previous interrupted runs.
|
||||
|
||||
When configured, Meeting Assistant SHALL manage a local FunASR backend lifecycle for the FunASR provider by starting a non-blocking backend warm-up when the application starts, installing the configured streaming runtime image, preparing the mounted model and hotword folder, starting the backend, keeping the container alive for the backgrounded FunASR server process, and stopping it when the application stops. Recording SHALL be able to start while the backend is still warming up; captured audio SHALL be queued and streamed once the backend is healthy. Backend installation, startup commands, and WebSocket readiness SHALL be bounded by configurable timeouts so startup failures are diagnosable.
|
||||
|
||||
#### Scenario: FunASR pipeline is configured
|
||||
- **WHEN** the configured provider is `funasr`
|
||||
- **THEN** Meeting Assistant streams captured PCM chunks to the configured FunASR WebSocket endpoint through the configured speech recognition pipeline without changing recording control or transcript persistence code
|
||||
|
||||
#### Scenario: FunASR returns speaker fields
|
||||
- **WHEN** FunASR returns transcript text with speaker identity fields
|
||||
- **THEN** Meeting Assistant emits ordered transcript segments with those speaker identities
|
||||
|
||||
#### Scenario: Final FunASR diarization rewrites transcript
|
||||
- **WHEN** recording stops after mixed audio was temporarily stored and the final FunASR diarization pass returns sentence-level speaker labels
|
||||
- **THEN** Meeting Assistant rewrites the transcript markdown with the final speaker-attributed sentence segments
|
||||
|
||||
#### Scenario: Final FunASR diarization has no speaker output
|
||||
- **WHEN** recording stops but the final diarization pass is disabled or returns no sentence-level speaker labels
|
||||
- **THEN** Meeting Assistant keeps the live streaming transcript instead of deleting it
|
||||
|
||||
#### Scenario: Temporary recording audio is deleted after completion
|
||||
- **WHEN** a recording run completes after using temporary audio for final processing
|
||||
- **THEN** Meeting Assistant deletes the temporary audio file from disk
|
||||
|
||||
#### Scenario: Stale temporary recordings are deleted on startup
|
||||
- **WHEN** Meeting Assistant starts and stale temporary recording files exist from a previous interrupted run
|
||||
- **THEN** Meeting Assistant deletes those stale temporary recording files before accepting new recordings
|
||||
|
||||
#### Scenario: Managed FunASR backend starts with the application
|
||||
- **WHEN** the configured provider is `funasr` and managed backend startup is enabled
|
||||
- **THEN** Meeting Assistant begins installing the configured FunASR backend image and starting the two-pass streaming backend without blocking recording startup
|
||||
|
||||
#### Scenario: Recording starts while managed FunASR backend is warming up
|
||||
- **WHEN** recording starts before the managed FunASR backend is ready
|
||||
- **THEN** Meeting Assistant starts audio capture immediately and queues captured audio until the backend WebSocket is healthy
|
||||
|
||||
#### Scenario: Managed FunASR backend installation stalls
|
||||
- **WHEN** a managed FunASR backend command exceeds its configured timeout
|
||||
- **THEN** Meeting Assistant reports a backend startup timeout instead of waiting indefinitely
|
||||
|
||||
#### Scenario: Managed FunASR backend WebSocket is not healthy
|
||||
- **WHEN** the managed backend container has started but the FunASR WebSocket endpoint is not accepting sessions
|
||||
- **THEN** Meeting Assistant continues waiting for backend readiness until the configured startup timeout is reached
|
||||
|
||||
#### Scenario: Managed FunASR backend stops with the application
|
||||
- **WHEN** Meeting Assistant stops after starting a managed FunASR backend
|
||||
- **THEN** Meeting Assistant stops the managed backend process or container
|
||||
|
||||
Reference in New Issue
Block a user