Public Access
Improve meeting audio mixing pipeline
PR and Push Build/Test / build-and-test (push) Successful in 11m22s
PR and Push Build/Test / build-and-test (push) Successful in 11m22s
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
# Add Local Echo Cancellation and MAS Assets
|
||||
|
||||
## Why
|
||||
The current mono meeting mix adds microphone and system samples directly, which can clip when both streams are loud. Speaker identity samples are extracted from this stream, so clipping and loopback bleed reduce sample quality.
|
||||
|
||||
Azure Speech SDK's Microsoft Audio Stack can perform local acoustic echo cancellation when the Azure backend is used, but it does not expose reusable cleaned PCM chunks back to Meeting Assistant. Meeting Assistant should keep the MAS package/assets available for experiments while using an owned local echo cleanup stage that produces normal 16 kHz mono PCM chunks for storage and every ASR provider.
|
||||
|
||||
## What Changes
|
||||
- Capture microphone and system loopback as separate PCM streams inside the composite audio source.
|
||||
- Clean the microphone stream with a local adaptive echo canceller that uses system loopback as the far-end reference.
|
||||
- Mix the cleaned microphone and system streams by addition, with configurable gains still available.
|
||||
- Write only the mixed temporary WAV used by transcription/finalization; separate channel files are not kept outside diagnostics.
|
||||
- Keep Microsoft MAS package/assets in the publish output without exposing an Azure AEC runtime feature flag.
|
||||
|
||||
## Impact
|
||||
- Current mixed WAV output should contain less loopback bleed on the microphone side while preserving remote/system audio.
|
||||
- Azure, FunASR, Whisper, pyannote, and speaker samples continue consuming 16 kHz mono PCM chunks.
|
||||
- Sidecar temporary WAV files are diagnostic artifacts and are deleted with the main temporary recording.
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Meeting Assistant captures meeting audio
|
||||
Meeting Assistant SHALL capture audio as 16 kHz mono PCM chunks for the existing recording and transcription pipeline.
|
||||
|
||||
Meeting Assistant SHALL capture microphone and system loopback as separate input streams before producing the final mono chunks.
|
||||
|
||||
Meeting Assistant SHALL clean the microphone stream with a local acoustic echo cancellation stage that uses system loopback as the far-end reference.
|
||||
|
||||
Meeting Assistant SHALL produce final mono chunks by adding the cleaned microphone samples and system samples.
|
||||
|
||||
Meeting Assistant SHALL align microphone and system samples through per-source buffers before mixing and SHALL NOT emit normal live audio chunks that contain only one source while the other source is merely delayed.
|
||||
|
||||
When one source stays quiet beyond the alignment timeout, Meeting Assistant SHALL mix the available source with synthetic silence for the missing source instead of blocking transcription.
|
||||
|
||||
Meeting Assistant SHALL allow the final microphone/system mono mix to apply configurable microphone and system gain before combining samples.
|
||||
|
||||
Meeting Assistant SHALL use the active run or launch profile recording options when configuring capture format and final microphone/system gains.
|
||||
|
||||
Meeting Assistant SHALL clamp mixed samples after gain is applied.
|
||||
|
||||
Meeting Assistant SHALL write only the mixed stream to the temporary WAV used by transcription and finalization.
|
||||
|
||||
#### Scenario: Mixed audio uses cleaned microphone and system audio
|
||||
- **GIVEN** the echo canceller cleans a microphone chunk to sample `2000`
|
||||
- **AND** the matching system chunk has sample `10000`
|
||||
- **WHEN** microphone and system chunks are mixed with gains `1` and `1`
|
||||
- **THEN** the mixed sample is `12000`
|
||||
|
||||
#### Scenario: Temporary recording stores only the mixed stream
|
||||
- **GIVEN** Meeting Assistant has mixed microphone and system audio into one PCM chunk
|
||||
- **WHEN** Meeting Assistant appends the chunk to the temporary recording
|
||||
- **THEN** the main temporary WAV contains the mixed PCM
|
||||
- **AND** no microphone or system sidecar WAV is written
|
||||
|
||||
#### Scenario: Launch profile recording options configure capture and gains
|
||||
- **GIVEN** an active launch profile configures sample format and microphone/system mix gains
|
||||
- **WHEN** Meeting Assistant captures and mixes audio for that run
|
||||
- **THEN** the microphone and system capture sources receive that launch profile recording configuration
|
||||
- **AND** the mixed output uses that launch profile's microphone/system gains
|
||||
|
||||
#### Scenario: Delayed sources are buffered before mixing
|
||||
- **GIVEN** microphone audio arrives before matching system audio
|
||||
- **WHEN** matching system audio arrives after a short delay
|
||||
- **THEN** Meeting Assistant emits one mixed chunk for the aligned samples
|
||||
- **AND** it does not emit separate microphone-only and system-only chunks for that delayed pair
|
||||
|
||||
#### Scenario: Quiet system audio does not block microphone transcription
|
||||
- **GIVEN** microphone audio arrives
|
||||
- **AND** system loopback audio does not arrive within the alignment timeout
|
||||
- **WHEN** Meeting Assistant mixes the available audio
|
||||
- **THEN** it emits the microphone audio mixed with silent system audio
|
||||
- **AND** live transcription can continue while system loopback is quiet
|
||||
|
||||
#### Scenario: Continuous microphone audio does not suppress the alignment timeout
|
||||
- **GIVEN** microphone audio keeps arriving
|
||||
- **AND** system loopback audio stays unavailable past the alignment timeout
|
||||
- **WHEN** Meeting Assistant checks the buffered microphone audio
|
||||
- **THEN** it emits the buffered microphone audio mixed with silent system audio
|
||||
- **AND** it does not wait indefinitely for a loopback chunk
|
||||
@@ -0,0 +1,12 @@
|
||||
## 1. Implementation
|
||||
- [x] Keep Microsoft MAS package/assets available without an Azure AEC runtime feature flag.
|
||||
- [x] Capture and buffer microphone/system input streams separately before mixing.
|
||||
- [x] Clean microphone chunks with a local echo cancellation stage before mixing.
|
||||
- [x] Mix cleaned microphone and system audio by addition with configurable gains.
|
||||
- [x] Keep temporary recording output to the mixed WAV; do not persist diagnostic channel sidecars.
|
||||
- [x] Add focused tests for local echo cleanup, additive mixing, aligned mixing, and mixed-only temporary recording.
|
||||
|
||||
## 2. Verification
|
||||
- [x] Run focused tests.
|
||||
- [x] Run `dotnet test MeetingAssistant.slnx`.
|
||||
- [x] Run `openspec validate add-audio-headroom-and-azure-aec --strict`.
|
||||
@@ -35,10 +35,68 @@ When no recording is active, aborting SHALL leave the current recording status u
|
||||
### Requirement: Recording mode captures microphone and computer output
|
||||
Meeting Assistant SHALL capture microphone input and computer output and combine them into one audio stream for transcription.
|
||||
|
||||
Meeting Assistant SHALL capture audio as 16 kHz mono PCM chunks for the existing recording and transcription pipeline.
|
||||
|
||||
Meeting Assistant SHALL capture microphone and system loopback as separate input streams before producing the final mono chunks.
|
||||
|
||||
Meeting Assistant SHALL clean the microphone stream with a local acoustic echo cancellation stage that uses system loopback as the far-end reference.
|
||||
|
||||
Meeting Assistant SHALL produce final mono chunks by adding the cleaned microphone samples and system samples.
|
||||
|
||||
Meeting Assistant SHALL align microphone and system samples through per-source buffers before mixing and SHALL NOT emit normal live audio chunks that contain only one source while the other source is merely delayed.
|
||||
|
||||
When one source stays quiet beyond the alignment timeout, Meeting Assistant SHALL mix the available source with synthetic silence for the missing source instead of blocking transcription.
|
||||
|
||||
Meeting Assistant SHALL allow the final microphone/system mono mix to apply configurable microphone and system gain before combining samples.
|
||||
|
||||
Meeting Assistant SHALL use the active run or launch profile recording options when configuring capture format and final microphone/system gains.
|
||||
|
||||
Meeting Assistant SHALL clamp mixed samples after gain is applied.
|
||||
|
||||
Meeting Assistant SHALL write only the mixed stream to the temporary WAV used by transcription and finalization.
|
||||
|
||||
#### Scenario: Both sources produce audio
|
||||
- **WHEN** microphone and computer output audio chunks are available
|
||||
- **THEN** Meeting Assistant mixes them into one PCM stream before transcription
|
||||
|
||||
#### Scenario: Mixed audio uses cleaned microphone and system audio
|
||||
- **GIVEN** the echo canceller cleans a microphone chunk to sample `2000`
|
||||
- **AND** the matching system chunk has sample `10000`
|
||||
- **WHEN** microphone and system chunks are mixed with gains `1` and `1`
|
||||
- **THEN** the mixed sample is `12000`
|
||||
|
||||
#### Scenario: Temporary recording stores only the mixed stream
|
||||
- **GIVEN** Meeting Assistant has mixed microphone and system audio into one PCM chunk
|
||||
- **WHEN** Meeting Assistant appends the chunk to the temporary recording
|
||||
- **THEN** the main temporary WAV contains the mixed PCM
|
||||
- **AND** no microphone or system sidecar WAV is written
|
||||
|
||||
#### Scenario: Launch profile recording options configure capture and gains
|
||||
- **GIVEN** an active launch profile configures sample format and microphone/system mix gains
|
||||
- **WHEN** Meeting Assistant captures and mixes audio for that run
|
||||
- **THEN** the microphone and system capture sources receive that launch profile recording configuration
|
||||
- **AND** the mixed output uses that launch profile's microphone/system gains
|
||||
|
||||
#### Scenario: Delayed sources are buffered before mixing
|
||||
- **GIVEN** microphone audio arrives before matching system audio
|
||||
- **WHEN** matching system audio arrives after a short delay
|
||||
- **THEN** Meeting Assistant emits one mixed chunk for the aligned samples
|
||||
- **AND** it does not emit separate microphone-only and system-only chunks for that delayed pair
|
||||
|
||||
#### Scenario: Quiet system audio does not block microphone transcription
|
||||
- **GIVEN** microphone audio arrives
|
||||
- **AND** system loopback audio does not arrive within the alignment timeout
|
||||
- **WHEN** Meeting Assistant mixes the available audio
|
||||
- **THEN** it emits the microphone audio mixed with silent system audio
|
||||
- **AND** live transcription can continue while system loopback is quiet
|
||||
|
||||
#### Scenario: Continuous microphone audio does not suppress the alignment timeout
|
||||
- **GIVEN** microphone audio keeps arriving
|
||||
- **AND** system loopback audio stays unavailable past the alignment timeout
|
||||
- **WHEN** Meeting Assistant checks the buffered microphone audio
|
||||
- **THEN** it emits the buffered microphone audio mixed with silent system audio
|
||||
- **AND** it does not wait indefinitely for a loopback chunk
|
||||
|
||||
#### Scenario: Device-level capture cannot be verified in tests
|
||||
- **WHEN** automated tests run without live audio devices
|
||||
- **THEN** Meeting Assistant verifies the audio mixer through deterministic source abstractions rather than depending on physical microphone or speaker devices
|
||||
|
||||
Reference in New Issue
Block a user