Harden speaker identity samples

2026-05-28 12:02:44 +02:00
parent c84f8a984a
commit a72cda0c03
23 changed files with 1251 additions and 123 deletions
@@ -0,0 +1,20 @@
+# Harden speaker samples and pyannote validation
+
+## Why
+
+Azure Speech speaker identity matching is too brittle when Meeting Assistant learns from short samples. Speaker identity samples should represent enough uninterrupted speech for the diarization backend to distinguish voices reliably.
+
+Meeting Assistant can also use pyannote as a secondary, optional confidence layer: first to reject samples that contain multiple speakers, and then to reject Azure identity matches when pyannote cannot confirm that the unknown and known snippets are the same speaker.
+
+## What Changes
+
+- Require speaker identity samples to meet a configurable minimum uninterrupted speech duration, defaulting to 30 seconds.
+- Accumulate adjacent same-speaker transcript segments into a single sample span when no other speaker interrupts the span.
+- Add optional pyannote-backed sample and match validation for speaker identity matching.
+- Reuse the existing pyannote Docker/runtime configuration shape where practical.
+
+## Impact
+
+- Speaker identity matching starts later for speakers who do not talk long enough uninterrupted.
+- Live matching tests that depended on very short samples need to configure a lower minimum sample duration.
+- Pyannote validation is opt-in and should not affect existing installations unless enabled.
@@ -0,0 +1,52 @@
+## ADDED Requirements
+
+### Requirement: Speaker identity samples require uninterrupted speech
+Meeting Assistant SHALL only retain speaker identity samples after a diarized speaker has produced at least the configured minimum duration of uninterrupted speech.
+
+The default minimum uninterrupted speech duration SHALL be 30 seconds.
+
+Meeting Assistant MAY combine adjacent transcript segments for the same diarized speaker into one sample span when no different diarized speaker interrupts the span.
+
+When a different diarized speaker appears before the configured minimum duration is reached, Meeting Assistant SHALL discard the pending sample span for the previous speaker.
+
+#### Scenario: Short speaker span is not retained
+- **GIVEN** the configured minimum sample duration is 30 seconds
+- **WHEN** a diarized speaker produces only 20 seconds of uninterrupted speech
+- **THEN** Meeting Assistant does not retain a speaker identity sample for that span
+
+#### Scenario: Adjacent same-speaker segments form a sample
+- **GIVEN** the configured minimum sample duration is 30 seconds
+- **WHEN** a diarized speaker produces adjacent segments totaling at least 30 seconds without another speaker interrupting
+- **THEN** Meeting Assistant retains one speaker identity sample covering the continuous span
+
+#### Scenario: Different speaker interrupts pending span
+- **GIVEN** the configured minimum sample duration is 30 seconds
+- **WHEN** `Guest01` speaks for 20 seconds and then `Guest02` speaks
+- **THEN** Meeting Assistant discards the pending `Guest01` span instead of retaining or later extending it
+
+### Requirement: Speaker identity matching can use pyannote secondary validation
+Meeting Assistant SHALL support an optional configurable pyannote secondary validation layer for speaker identity matching.
+
+When pyannote secondary validation is enabled, Meeting Assistant SHALL verify candidate speaker samples before retaining them for identity matching. Samples that pyannote reports as containing multiple speakers SHALL be rejected.
+
+When pyannote secondary validation is enabled and the primary identity matcher confirms a speaker, Meeting Assistant SHALL run a second validation pass through pyannote before accepting the match.
+
+If pyannote secondary validation cannot confirm that the unknown live sample and matched identity samples belong to one speaker, Meeting Assistant SHALL reject the match.
+
+When pyannote secondary validation is disabled, Meeting Assistant SHALL preserve the primary identity matching behavior.
+
+#### Scenario: Multi-speaker sample is rejected
+- **GIVEN** pyannote secondary validation is enabled
+- **WHEN** pyannote reports multiple speakers in a candidate sample
+- **THEN** Meeting Assistant does not retain that sample for identity matching
+
+#### Scenario: Pyannote rejects primary match
+- **GIVEN** pyannote secondary validation is enabled
+- **AND** the primary identity matcher confirms `Guest03` as `Chris`
+- **WHEN** pyannote reports that the unknown `Guest03` sample and known `Chris` samples contain different speakers
+- **THEN** Meeting Assistant rejects the match
+
+#### Scenario: Disabled pyannote validation preserves primary match
+- **GIVEN** pyannote secondary validation is disabled
+- **WHEN** the primary identity matcher confirms `Guest03` as `Chris`
+- **THEN** Meeting Assistant accepts the match without running pyannote secondary validation
@@ -0,0 +1,24 @@
+## 1. Specification
+
+- [x] Add requirements for minimum uninterrupted speaker samples.
+- [x] Add requirements for optional pyannote secondary validation.
+
+## 2. Tests
+
+- [x] Cover sample collection waiting for the configured minimum uninterrupted speech duration.
+- [x] Cover sample collection resetting when another speaker interrupts.
+- [x] Cover pyannote sample validation rejecting multi-speaker samples.
+- [x] Cover pyannote match validation rejecting an Azure-confirmed match.
+- [x] Update existing live matching tests to configure short samples where they intentionally exercise fast matching.
+
+## 3. Implementation
+
+- [x] Add speaker identification configuration for minimum sample duration and pyannote validation.
+- [x] Refactor live sample collection to accumulate continuous same-speaker spans.
+- [x] Add pyannote diarization reuse for speaker identity validation.
+- [x] Add secondary validation into the identity matching pipeline.
+
+## 4. Verification
+
+- [x] Run focused speaker/transcription tests.
+- [x] Run `openspec validate harden-speaker-samples-and-pyannote-validation --strict`.