meeting-assistant/openspec/changes/add-resemblyzer-speaker-recognition/specs/meeting-transcription/spec.md

ADDED Requirements

Requirement: Speaker recognition can use local Resemblyzer voice vectors

Meeting Assistant SHALL expose SpeakerIdentification:Resemblyzer:Enabled as an application-level feature flag that defaults to disabled.

When the feature is disabled, Meeting Assistant SHALL use the existing WAV-snippet, Azure Speech, and optional pyannote speaker-identification backend without reading or writing Resemblyzer voice vectors.

When the feature is enabled, Meeting Assistant SHALL use a separate local Resemblyzer speaker-identification backend and SHALL NOT invoke the Azure Speech or pyannote speaker-identity matchers.

The Resemblyzer backend SHALL create temporary WAV samples from diarized same-speaker runs during recording, SHALL encode each retained sample locally as a versioned 256-value voice vector, and SHALL NOT persist those temporary WAV samples as identity evidence.

Resemblyzer sample spans retained for one speaker SHALL not overlap. For transcription providers that only produce diarized speakers during finalization, Meeting Assistant SHALL extract qualifying non-overlapping samples from the completed mixed recording.

Automatic matching SHALL wait until the configured required number of valid vectors is available for a diarized speaker. The default required count SHALL be five.

The required vector count SHALL be an automatic-decision threshold and SHALL NOT cap collection, encoding, or persistence. After the threshold is met, Meeting Assistant SHALL retain every distinct qualifying current-run vector up to the configured per-identity limit. When a speaker was assigned during live transcription, final processing SHALL attach qualifying vectors collected after that assignment to the same identity.

The matcher SHALL reject a query cluster whose mean pairwise cosine similarity is below the configured minimum cluster cohesion. For a coherent query, it SHALL represent each known identity by the normalized centroid of compatible stored vectors, SHALL score that identity using the median cosine similarity from query vectors to the centroid, and SHALL select an identity only when the best score meets the configured minimum identity similarity and exceeds the runner-up by the configured minimum similarity margin. The runner-up margin SHALL be waived when only one candidate can be scored.

The required vector count, minimum cluster cohesion, minimum identity similarity, minimum runner-up margin, encoder model identifier, local runtime settings, and command timeout SHALL be configurable.

When an identity has at least the configured outlier-pruning minimum number of valid vectors for the active model, defaulting to 20, Meeting Assistant SHALL run a separate cosine-density clustering pass. Neighbor similarity, minimum neighbors, and minimum dominant-cluster ratio SHALL be configurable.

Meeting Assistant SHALL remove vectors outside the uniquely largest dense cluster only when that cluster meets the configured minimum ratio of compatible evidence, defaulting to 60%. When no cluster qualifies or the largest cluster is tied, Meeting Assistant SHALL retain the evidence and log that pruning was skipped. Vectors for other model identifiers SHALL NOT be removed by this pass.

When enabled, the local encoder SHALL provision and reuse an application-managed Python virtual environment under the configured runtime folder. It SHALL install a pinned CPU-only PyTorch distribution and Windows-compatible VAD wheel without requiring Docker or a system-wide Python package installation.

The local encoder SHALL reject missing, malformed, non-finite, zero-magnitude, wrong-count, and wrong-dimension results without persisting them or falling back to the existing remote matcher.

Scenario: Disabled feature preserves existing backend

• GIVEN Resemblyzer speaker recognition is disabled
• WHEN Meeting Assistant tries to identify a diarized speaker
• THEN it uses the existing WAV-snippet speaker-identification backend
• AND does not create or compare Resemblyzer voice vectors

Scenario: Automatic matching waits for five vectors

• GIVEN Resemblyzer speaker recognition requires five vectors
• AND an unresolved diarized speaker has four valid samples
• WHEN live speaker identification runs
• THEN Meeting Assistant does not compare that speaker with known identities
• WHEN a fifth valid sample becomes available
• THEN Meeting Assistant can encode and compare the coherent five-vector cluster

Scenario: Five vectors do not cap retained evidence

• GIVEN Resemblyzer automatic matching requires five vectors
• AND a meeting yields eight distinct qualifying vectors for one speaker
• WHEN Meeting Assistant accepts or creates that speaker identity
• THEN it stores all eight vectors within the configured identity limit

Scenario: Final processing retains evidence collected after a live match

• GIVEN a diarized speaker was matched after five vectors during live transcription
• AND three more qualifying vectors were collected later in the meeting
• AND the finished transcript already uses the matched speaker's name while retained samples use the original diarized label
• WHEN final speaker processing runs with the existing mapping
• THEN the three later vectors are attached to the matched identity

Scenario: Mature identity outliers are pruned

• GIVEN an identity has at least 20 compatible vectors
• AND a uniquely largest cosine-density cluster contains at least 60% of them
• WHEN vector evidence is added or identities are merged
• THEN vectors outside the dominant cluster are removed
• AND the pruning decision and removed count are logged

Scenario: Ambiguous clusters are retained

• GIVEN an identity has at least 20 compatible vectors split between equally large or non-dominant dense clusters
• WHEN outlier pruning runs
• THEN Meeting Assistant removes no vectors
• AND logs that no uniquely dominant cluster qualified

Scenario: Incoherent query cluster is rejected

• GIVEN five query vectors have mean pairwise cosine similarity below the configured cohesion threshold
• WHEN Resemblyzer speaker identification runs
• THEN Meeting Assistant does not assign the speaker to a known identity
• AND logs the measured cohesion and rejection reason

Scenario: Similar and unambiguous cluster is accepted

• GIVEN a coherent five-vector query cluster
• AND its median similarity to Chris's vector centroid meets the configured identity threshold
• AND its score exceeds every other scored identity by the configured margin
• WHEN Resemblyzer speaker identification runs
• THEN Meeting Assistant identifies the diarized speaker as Chris
• AND adds the five query vectors to Chris's identity within the configured limit

Scenario: Ambiguous best cluster is rejected

• GIVEN a coherent five-vector query cluster meets the identity similarity threshold for Chris
• AND another identity's score is within the configured runner-up margin
• WHEN Resemblyzer speaker identification runs
• THEN Meeting Assistant leaves the diarized speaker unresolved
• AND logs both candidate scores and the insufficient margin

Scenario: Encoder failure preserves diarized labels

• GIVEN Resemblyzer speaker recognition is enabled
• WHEN the local encoder fails or returns invalid vectors
• THEN Meeting Assistant does not invoke the existing Azure or pyannote identity matcher as a fallback
• AND keeps the available diarized speaker labels

Scenario: Encoder provisions an isolated CPU environment

• GIVEN Resemblyzer speaker recognition is enabled
• AND its versioned virtual environment is not ready
• WHEN encoder warm-up runs
• THEN Meeting Assistant creates the virtual environment with the configured Python command
• AND installs the configured CPU-only PyTorch, Windows-compatible VAD, and Resemblyzer versions inside that environment
• AND does not invoke Docker

MODIFIED Requirements

Requirement: Speaker identity samples require uninterrupted speech

Meeting Assistant SHALL only retain speaker identity samples after a diarized speaker has produced a same-speaker sample span meeting the configured minimum duration.

The default minimum sample duration SHALL be 10 seconds.

Meeting Assistant SHALL combine consecutive transcript segments for the same diarized speaker into one sample span even when the transcription provider splits those segments around pauses. Provider-created pauses SHALL remain in the bounded extracted WAV but SHALL NOT count toward the configured minimum speaker-audio duration.

This aggregation behavior SHALL apply uniformly to diarized segments from every configured STT backend, whether segments arrive during live transcription or become available during finalization.

Meeting Assistant SHALL end the pending span when a different diarized speaker interrupts it or when the configured maximum sample duration is reached. The default maximum sample duration SHALL be 60 seconds, and no extracted recognition WAV SHALL exceed it.

When Resemblyzer recognition is enabled, Meeting Assistant SHALL start a new non-overlapping sample after accepting the previous sample from the same speaker.

Scenario: Short speaker span is not retained

• GIVEN the configured minimum sample duration is 10 seconds
• WHEN a diarized speaker produces only 8 seconds of uninterrupted speech
• THEN Meeting Assistant does not retain a speaker identity sample for that span

Scenario: Adjacent same-speaker segments form a sample

• GIVEN the configured minimum sample duration is 10 seconds
• WHEN a diarized speaker produces consecutive provider segments containing at least 10 seconds of speaker audio without another speaker interrupting
• THEN Meeting Assistant retains one speaker identity sample covering the continuous span

Scenario: Provider pause does not split a same-speaker sample

• GIVEN any configured STT backend emits consecutive lines for Guest01 with a pause longer than the former segment-gap threshold
• WHEN no differently labeled speaker appears between those lines
• THEN Meeting Assistant combines the lines into one speaker-recognition sample span
• AND counts only their diarized speaker-audio durations toward the minimum

Scenario: Speaker sample is capped at 60 seconds

• GIVEN the maximum sample duration is 60 seconds
• WHEN consecutive transcript lines for one speaker span more than 60 seconds
• THEN every extracted speaker-recognition WAV is at most 60 seconds long

Scenario: Different speaker interrupts pending span

• GIVEN the configured minimum sample duration is 10 seconds
• WHEN Guest01 speaks for 8 seconds and then Guest02 speaks
• THEN Meeting Assistant discards the pending Guest01 span instead of retaining or later extending it

Requirement: Meeting Assistant learns speaker identities locally

Meeting Assistant SHALL maintain a local SQLite speaker identity database in the user's application data folder.

The speaker identity database SHALL store speaker identities, optional canonical names, aliases, candidate names, meeting file references, a bounded set of WAV snippets per identity for the existing backend, and a separate bounded set of versioned voice vectors per identity for the Resemblyzer backend.

Each persisted voice vector SHALL store its model identifier, dimension, creation time, and a fingerprint that makes adding the same vector to the same identity idempotent.

Meeting file references SHALL include the meeting note file address and the transcript file address.

Meeting Assistant SHALL calculate speaker identity participation counts from meeting file references when needed instead of persisting a denormalized transcript count.

Each speaker identity SHALL store a last-modified timestamp used by active-age filtering, and Meeting Assistant SHALL update it whenever the identity is created or modified by identification, candidate updates, snippet changes, voice-vector changes, reference changes, or merge operations.

The configured maximum snippet count and maximum voice-vector count per identity SHALL prevent unbounded growth. The default maximum voice-vector count SHALL be 1,000.

Except for adding newly accepted Resemblyzer match evidence and its meeting reference, final candidate elimination, canonical promotion, and new unmatched identity creation SHALL happen only after transcription is finished and after automatic summary generation has completed, using the latest meeting note frontmatter.

When the summary agent records a speaker override from a diarized transcript label to a named speaker, final speaker identity processing SHALL attach the current run's evidence to an existing identity with that name when one exists, or create a new canonical speaker identity with that name when none exists. For the existing backend that evidence SHALL be the resolved WAV snippet; for the Resemblyzer backend it SHALL be all available valid current-run voice vectors up to the configured per-run count. Meeting Assistant SHALL NOT create a new speaker identity for an override when no current run evidence or current run candidate can be resolved for the source speaker label.

When a speaker override maps a current-run unnamed candidate to an existing named identity, Meeting Assistant SHALL merge the candidate's meeting reference and useful backend-specific evidence into the named identity instead of leaving a duplicate candidate.

When the summary agent records that a speaker identity was wrongfully matched, final speaker identity processing SHALL delete the matching identity and all of its WAV and voice-vector evidence from the local speaker identity database so it cannot be matched again unless it is newly created in the future.

Scenario: Unknown speaker is learned from meeting attendees

• WHEN a finished transcript contains an unmatched diarized speaker and the meeting note has attendees
• THEN Meeting Assistant stores a new unnamed speaker identity with candidate names from the attendees that were not already matched in that meeting
• AND stores a meeting file reference for that identity

Scenario: Speaker snippets are bounded

• WHEN Meeting Assistant adds a snippet for an identity that already has the configured maximum number of snippets
• THEN Meeting Assistant does not store more snippets for that identity

Scenario: Speaker voice vectors are bounded

• GIVEN the Resemblyzer vector limit is 1,000
• WHEN Meeting Assistant adds vectors to an identity that already has 1,000 stored vectors
• THEN Meeting Assistant does not store more than 1,000 vectors for that identity

Scenario: Retried vector evidence is idempotent

• GIVEN an identity already contains a voice vector
• WHEN Meeting Assistant retries adding the same vector to that identity
• THEN it stores only one copy of that vector

Scenario: Identity modification updates active-age timestamp

• WHEN Meeting Assistant creates, identifies, updates candidates for, stores snippets or voice vectors for, stores references for, or merges a speaker identity
• THEN Meeting Assistant updates that identity's last-modified timestamp

Scenario: Final speaker identity learning uses summary-refined attendees

• GIVEN the summary agent changes meeting note attendees during automatic summary generation
• WHEN Meeting Assistant performs final speaker identity learning and candidate creation
• THEN it uses the attendee list from the meeting note after the summary agent changes

Scenario: Speaker override attaches to existing identity

• GIVEN the speaker identity database contains canonical speaker Sabrina
• AND the summary agent records that transcript speaker Guest-01 is Sabrina
• WHEN final speaker identity processing runs
• THEN Meeting Assistant stores the meeting reference and current backend-specific speaker evidence on Sabrina's identity
• AND does not create a separate unnamed candidate for Guest-01

Scenario: Resemblyzer override stores available vectors

• GIVEN Resemblyzer speaker recognition is enabled
• AND the current run has three valid vectors for Guest-01
• WHEN the summary agent assigns Guest-01 to Sabrina
• THEN Meeting Assistant attaches those three vectors to Sabrina's identity
• AND does not require five vectors for the explicit assignment

Scenario: Speaker override creates named identity

• GIVEN the speaker identity database has no accepted name Sabrina
• AND the summary agent records that transcript speaker Guest-01 is Sabrina
• WHEN final speaker identity processing runs
• THEN Meeting Assistant creates a canonical speaker identity named Sabrina
• AND stores the meeting reference and current backend-specific speaker evidence on that identity

Scenario: Speaker override with missing source sample is skipped

• GIVEN the speaker identity database has no accepted name Sabrina
• AND the summary agent records that transcript speaker Guest-5 is Sabrina
• AND final speaker identity processing has no sample, vector, or segment for Guest-5
• WHEN final speaker identity processing runs
• THEN Meeting Assistant does not create a speaker identity for Sabrina

Scenario: Speaker identity deletion removes a wrong match

• GIVEN the speaker identity database contains canonical speaker Sabrina
• AND the summary agent records that Sabrina was wrongfully matched
• WHEN final speaker identity processing runs
• THEN Meeting Assistant removes Sabrina's identity and backend-specific evidence from the speaker identity database
• AND the relabeled transcript uses Removed-1 instead of Sabrina

Requirement: Speaker identities can be merged diagnostically

Meeting Assistant SHALL expose a diagnostic endpoint that merges duplicate speaker identities.

The merge process SHALL compare recently-created identities, using a configurable recent age that defaults to two weeks, against all other identities using the selected backend's candidate-scoring strategy.

For the existing WAV backend, the merge process SHALL require a match and a second validation match using a different source sample. For the Resemblyzer backend, it SHALL require two disjoint coherent source-vector clusters to select the same target identity.

When identities are merged, Meeting Assistant SHALL retain one identity, move useful names from the merged identity into aliases, combine meeting file references, retain bounded sets of snippets and voice vectors from both identities, and append an audit line to each referenced transcript in the form <date> <name 1> and <name 2> were merged.

When combined Resemblyzer evidence exceeds the configured vector limit, Meeting Assistant SHALL keep no more than that limit, preferring the most recently created distinct vectors.

After combining Resemblyzer evidence, Meeting Assistant SHALL apply the configured mature-identity outlier-pruning policy.

Scenario: Recently-created duplicate identity is merged

• GIVEN a recently-created identity and an older identity have matching backend-specific speaker evidence
• WHEN the diagnostic merge endpoint is triggered
• THEN Meeting Assistant validates the match twice with different source evidence
• AND merges the recent identity into the older identity
• AND stores the recent identity name as an alias on the retained identity
• AND keeps meeting file references and bounded backend-specific evidence from both identities
• AND appends the merge audit line to the referenced transcripts

Scenario: Resemblyzer merge needs two clusters

• GIVEN Resemblyzer speaker recognition requires five vectors per cluster
• AND a recent identity has ten vectors split into two coherent clusters
• WHEN both clusters independently match the same target identity
• THEN Meeting Assistant merges the recent identity into that target

Scenario: Old identities are not used as merge sources

• GIVEN two identities older than the configured recent age
• WHEN the diagnostic merge endpoint is triggered
• THEN Meeting Assistant does not compare them as source identities

Requirement: Speaker identity matches relabel transcripts

Meeting Assistant SHALL attempt to match unknown diarized speaker evidence against known speaker identities ordered by calculated meeting reference count.

When Resemblyzer speaker recognition is disabled, matching SHALL use the existing dedicated Azure Speech diarization verifier and optional pyannote validator with WAV snippets. When Resemblyzer speaker recognition is enabled, matching SHALL instead use only compatible locally calculated Resemblyzer voice-vector clusters.

For the existing WAV backend, the matcher SHALL test at most the configured batch size of known people per matching round and continue with later batches until a match is found or no candidates remain. The Resemblyzer backend SHALL score the capped candidate set together so ambiguity is measured against the global runner-up.

The matcher SHALL prioritize identities whose canonical name or aliases match current meeting attendees.

After attendee-matched identities, the matcher SHALL order identities by calculated meeting reference count, filter out non-attendee identities whose last update is older than the configured active age, and cap the candidate set at the configured maximum match candidate count.

When a match is confirmed, Meeting Assistant SHALL store a meeting file reference and the accepted backend-specific evidence for that identity within its configured limit.

When a match is confirmed and the identity has a canonical name, Meeting Assistant SHALL rewrite finished transcript segments for that diarized speaker with the canonical name.

When a match is confirmed and the matched speaker is not already listed in meeting note attendees by display name or alias, Meeting Assistant SHALL add the speaker display name to the attendee list.

When a match is confirmed and the meeting note attendees contain both the speaker display name and one or more accepted aliases for that same speaker, Meeting Assistant SHALL remove the alias attendee entries and keep the display name entry.

When Meeting Assistant writes attendees from calendar metadata, it SHALL match attendee display names exactly against known identity canonical names and aliases, replace matches with the identity display name, and deduplicate attendees that map to the same identity.

Scenario: Finished transcript is relabeled after a confirmed match

• GIVEN the speaker identity database contains canonical speaker Chris
• WHEN a finished transcript has diarized speaker Guest03 and the selected matching backend confirms it is Chris
• THEN Meeting Assistant rewrites Guest03 segments in the transcript as Chris

Scenario: Confirmed match stores meeting reference

• GIVEN the speaker identity database contains canonical speaker Chris
• WHEN a finished transcript has diarized speaker Guest03 and the selected matching backend confirms it is Chris
• THEN Meeting Assistant stores the meeting note and transcript file addresses as a reference for Chris
• AND stores the accepted backend-specific evidence within its configured limit

Scenario: Confirmed match removes duplicate aliases

• GIVEN the speaker identity database contains canonical speaker Christopher with alias Chris
• AND the meeting note attendees contain both Christopher and Chris <chris@example.com>
• WHEN live or final speaker matching confirms a diarized speaker is Christopher
• THEN Meeting Assistant keeps Christopher in the meeting note attendees
• AND removes Chris <chris@example.com> from the meeting note attendees

Requirement: Speaker matching runs during active transcription

Meeting Assistant SHALL start speaker identity matching only after the configured initial transcription duration has elapsed.

For backends that emit live diarized transcript segments, Meeting Assistant SHALL keep a bounded in-memory sliding audio buffer with chunk timestamps and extract candidate WAV samples from that buffer when live diarized segments arrive.

Meeting Assistant SHALL keep only the configured best candidate samples per diarized speaker in memory. Better samples SHALL be preferred when the segment looks like a continuous medium-length sentence. When Resemblyzer is enabled, accepted samples for one speaker SHALL be non-overlapping and the retained count SHALL be at least the configured required vector count.

Meeting Assistant SHALL periodically match unresolved diarized speaker evidence while transcription is active and attempt to match it against the local identity database.

Meeting Assistant SHALL run live matching incrementally at the configured interval only when at least one new unmapped diarized speaker sample appears or the meeting note attendee frontmatter changes while unmapped speaker samples still exist. For Resemblyzer, additional samples for an existing unresolved speaker SHALL also trigger another attempt so an earlier insufficient-vector result does not suppress matching when the required count becomes available.

When a speaker is matched during transcription, Meeting Assistant SHALL rewrite already-written live transcript segments for that diarized speaker and write future transcript segments using the canonical name.

For the existing WAV backend, live speaker matching SHALL be read-only with respect to the speaker identity database. For the Resemblyzer backend, a confirmed live match SHALL persist the accepted deduplicated voice vectors and meeting reference immediately so an assignment made during transcription is learned. Candidate elimination, canonical promotion, and new unmatched identity creation SHALL still happen only after transcription is finished and after automatic summary generation has completed, using the latest meeting note frontmatter.

For backends that only provide diarization after finalization, Meeting Assistant SHALL defer speaker identity matching until finished diarization is available, extract candidate samples from the completed temporary recording, complete identity matching, and only then allow summary generation to start.

Scenario: Matching waits for useful speech duration

• WHEN transcription has been active for less than the configured speaker identification initial delay
• THEN Meeting Assistant does not run speaker identity matching yet

Scenario: Live matching uses in-memory speaker samples

• WHEN a live diarized transcript segment identifies an unresolved speaker
• THEN Meeting Assistant extracts a temporary WAV sample for that segment from the in-memory sliding audio buffer
• AND uses retained speaker evidence for live identity matching without reading the temporary recording file

Scenario: Live match rewrites current and future transcript writes

• WHEN periodic matching confirms that diarized speaker Guest03 is canonical speaker Chris
• THEN already-written live transcript segments for Guest03 are rewritten as Chris
• AND later live transcript segments for Guest03 are written as Chris

Scenario: Resemblyzer live match persists vectors

• GIVEN Resemblyzer speaker recognition is enabled
• WHEN periodic matching confirms a coherent five-vector cluster for Guest03 as canonical speaker Chris
• THEN Meeting Assistant stores those vectors and the current meeting reference on Chris's identity
• AND does not persist the temporary WAV samples

Scenario: New live speaker triggers another identification round

• GIVEN live matching already checked the current unresolved speaker samples
• WHEN a new unmapped diarized speaker sample appears
• THEN Meeting Assistant runs another live matching round at the next configured interval

Scenario: Additional samples unlock live Resemblyzer matching

• GIVEN an earlier live attempt had fewer than five samples for an unresolved speaker
• AND no new speaker or attendee change occurs
• WHEN that speaker accumulates five qualifying samples
• THEN Meeting Assistant attempts matching again at the next configured interval
• AND does not repeatedly match unchanged evidence

Scenario: Attendee changes trigger another identification round

• GIVEN live matching already checked unresolved speaker samples
• WHEN the meeting note attendee frontmatter changes
• THEN Meeting Assistant runs another live matching round at the next configured interval using the latest attendees

Scenario: Final decisions use summary-refined attendees

• WHEN live matching finds or does not find a possible speaker identity during transcription
• THEN Meeting Assistant does not eliminate candidate names, promote canonical names, or create unmatched identities during that live pass
• AND the final speaker identity pass uses the latest meeting note attendees after transcription finishes