Archive summary refinements and profile switching

2026-05-27 23:11:21 +02:00
parent c12febe029
commit 7777b349a5
37 changed files with 3190 additions and 121 deletions
@@ -0,0 +1,14 @@
+# Change: Refine attendees from screenshot OCR during summary
+
+## Why
+Calendar metadata often contains invited people rather than actual participants. Screenshots can reveal visible meeting participants, but OCR needs to state whether the visible people are complete or only partial evidence. The summary agent should be able to use that evidence to adjust the meeting attendee list before final speaker identity candidate updates run.
+
+## What Changes
+- The default screenshot OCR prompt asks the vision model to say whether visible people are the exact meeting participant set or only a partial result.
+- The summary agent has scoped tools to add and remove attendees from the current meeting note.
+- Summary instructions tell the agent to use transcript and OCR evidence to sharpen meeting attendees conservatively.
+- Final speaker identity learning/candidate creation runs after summary generation so it uses the summary-refined meeting attendees.
+
+## Impact
+- Affects screenshot OCR prompt defaults, summary-agent tool surface, summary instructions, recording finalization order, and speaker identity candidate timing.
+- Adds behavior tests for OCR instructions, summary attendee tools, and final speaker processing after summary.
@@ -0,0 +1,82 @@
+## MODIFIED Requirements
+### Requirement: Meeting Assistant generates meeting outputs
+Meeting Assistant SHALL generate a summary, decisions, and next steps from the meeting transcript, metadata, user notes, and assistant-discovered context.
+
+The summary pipeline SHALL expose tools scoped to the current meeting:
+
+- `read_meetingnote`
+- `read_transcript`
+- `read_context`
+- `read_usernotes`
+- `read_glossary`
+- `add_dictation_word`
+- `add_attendee`
+- `remove_attendee`
+- `override_speaker`
+- `delete_identity`
+- `write_summary`
+- `write_context`
+- `list_projects`
+- `list_projectfiles`
+- `read_projectfile`
+- `write_projectfile`
+- `search`
+
+The summary instructions SHALL tell the summary agent to use transcript evidence and screenshot OCR participant evidence to refine the meeting note attendee list conservatively, adding or removing attendees only when the evidence is clear.
+
+The summary instructions SHALL tell the summary agent to call `override_speaker` only when it is very certain that a transcript speaker label belongs to a named speaker, such as from user notes, OCR with a correlating meeting timestamp, or very clear context cues.
+
+When `override_speaker` is called with an existing transcript speaker label and a replacement speaker name, Meeting Assistant SHALL rewrite that speaker label in the transcript and record the override for final speaker identity processing.
+
+When `override_speaker` is called without merge enabled and the transcript already contains the replacement speaker name as a speaker label, Meeting Assistant SHALL refuse the override. When merge is enabled, Meeting Assistant SHALL allow the override, rewrite the source speaker label to the replacement speaker name, and record the override as a speaker merge for final speaker identity processing.
+
+The summary instructions SHALL tell the summary agent to call `delete_identity` only when it is certain that an existing speaker identity was wrongfully matched.
+
+When `delete_identity` is called with an existing transcript speaker identity, Meeting Assistant SHALL rewrite transcript speaker labels for that identity to `Removed-<n>` and record the deletion for final speaker identity processing.
+
+#### Scenario: Summary agent refines attendees from clear participant evidence
+- **GIVEN** assistant context OCR states that visible meeting participants are complete
+- **AND** the OCR result names a participant missing from the meeting note
+- **WHEN** the summary pipeline runs
+- **THEN** the summary agent can add that attendee to the meeting note through `add_attendee`
+
+#### Scenario: Summary agent does not over-trim partial participant evidence
+- **GIVEN** assistant context OCR states that visible meeting participants are only a partial result
+- **WHEN** the summary pipeline runs
+- **THEN** the summary instructions tell the agent not to remove attendees solely because they are absent from that partial screenshot evidence
+
+#### Scenario: Summary agent overrides a certain speaker label
+- **GIVEN** the transcript contains speaker label `Guest-01`
+- **AND** user notes or OCR evidence clearly identifies that speaker as `Sabrina`
+- **WHEN** the summary agent calls `override_speaker` with `Guest-01` and `Sabrina`
+- **THEN** Meeting Assistant rewrites the transcript speaker label to `Sabrina`
+- **AND** records the override for final speaker identity processing
+
+#### Scenario: Summary agent cannot accidentally merge speakers
+- **GIVEN** the transcript contains speaker labels `Guest-01` and `Sabrina`
+- **WHEN** the summary agent calls `override_speaker` with `Guest-01` and `Sabrina` without merge enabled
+- **THEN** Meeting Assistant refuses the override
+- **AND** keeps the transcript unchanged
+
+#### Scenario: Summary agent explicitly merges speakers
+- **GIVEN** the transcript contains speaker labels `Guest-01` and `Sabrina`
+- **AND** user notes or OCR evidence clearly proves they are the same person
+- **WHEN** the summary agent calls `override_speaker` with `Guest-01`, `Sabrina`, and merge enabled
+- **THEN** Meeting Assistant rewrites `Guest-01` transcript speaker labels to `Sabrina`
+- **AND** records the override as a merge for final speaker identity processing
+
+#### Scenario: Summary agent deletes a wrong speaker identity
+- **GIVEN** the transcript contains speaker identity `Sabrina`
+- **AND** user notes or OCR evidence clearly proves that `Sabrina` was wrongfully matched
+- **WHEN** the summary agent calls `delete_identity` with `Sabrina`
+- **THEN** Meeting Assistant rewrites the transcript speaker label to `Removed-1`
+- **AND** records the deletion for final speaker identity processing
+
+### Requirement: Meeting screenshots are captured into assistant context
+Meeting Assistant SHALL expose a configurable screenshot hotkey.
+
+The default OCR prompt SHALL explain that the image is from a meeting and ask the model to identify who is talking, who is presenting, what is presented, capture slide text in markdown, convert diagrams to Mermaid when possible, indicate whether visible people are clearly the exact meeting participants or only a partial result, return crop coordinates only for confidently isolated presentation/shared-screen content, and otherwise describe the scene.
+
+#### Scenario: OCR reports whether visible people are complete or partial
+- **WHEN** Meeting Assistant uses the built-in screenshot OCR prompt
+- **THEN** the prompt asks the model to state whether the screenshot clearly shows exactly who is in the meeting or only a partial participant result
@@ -0,0 +1,37 @@
+## MODIFIED Requirements
+### Requirement: Meeting Assistant learns speaker identities locally
+Meeting Assistant SHALL maintain a local SQLite speaker identity database in the user's application data folder.
+
+Live speaker matching SHALL be read-only with respect to the speaker identity database. Candidate elimination, canonical promotion, transcription counters, stored snippet updates, and new unmatched identity creation SHALL happen only after transcription is finished and after automatic summary generation has completed, using the latest meeting note frontmatter.
+
+When the summary agent records a speaker override from a diarized transcript label to a named speaker, final speaker identity processing SHALL attach the current run speaker sample to an existing identity with that name when one exists, or create a new canonical speaker identity with that name when none exists.
+
+When a speaker override maps a current-run unnamed candidate to an existing named identity, Meeting Assistant SHALL merge the candidate's meeting reference and useful snippets into the named identity instead of leaving a duplicate candidate.
+
+When the summary agent records that a speaker identity was wrongfully matched, final speaker identity processing SHALL delete the matching identity from the local speaker identity database so it cannot be matched again unless it is newly created in the future.
+
+#### Scenario: Final speaker identity learning uses summary-refined attendees
+- **GIVEN** the summary agent changes meeting note attendees during automatic summary generation
+- **WHEN** Meeting Assistant performs final speaker identity learning and candidate creation
+- **THEN** it uses the attendee list from the meeting note after the summary agent changes
+
+#### Scenario: Speaker override attaches to existing identity
+- **GIVEN** the speaker identity database contains canonical speaker `Sabrina`
+- **AND** the summary agent records that transcript speaker `Guest-01` is `Sabrina`
+- **WHEN** final speaker identity processing runs
+- **THEN** Meeting Assistant stores the meeting reference and current speaker sample on Sabrina's identity
+- **AND** does not create a separate unnamed candidate for `Guest-01`
+
+#### Scenario: Speaker override creates named identity
+- **GIVEN** the speaker identity database has no accepted name `Sabrina`
+- **AND** the summary agent records that transcript speaker `Guest-01` is `Sabrina`
+- **WHEN** final speaker identity processing runs
+- **THEN** Meeting Assistant creates a canonical speaker identity named `Sabrina`
+- **AND** stores the meeting reference and current speaker sample on that identity
+
+#### Scenario: Speaker identity deletion removes a wrong match
+- **GIVEN** the speaker identity database contains canonical speaker `Sabrina`
+- **AND** the summary agent records that `Sabrina` was wrongfully matched
+- **WHEN** final speaker identity processing runs
+- **THEN** Meeting Assistant removes Sabrina's identity from the speaker identity database
+- **AND** the relabeled transcript uses `Removed-1` instead of `Sabrina`
@@ -0,0 +1,27 @@
+## 1. Implementation
+- [x] 1.1 Add OpenSpec requirements for OCR participant certainty, summary attendee tools, and speaker identity timing.
+- [x] 1.2 Add failing behavior tests for the default OCR prompt and summary attendee tool registration/behavior.
+- [x] 1.3 Add failing behavior test proving final speaker identity processing sees attendees changed by summary.
+- [x] 1.4 Implement OCR prompt and summary instructions/tool changes.
+- [x] 1.5 Move final speaker identity learning/candidate update after summary completion without regressing transcript relabeling.
+- [x] 1.6 Update README and run focused/full tests plus `openspec validate refine-attendees-from-ocr-summary --strict`.
+
+## 2. Speaker overrides
+- [x] 2.1 Add behavior tests for the summary `override_speaker` tool rewriting transcript labels and recording an override.
+- [x] 2.2 Add behavior tests for final speaker identity processing attaching an override to an existing named identity or creating one.
+- [x] 2.3 Implement the summary tool, instructions, and agent registration.
+- [x] 2.4 Apply recorded overrides during post-summary final speaker identity processing.
+- [x] 2.5 Update README and rerun focused/full tests plus `openspec validate refine-attendees-from-ocr-summary --strict`.
+
+## 3. Speaker identity deletion
+- [x] 3.1 Add behavior tests for the summary `delete_identity` tool rewriting transcript labels and recording a deletion.
+- [x] 3.2 Add behavior tests for final speaker identity processing deleting the matching database identity and preserving the removed transcript label.
+- [x] 3.3 Implement the summary tool, instructions, and agent registration.
+- [x] 3.4 Apply recorded identity deletions during post-summary final speaker identity processing.
+- [x] 3.5 Update README and rerun focused/full tests plus `openspec validate refine-attendees-from-ocr-summary --strict`.
+
+## 4. Speaker override merge guard
+- [x] 4.1 Add behavior tests for `override_speaker` refusing duplicate target speaker labels unless merge is enabled.
+- [x] 4.2 Add behavior tests for `override_speaker` merge mode rewriting the transcript and recording the merge intent.
+- [x] 4.3 Implement the merge parameter, duplicate target guard, instructions, and agent registration description.
+- [x] 4.4 Update README and rerun focused/full tests plus `openspec validate refine-attendees-from-ocr-summary --strict`.
@@ -0,0 +1,14 @@
+# Change: Switch transcription profile during an active meeting
+
+## Why
+Meetings can change spoken language or transcription requirements without being separate meetings. The user needs to switch from one launch profile to another during an active recording, for example from German to English, without creating new meeting artifacts or prematurely starting summarization.
+
+## What Changes
+- Profile hotkeys pressed during active recording switch the active meeting to that profile instead of stopping and summarizing the meeting.
+- Switching keeps the current meeting note, transcript, assistant context, summary path, and metadata.
+- Switching stops and drains the current speech recognition pipeline, buffers captured audio while the old pipeline drains, inserts a transcript marker, clears run-local speaker label mappings, and starts a new speech recognition pipeline using the requested profile options.
+- The final stop after one or more switches still summarizes once for the same meeting.
+
+## Impact
+- Affects recording coordinator lifecycle, launch-profile hotkey semantics, live transcription buffering, speaker mapping lifecycle, and transcript output.
+- Adds behavior tests for profile switching without metadata recollection or new artifacts.
@@ -0,0 +1,34 @@
+## ADDED Requirements
+### Requirement: Active recording can switch launch profiles
+Meeting Assistant SHALL allow a launch profile hotkey or named-profile recording toggle request to switch the active meeting to that profile while recording capture is active.
+
+When switching profiles during an active meeting, Meeting Assistant SHALL keep the existing meeting note, transcript, assistant context, summary path, user notes, calendar metadata, projects, attendees, and assistant-context state.
+
+When switching profiles during an active meeting, Meeting Assistant SHALL NOT create new meeting artifacts, SHALL NOT collect calendar metadata again, and SHALL NOT run the summary pipeline for the old profile segment.
+
+When switching profiles during an active meeting, Meeting Assistant SHALL stop and drain the current speech recognition pipeline, append a transcript marker indicating the target profile, start a new speech recognition pipeline using the target profile's resolved options, and continue writing live transcription to the same transcript file.
+
+When audio is captured while the old speech recognition pipeline is draining, Meeting Assistant SHALL buffer that audio and send it to the new profile's speech recognition pipeline after it is ready.
+
+When switching profiles during an active meeting, Meeting Assistant SHALL clear run-local speaker label mappings for future transcript segments because backend speaker IDs can change across speech recognition processes. Already-written named transcript segments SHALL remain unchanged.
+
+#### Scenario: Active meeting switches to named profile
+- **GIVEN** a meeting was started with the default launch profile
+- **WHEN** the user triggers launch profile `english` while recording is active
+- **THEN** Meeting Assistant keeps the existing meeting artifacts
+- **AND** drains the old speech recognition pipeline without running summary generation
+- **AND** appends a transcript marker for the profile switch
+- **AND** starts transcription with the resolved `english` profile
+- **AND** continues writing transcript segments to the same transcript file
+
+#### Scenario: Captured audio is buffered while switching
+- **GIVEN** a meeting is switching from one launch profile to another
+- **WHEN** audio chunks are captured before the new speech recognition pipeline is ready
+- **THEN** Meeting Assistant buffers those chunks
+- **AND** sends the buffered chunks to the new speech recognition pipeline after it starts
+
+#### Scenario: Switching clears future speaker mappings only
+- **GIVEN** a live speaker label was mapped to a named speaker before switching profiles
+- **WHEN** Meeting Assistant switches to another launch profile
+- **THEN** already-written transcript text keeps the named speaker
+- **AND** later transcript segments from the new pipeline do not reuse the old backend speaker label mapping
@@ -0,0 +1,11 @@
+## ADDED Requirements
+### Requirement: Speech recognition can be restarted within one meeting
+Meeting Assistant SHALL support replacing the active speech recognition pipeline for a recording run while preserving the run's meeting artifacts and transcript session.
+
+The replacement speech recognition pipeline SHALL use the target launch profile's resolved transcription options.
+
+#### Scenario: Replacement pipeline uses target profile options
+- **GIVEN** a recording run started with one launch profile
+- **WHEN** the active transcription profile is switched to another launch profile
+- **THEN** the new speech recognition pipeline is created from the target profile options
+- **AND** it receives audio captured after the switch request and audio buffered during the switch
@@ -0,0 +1,8 @@
+## 1. Implementation
+- [x] 1.1 Add behavior tests for switching from default to named launch profile during an active recording.
+- [x] 1.2 Keep existing meeting artifacts and skip metadata recollection when switching profile.
+- [x] 1.3 Drain the old speech recognition pipeline without moving to summarizing.
+- [x] 1.4 Buffer captured audio during the switch and feed it to the new profile pipeline.
+- [x] 1.5 Append a transcript marker when the profile changes.
+- [x] 1.6 Clear run-local speaker label mappings for the new transcription process.
+- [x] 1.7 Update README and run focused/full tests plus `openspec validate switch-transcription-profile --strict`.