Public Access
279 lines
19 KiB
Markdown
279 lines
19 KiB
Markdown
# meeting-summary Specification
|
|
|
|
## Purpose
|
|
TBD - created by archiving change define-meeting-assistant-v1. Update Purpose after archive.
|
|
## Requirements
|
|
### Requirement: Meeting Assistant generates meeting outputs
|
|
Meeting Assistant SHALL generate a summary, decisions, and next steps from the meeting transcript, metadata, user notes, and assistant-discovered context.
|
|
|
|
Meeting Assistant SHALL use a Microsoft Agent Framework pipeline for the first summary implementation. The pipeline SHALL use a configurable OpenAI-compatible endpoint, model, and API key source, including support for a direct key or an environment variable name.
|
|
|
|
The summary pipeline SHALL retry transient model endpoint failures according to configurable reconnection attempts and delay settings.
|
|
|
|
The summary pipeline SHALL track context-window usage for the configured model using response usage when available and request-size estimates otherwise. The context-window limit, maximum output reserve, compaction enablement, compaction threshold, and Responses compact endpoint path SHALL be configurable.
|
|
|
|
When only the configured remaining context ratio is available, the summary pipeline SHALL try to compact the conversation through the configured OpenAI-compatible `POST /v1/responses/compact` endpoint. If that endpoint is unavailable or returns invalid data, the pipeline SHALL fall back to Microsoft Agent Framework compaction.
|
|
|
|
Fallback compaction SHALL become increasingly aggressive: collapse old tool results, summarize older conversation spans, keep only the last four user turns, and drop oldest groups if still over budget.
|
|
|
|
When summary generation fails after retries, Meeting Assistant SHALL write a markdown failure report to the configured summary note path. The failure report SHALL include a clickable retry link, error details, and the meeting artifact paths needed to diagnose or retry the run.
|
|
|
|
After transcript processing finishes for a recording, Meeting Assistant SHALL automatically invoke the summary pipeline for the meeting.
|
|
|
|
Before automatic summary generation starts, Meeting Assistant SHALL wait for pending screenshot OCR jobs for that meeting to complete or reach their configured timeout.
|
|
|
|
Meeting Assistant SHALL expose an API operation to retry summary generation for a given summary note path. The retry operation SHALL resolve the linked meeting artifacts from the meeting note frontmatter and SHALL overwrite the existing summary note with either the new summary or a new failure report.
|
|
|
|
The summary pipeline SHALL expose tools scoped to the current meeting:
|
|
|
|
- `read_meetingnote`
|
|
- `read_transcript`
|
|
- `read_context`
|
|
- `read_usernotes`
|
|
- `read_glossary`
|
|
- `add_dictation_word`
|
|
- `add_attendee`
|
|
- `remove_attendee`
|
|
- `override_speaker`
|
|
- `delete_identity`
|
|
- `write_summary`
|
|
- `write_context`
|
|
- `list_projects`
|
|
- `list_projectfiles`
|
|
- `read_projectfile`
|
|
- `write_projectfile`
|
|
- `search`
|
|
|
|
All summary-agent read tools that return file-like content SHALL support either reading the whole file or reading a clamped inclusive 1-based line range when `from` and `to` are supplied.
|
|
|
|
The summary pipeline SHALL write the summary note as a markdown artifact linked to the meeting note, transcript, and assistant context.
|
|
|
|
The summary instructions SHALL tell the summary agent to include only the most relevant cropped screenshot links from assistant context in the summary, placing them near the related summary text.
|
|
|
|
The summary agent SHALL be able to read the meeting note, transcript, assistant context, glossary, and bound project files through tools.
|
|
|
|
The summary agent SHALL be able to write the summary and assistant context files as its owned artifacts.
|
|
|
|
For summary-agent writes to any file other than the summary file and assistant context file, Meeting Assistant SHALL record an in-memory diff containing removed and added lines.
|
|
|
|
Meeting Assistant SHALL use a diff implementation that can reduce whole-file rewrites to changed lines instead of treating every rewrite as a full replacement.
|
|
|
|
When the summary agent finishes, Meeting Assistant SHALL append the collected external write diffs to the assistant context file.
|
|
|
|
After writing the meeting summary, the summary agent SHALL update existing project files when the meeting produced durable project knowledge, decisions, next steps, or context.
|
|
|
|
The assistant context note SHALL keep `title`, `start_time`, `end_time`, `meeting`, `transcript`, `summary`, `state`, and `agenda` in frontmatter. Meeting Assistant SHALL write `title` and `start_time` when the assistant context note is created, update `title` when later meeting metadata is discovered, and write `end_time` when transcript processing stops. When a Teams appointment is detected at recording start, the assistant context note SHALL also keep `scheduled_end` in frontmatter. The `state` value SHALL be one of `collecting metadata`, `transcribing`, `speaker recognition`, `summarizing`, `finished`, or `error`.
|
|
|
|
The summary note SHALL keep frontmatter links to meeting artifacts and SHALL copy `title`, `start_time`, `end_time`, `attendees`, and `projects` from the meeting note when available. If the meeting note has no title, the summary agent SHALL be able to provide a title when writing the summary.
|
|
|
|
When the summary note already has an `attendees` frontmatter property, Meeting Assistant SHALL preserve the existing summary attendees instead of overwriting them from the meeting note.
|
|
|
|
The summary agent SHALL be able to read and write the assistant context body as its own notebook using overwrite, replace, and insert line modes.
|
|
|
|
The summary instructions SHALL tell the summary agent to use transcript evidence and screenshot OCR participant evidence to refine the meeting note attendee list conservatively, adding or removing attendees only when the evidence is clear.
|
|
|
|
The summary instructions SHALL tell the summary agent to call `override_speaker` only when it is very certain that a transcript speaker label belongs to a named speaker, such as from user notes, OCR with a correlating meeting timestamp, or very clear context cues.
|
|
|
|
When `override_speaker` is called with an existing transcript speaker label and a replacement speaker name, Meeting Assistant SHALL rewrite that speaker label in the transcript and record the override for final speaker identity processing.
|
|
|
|
When `override_speaker` is called without merge enabled and the transcript already contains the replacement speaker name as a speaker label, Meeting Assistant SHALL refuse the override. When merge is enabled, Meeting Assistant SHALL allow the override, rewrite the source speaker label to the replacement speaker name, and record the override as a speaker merge for final speaker identity processing.
|
|
|
|
The summary instructions SHALL tell the summary agent to call `delete_identity` only when it is certain that an existing speaker identity was wrongfully matched.
|
|
|
|
When `delete_identity` is called with an existing transcript speaker identity, Meeting Assistant SHALL rewrite transcript speaker labels for that identity to `Removed-<n>` and record the deletion for final speaker identity processing.
|
|
|
|
#### Scenario: Assistant context note is initialized
|
|
- **WHEN** Meeting Assistant starts a meeting session
|
|
- **THEN** it creates the assistant context note with frontmatter links to the meeting note, transcript note, and summary note
|
|
- **AND** the assistant context frontmatter state is `collecting metadata`
|
|
- **AND** the assistant context frontmatter includes `agenda`, empty when no agenda is known
|
|
- **AND** the assistant context frontmatter includes `scheduled_end` when Teams metadata supplied a scheduled end time
|
|
|
|
#### Scenario: Assistant context state follows meeting processing
|
|
- **WHEN** transcription, final speaker recognition, and summary generation progress
|
|
- **THEN** Meeting Assistant updates assistant context frontmatter state to `collecting metadata`, `transcribing`, `speaker recognition`, `summarizing`, `finished`, or `error` as appropriate
|
|
- **AND** preserves existing `agenda` and `scheduled_end` frontmatter values
|
|
|
|
#### Scenario: Summary copies final meeting attendees
|
|
- **WHEN** the summary agent writes a summary note after meeting attendees have changed
|
|
- **THEN** the summary note frontmatter includes the final meeting note attendees
|
|
|
|
#### Scenario: Existing summary attendees are preserved
|
|
- **GIVEN** an existing summary note has `attendees` in frontmatter
|
|
- **WHEN** the summary agent rewrites the summary
|
|
- **THEN** Meeting Assistant preserves the existing summary attendees
|
|
|
|
#### Scenario: Summary copies final meeting projects
|
|
- **WHEN** the summary agent writes a summary note after meeting projects have changed
|
|
- **THEN** the summary note frontmatter includes the final meeting note projects
|
|
|
|
#### Scenario: Summary waits for pending screenshot OCR
|
|
- **GIVEN** screenshot OCR is still running for a meeting
|
|
- **WHEN** transcript processing is ready to run the automatic summary
|
|
- **THEN** Meeting Assistant waits for the screenshot OCR to complete before moving the assistant context to `summarizing`
|
|
- **AND** proceeds after the configured OCR timeout if OCR does not finish
|
|
|
|
#### Scenario: Summary can include relevant cropped screenshots
|
|
- **GIVEN** assistant context contains cropped screenshot markdown links
|
|
- **WHEN** Meeting Assistant builds summary-agent instructions
|
|
- **THEN** the instructions tell the summary agent to include only the most relevant cropped screenshots in the summary
|
|
- **AND** to markdown-link them near the related summary text
|
|
|
|
#### Scenario: External project write is audited
|
|
- **WHEN** the summary agent writes a bound project file
|
|
- **THEN** Meeting Assistant records the changed removed and added lines for that project file
|
|
- **AND** appends that diff to the assistant context after the agent finishes
|
|
|
|
#### Scenario: Dictionary write is audited
|
|
- **WHEN** the summary agent adds a dictation word and the dictionary file changes
|
|
- **THEN** Meeting Assistant records the changed removed and added lines for the dictionary file
|
|
- **AND** appends that diff to the assistant context after the agent finishes
|
|
|
|
#### Scenario: Owned artifact writes are not audited
|
|
- **WHEN** the summary agent writes the summary or assistant context file
|
|
- **THEN** Meeting Assistant does not include those writes in the external write diff audit
|
|
|
|
#### Scenario: Assistant context body writes do not duplicate frontmatter
|
|
- **WHEN** the summary agent writes assistant context content that accidentally includes a markdown frontmatter block
|
|
- **THEN** Meeting Assistant preserves the existing assistant context frontmatter
|
|
- **AND** writes only the replacement body content after that frontmatter
|
|
|
|
#### Scenario: Summary agent refines attendees from clear participant evidence
|
|
- **GIVEN** assistant context OCR states that visible meeting participants are complete
|
|
- **AND** the OCR result names a participant missing from the meeting note
|
|
- **WHEN** the summary pipeline runs
|
|
- **THEN** the summary agent can add that attendee to the meeting note through `add_attendee`
|
|
|
|
#### Scenario: Summary agent does not over-trim partial participant evidence
|
|
- **GIVEN** assistant context OCR states that visible meeting participants are only a partial result
|
|
- **WHEN** the summary pipeline runs
|
|
- **THEN** the summary instructions tell the agent not to remove attendees solely because they are absent from that partial screenshot evidence
|
|
|
|
#### Scenario: Summary agent overrides a certain speaker label
|
|
- **GIVEN** the transcript contains speaker label `Guest-01`
|
|
- **AND** user notes or OCR evidence clearly identifies that speaker as `Sabrina`
|
|
- **WHEN** the summary agent calls `override_speaker` with `Guest-01` and `Sabrina`
|
|
- **THEN** Meeting Assistant rewrites the transcript speaker label to `Sabrina`
|
|
- **AND** records the override for final speaker identity processing
|
|
|
|
#### Scenario: Summary agent cannot accidentally merge speakers
|
|
- **GIVEN** the transcript contains speaker labels `Guest-01` and `Sabrina`
|
|
- **WHEN** the summary agent calls `override_speaker` with `Guest-01` and `Sabrina` without merge enabled
|
|
- **THEN** Meeting Assistant refuses the override
|
|
- **AND** keeps the transcript unchanged
|
|
|
|
#### Scenario: Summary agent explicitly merges speakers
|
|
- **GIVEN** the transcript contains speaker labels `Guest-01` and `Sabrina`
|
|
- **AND** user notes or OCR evidence clearly proves they are the same person
|
|
- **WHEN** the summary agent calls `override_speaker` with `Guest-01`, `Sabrina`, and merge enabled
|
|
- **THEN** Meeting Assistant rewrites `Guest-01` transcript speaker labels to `Sabrina`
|
|
- **AND** records the override as a merge for final speaker identity processing
|
|
|
|
#### Scenario: Summary agent deletes a wrong speaker identity
|
|
- **GIVEN** the transcript contains speaker identity `Sabrina`
|
|
- **AND** user notes or OCR evidence clearly proves that `Sabrina` was wrongfully matched
|
|
- **WHEN** the summary agent calls `delete_identity` with `Sabrina`
|
|
- **THEN** Meeting Assistant rewrites the transcript speaker label to `Removed-1`
|
|
- **AND** records the deletion for final speaker identity processing
|
|
|
|
### Requirement: Meeting screenshots are captured into assistant context
|
|
Meeting Assistant SHALL expose a configurable screenshot hotkey.
|
|
|
|
When a meeting is active and the screenshot hotkey is pressed, Meeting Assistant SHALL capture the currently active window.
|
|
|
|
The screenshot image SHALL be saved into a configurable attachments folder for the assistant context note. By default, the folder SHALL be `Attachments` beside the assistant context note.
|
|
|
|
After the image is saved, Meeting Assistant SHALL append a markdown image link to the assistant context note with a meeting-relative timestamp that correlates to transcript timestamps.
|
|
|
|
Meeting Assistant SHALL allow optional screenshot OCR configuration with endpoint URL, API key or key environment variable, model, prompt, and timeout.
|
|
|
|
When screenshot OCR is configured, Meeting Assistant SHALL send the screenshot and prompt to the configured OpenAI-compatible Responses endpoint and append the model result after the screenshot link in the assistant context note.
|
|
|
|
The screenshot OCR prompt SHALL ask the model to return pixel crop coordinates when it can confidently isolate only the presentation, shared screen, or similarly relevant meeting content.
|
|
|
|
When OCR returns valid crop coordinates within the original image bounds, Meeting Assistant SHALL save a cropped PNG beside the original screenshot and SHALL link the cropped image before the OCR result in the assistant context note.
|
|
|
|
When OCR returns no crop coordinates or invalid crop coordinates, Meeting Assistant SHALL keep the original screenshot link and OCR result without writing a cropped image.
|
|
|
|
When screenshot OCR is not configured, Meeting Assistant SHALL skip OCR and keep the screenshot link.
|
|
|
|
The default OCR prompt SHALL explain that the image is from a meeting and ask the model to identify who is talking, who is presenting, what is presented, capture slide text in markdown, convert diagrams to Mermaid when possible, indicate whether visible people are clearly the exact meeting participants or only a partial result, return crop coordinates only for confidently isolated presentation/shared-screen content, and otherwise describe the scene.
|
|
|
|
#### Scenario: Screenshot is linked with meeting timestamp
|
|
- **GIVEN** a meeting started at `10:00:00`
|
|
- **WHEN** the user captures a screenshot at `10:03:05`
|
|
- **THEN** Meeting Assistant saves the screenshot under the configured attachments folder
|
|
- **AND** appends a markdown image link to assistant context with timestamp `[00:03:05]`
|
|
|
|
#### Scenario: OCR result is appended after screenshot
|
|
- **GIVEN** screenshot OCR is configured
|
|
- **WHEN** the user captures a screenshot
|
|
- **THEN** Meeting Assistant appends the screenshot link to assistant context
|
|
- **AND** appends the OCR result for that screenshot after the link when processing completes
|
|
|
|
#### Scenario: OCR crop is saved and linked before OCR text
|
|
- **GIVEN** screenshot OCR is configured
|
|
- **AND** OCR returns valid crop coordinates for a shared screen
|
|
- **WHEN** OCR processing completes
|
|
- **THEN** Meeting Assistant saves a cropped screenshot beside the original image
|
|
- **AND** links the cropped screenshot before the OCR text in assistant context
|
|
|
|
#### Scenario: OCR is skipped when not configured
|
|
- **GIVEN** screenshot OCR is not configured
|
|
- **WHEN** the user captures a screenshot
|
|
- **THEN** Meeting Assistant saves and links the screenshot without calling a model endpoint
|
|
|
|
#### Scenario: OCR reports whether visible people are complete or partial
|
|
- **WHEN** Meeting Assistant uses the built-in screenshot OCR prompt
|
|
- **THEN** the prompt asks the model to state whether the screenshot clearly shows exactly who is in the meeting or only a partial participant result
|
|
|
|
### Requirement: Summary agent instructions are configurable
|
|
Meeting Assistant SHALL allow the summary agent initial prompt to be configured through application settings.
|
|
|
|
When no configured initial prompt is supplied, Meeting Assistant SHALL use the built-in default prompt.
|
|
|
|
#### Scenario: Configured initial prompt is used
|
|
- **WHEN** `MeetingAssistant:Agent:InitialPrompt` is configured with non-empty content
|
|
- **THEN** the summary agent uses that configured prompt as its base instructions
|
|
|
|
#### Scenario: Empty initial prompt falls back to built-in default
|
|
- **WHEN** `MeetingAssistant:Agent:InitialPrompt` is missing or blank
|
|
- **THEN** the summary agent uses the built-in default instructions
|
|
|
|
### Requirement: Summary agent receives bound project instructions
|
|
Meeting Assistant SHALL append project instructions to the summary agent instructions for projects bound to the meeting note frontmatter.
|
|
|
|
For each bound project folder under the configured projects folder, when an `AGENTS.md` file exists directly in that project folder, Meeting Assistant SHALL append the project name and the `AGENTS.md` content under a project instructions section.
|
|
|
|
When no bound projects have `AGENTS.md`, Meeting Assistant SHALL not append the project instructions section.
|
|
|
|
#### Scenario: Project AGENTS files are appended
|
|
- **GIVEN** the meeting note frontmatter lists projects `Alpha` and `Beta`
|
|
- **AND** both configured project folders contain `AGENTS.md`
|
|
- **WHEN** the summary agent is created
|
|
- **THEN** its instructions include `---`, `projects:`, `# Alpha`, Alpha's `AGENTS.md`, `# Beta`, and Beta's `AGENTS.md`
|
|
|
|
#### Scenario: Projects without AGENTS files are skipped
|
|
- **GIVEN** the meeting note frontmatter lists project `Alpha`
|
|
- **AND** the configured project folder does not contain `AGENTS.md`
|
|
- **WHEN** the summary agent is created
|
|
- **THEN** no empty project instruction entry is appended for `Alpha`
|
|
|
|
### Requirement: Summary note includes a one-line blurb
|
|
The `write_summary` tool SHALL require a one-line summary parameter in addition to the markdown summary body.
|
|
|
|
Meeting Assistant SHALL reject `write_summary` calls when the one-line summary contains line breaks.
|
|
|
|
Meeting Assistant SHALL write the one-line summary into summary note frontmatter as `oneliner`.
|
|
|
|
The summary-agent instructions SHALL tell the agent to keep the one-line summary very short, blurb-like, and concise, with complete sentences not required.
|
|
|
|
#### Scenario: Summary one-liner is written to frontmatter
|
|
- **WHEN** the summary agent calls `write_summary` with markdown content and one-line summary `Architecture risks and next steps`
|
|
- **THEN** Meeting Assistant writes `oneliner: Architecture risks and next steps` to the summary note frontmatter
|
|
|
|
#### Scenario: Multiline one-liner is rejected
|
|
- **WHEN** the summary agent calls `write_summary` with a one-line summary containing a line break
|
|
- **THEN** Meeting Assistant refuses the write
|
|
- **AND** does not mark the summary as written
|