meeting-assistant/openspec/specs/meeting-summary/spec.md

meeting-summary Specification

Purpose

TBD - created by archiving change define-meeting-assistant-v1. Update Purpose after archive.

Requirements

Requirement: Meeting Assistant generates meeting outputs

Meeting Assistant SHALL generate a summary, decisions, and next steps from the meeting transcript, metadata, user notes, and assistant-discovered context.

Meeting Assistant SHALL use a Microsoft Agent Framework pipeline for the first summary implementation. The pipeline SHALL use a configurable OpenAI-compatible endpoint, model, and API key source, including support for a direct key or an environment variable name.

The summary pipeline SHALL retry transient model endpoint failures according to configurable reconnection attempts and delay settings.

The summary pipeline SHALL track context-window usage for the configured model using response usage when available and request-size estimates otherwise. The context-window limit, maximum output reserve, compaction enablement, compaction threshold, and Responses compact endpoint path SHALL be configurable.

When only the configured remaining context ratio is available, the summary pipeline SHALL try to compact the conversation through the configured OpenAI-compatible POST /v1/responses/compact endpoint. If that endpoint is unavailable or returns invalid data, the pipeline SHALL fall back to Microsoft Agent Framework compaction.

Fallback compaction SHALL become increasingly aggressive: collapse old tool results, summarize older conversation spans, keep only the last four user turns, and drop oldest groups if still over budget.

When summary generation fails after retries, Meeting Assistant SHALL write a markdown failure report to the configured summary note path. The failure report SHALL include a clickable retry link, error details, and the meeting artifact paths needed to diagnose or retry the run.

After transcript processing finishes for a recording, Meeting Assistant SHALL automatically invoke the summary pipeline for the meeting.

Before automatic summary generation starts, Meeting Assistant SHALL wait for pending screenshot OCR jobs for that meeting to complete or reach their configured timeout.

Meeting Assistant SHALL expose an API operation to retry summary generation for a given summary note path. The retry operation SHALL resolve the linked meeting artifacts from the meeting note frontmatter and SHALL overwrite the existing summary note with either the new summary or a new failure report.

The summary pipeline SHALL expose tools scoped to the current meeting:

• read_meetingnote
• read_transcript
• read_context
• read_usernotes
• read_glossary
• add_dictation_word
• add_attendee
• remove_attendee
• override_speaker
• delete_identity
• write_summary
• write_context
• list_projects
• list_projectfiles
• read_projectfile
• write_projectfile
• search

All summary-agent read tools that return file-like content SHALL support either reading the whole file or reading a clamped inclusive 1-based line range when from and to are supplied.

The summary pipeline SHALL write the summary note as a markdown artifact linked to the meeting note, transcript, and assistant context.

The summary instructions SHALL tell the summary agent to include only the most relevant cropped screenshot links from assistant context in the summary, placing them near the related summary text.

The summary agent SHALL be able to read the meeting note, transcript, assistant context, glossary, and bound project files through tools.

The summary agent SHALL be able to write the summary and assistant context files as its owned artifacts.

For summary-agent writes to any file other than the summary file and assistant context file, Meeting Assistant SHALL record an in-memory diff containing removed and added lines.

Meeting Assistant SHALL use a diff implementation that can reduce whole-file rewrites to changed lines instead of treating every rewrite as a full replacement.

When the summary agent finishes, Meeting Assistant SHALL append the collected external write diffs to the assistant context file.

After writing the meeting summary, the summary agent SHALL update existing project files when the meeting produced durable project knowledge, decisions, next steps, or context.

The assistant context note SHALL keep title, start_time, end_time, meeting, transcript, summary, state, and agenda in frontmatter. Meeting Assistant SHALL write title and start_time when the assistant context note is created, update title when later meeting metadata is discovered, and write end_time when transcript processing stops. When a Teams appointment is detected at recording start, the assistant context note SHALL also keep scheduled_end in frontmatter. The state value SHALL be one of collecting metadata, transcribing, speaker recognition, summarizing, finished, or error.

The summary note SHALL keep frontmatter links to meeting artifacts and SHALL copy title, start_time, end_time, attendees, and projects from the meeting note when available. If the meeting note has no title, the summary agent SHALL be able to provide a title when writing the summary.

When the summary note already has an attendees frontmatter property, Meeting Assistant SHALL preserve the existing summary attendees instead of overwriting them from the meeting note.

The summary agent SHALL be able to read and write the assistant context body as its own notebook using overwrite, replace, and insert line modes.

The summary instructions SHALL tell the summary agent to use transcript evidence and screenshot OCR participant evidence to refine the meeting note attendee list conservatively, adding or removing attendees only when the evidence is clear.

The summary instructions SHALL tell the summary agent to call override_speaker only when it is very certain that a transcript speaker label belongs to a named speaker, such as from user notes, OCR with a correlating meeting timestamp, or very clear context cues.

When override_speaker is called with an existing transcript speaker label and a replacement speaker name, Meeting Assistant SHALL rewrite that speaker label in the transcript and record the override for final speaker identity processing.

When override_speaker is called without merge enabled and the transcript already contains the replacement speaker name as a speaker label, Meeting Assistant SHALL refuse the override. When merge is enabled, Meeting Assistant SHALL allow the override, rewrite the source speaker label to the replacement speaker name, and record the override as a speaker merge for final speaker identity processing.

The summary instructions SHALL tell the summary agent to call delete_identity only when it is certain that an existing speaker identity was wrongfully matched.

When delete_identity is called with an existing transcript speaker identity, Meeting Assistant SHALL rewrite transcript speaker labels for that identity to Removed-<n> and record the deletion for final speaker identity processing.

Scenario: Assistant context note is initialized

• WHEN Meeting Assistant starts a meeting session
• THEN it creates the assistant context note with frontmatter links to the meeting note, transcript note, and summary note
• AND the assistant context frontmatter state is collecting metadata
• AND the assistant context frontmatter includes agenda, empty when no agenda is known
• AND the assistant context frontmatter includes scheduled_end when Teams metadata supplied a scheduled end time

Scenario: Assistant context state follows meeting processing

• WHEN transcription, final speaker recognition, and summary generation progress
• THEN Meeting Assistant updates assistant context frontmatter state to collecting metadata, transcribing, speaker recognition, summarizing, finished, or error as appropriate
• AND preserves existing agenda and scheduled_end frontmatter values

Scenario: Summary copies final meeting attendees

• WHEN the summary agent writes a summary note after meeting attendees have changed
• THEN the summary note frontmatter includes the final meeting note attendees

Scenario: Existing summary attendees are preserved

• GIVEN an existing summary note has attendees in frontmatter
• WHEN the summary agent rewrites the summary
• THEN Meeting Assistant preserves the existing summary attendees

Scenario: Summary copies final meeting projects

• WHEN the summary agent writes a summary note after meeting projects have changed
• THEN the summary note frontmatter includes the final meeting note projects

Scenario: Summary waits for pending screenshot OCR

• GIVEN screenshot OCR is still running for a meeting
• WHEN transcript processing is ready to run the automatic summary
• THEN Meeting Assistant waits for the screenshot OCR to complete before moving the assistant context to summarizing
• AND proceeds after the configured OCR timeout if OCR does not finish

Scenario: Summary can include relevant cropped screenshots

• GIVEN assistant context contains cropped screenshot markdown links
• WHEN Meeting Assistant builds summary-agent instructions
• THEN the instructions tell the summary agent to include only the most relevant cropped screenshots in the summary
• AND to markdown-link them near the related summary text

Scenario: External project write is audited

• WHEN the summary agent writes a bound project file
• THEN Meeting Assistant records the changed removed and added lines for that project file
• AND appends that diff to the assistant context after the agent finishes

Scenario: Dictionary write is audited

• WHEN the summary agent adds a dictation word and the dictionary file changes
• THEN Meeting Assistant records the changed removed and added lines for the dictionary file
• AND appends that diff to the assistant context after the agent finishes

Scenario: Owned artifact writes are not audited

• WHEN the summary agent writes the summary or assistant context file
• THEN Meeting Assistant does not include those writes in the external write diff audit

Scenario: Assistant context body writes do not duplicate frontmatter

• WHEN the summary agent writes assistant context content that accidentally includes a markdown frontmatter block
• THEN Meeting Assistant preserves the existing assistant context frontmatter
• AND writes only the replacement body content after that frontmatter

Scenario: Summary agent refines attendees from clear participant evidence

• GIVEN assistant context OCR states that visible meeting participants are complete
• AND the OCR result names a participant missing from the meeting note
• WHEN the summary pipeline runs
• THEN the summary agent can add that attendee to the meeting note through add_attendee

Scenario: Summary agent does not over-trim partial participant evidence

• GIVEN assistant context OCR states that visible meeting participants are only a partial result
• WHEN the summary pipeline runs
• THEN the summary instructions tell the agent not to remove attendees solely because they are absent from that partial screenshot evidence

Scenario: Summary agent overrides a certain speaker label

• GIVEN the transcript contains speaker label Guest-01
• AND user notes or OCR evidence clearly identifies that speaker as Sabrina
• WHEN the summary agent calls override_speaker with Guest-01 and Sabrina
• THEN Meeting Assistant rewrites the transcript speaker label to Sabrina
• AND records the override for final speaker identity processing

Scenario: Summary agent cannot accidentally merge speakers

• GIVEN the transcript contains speaker labels Guest-01 and Sabrina
• WHEN the summary agent calls override_speaker with Guest-01 and Sabrina without merge enabled
• THEN Meeting Assistant refuses the override
• AND keeps the transcript unchanged

Scenario: Summary agent explicitly merges speakers

• GIVEN the transcript contains speaker labels Guest-01 and Sabrina
• AND user notes or OCR evidence clearly proves they are the same person
• WHEN the summary agent calls override_speaker with Guest-01, Sabrina, and merge enabled
• THEN Meeting Assistant rewrites Guest-01 transcript speaker labels to Sabrina
• AND records the override as a merge for final speaker identity processing

Scenario: Summary agent deletes a wrong speaker identity

• GIVEN the transcript contains speaker identity Sabrina
• AND user notes or OCR evidence clearly proves that Sabrina was wrongfully matched
• WHEN the summary agent calls delete_identity with Sabrina
• THEN Meeting Assistant rewrites the transcript speaker label to Removed-1
• AND records the deletion for final speaker identity processing

Requirement: Meeting screenshots are captured into assistant context

Meeting Assistant SHALL expose a configurable screenshot hotkey.

When a meeting is active and the screenshot hotkey is pressed, Meeting Assistant SHALL capture the currently active window.

The screenshot image SHALL be saved into a configurable attachments folder for the assistant context note. By default, the folder SHALL be Attachments beside the assistant context note.

After the image is saved, Meeting Assistant SHALL append a markdown image link to the assistant context note with a meeting-relative timestamp that correlates to transcript timestamps.

Meeting Assistant SHALL allow optional screenshot OCR configuration with endpoint URL, API key or key environment variable, model, prompt, and timeout.

When screenshot OCR is configured, Meeting Assistant SHALL send the screenshot and prompt to the configured OpenAI-compatible Responses endpoint and append the model result after the screenshot link in the assistant context note.

The screenshot OCR prompt SHALL ask the model to return pixel crop coordinates when it can confidently isolate only the presentation, shared screen, or similarly relevant meeting content.

When OCR returns valid crop coordinates within the original image bounds, Meeting Assistant SHALL save a cropped PNG beside the original screenshot and SHALL link the cropped image before the OCR result in the assistant context note.

When OCR returns no crop coordinates or invalid crop coordinates, Meeting Assistant SHALL keep the original screenshot link and OCR result without writing a cropped image.

When screenshot OCR is not configured, Meeting Assistant SHALL skip OCR and keep the screenshot link.

The default OCR prompt SHALL explain that the image is from a meeting and ask the model to identify who is talking, who is presenting, what is presented, capture slide text in markdown, convert diagrams to Mermaid when possible, indicate whether visible people are clearly the exact meeting participants or only a partial result, return crop coordinates only for confidently isolated presentation/shared-screen content, and otherwise describe the scene.

Scenario: Screenshot is linked with meeting timestamp

• GIVEN a meeting started at 10:00:00
• WHEN the user captures a screenshot at 10:03:05
• THEN Meeting Assistant saves the screenshot under the configured attachments folder
• AND appends a markdown image link to assistant context with timestamp [00:03:05]

Scenario: OCR result is appended after screenshot

• GIVEN screenshot OCR is configured
• WHEN the user captures a screenshot
• THEN Meeting Assistant appends the screenshot link to assistant context
• AND appends the OCR result for that screenshot after the link when processing completes

Scenario: OCR crop is saved and linked before OCR text

• GIVEN screenshot OCR is configured
• AND OCR returns valid crop coordinates for a shared screen
• WHEN OCR processing completes
• THEN Meeting Assistant saves a cropped screenshot beside the original image
• AND links the cropped screenshot before the OCR text in assistant context

Scenario: OCR is skipped when not configured

• GIVEN screenshot OCR is not configured
• WHEN the user captures a screenshot
• THEN Meeting Assistant saves and links the screenshot without calling a model endpoint

Scenario: OCR reports whether visible people are complete or partial

• WHEN Meeting Assistant uses the built-in screenshot OCR prompt
• THEN the prompt asks the model to state whether the screenshot clearly shows exactly who is in the meeting or only a partial participant result

Requirement: Summary agent instructions are configurable

Meeting Assistant SHALL allow the summary agent initial prompt to be configured through application settings.

When no configured initial prompt is supplied, Meeting Assistant SHALL use the built-in default prompt.

Scenario: Configured initial prompt is used

• WHEN MeetingAssistant:Agent:InitialPrompt is configured with non-empty content
• THEN the summary agent uses that configured prompt as its base instructions

Scenario: Empty initial prompt falls back to built-in default

• WHEN MeetingAssistant:Agent:InitialPrompt is missing or blank
• THEN the summary agent uses the built-in default instructions

Requirement: Summary agent receives bound project instructions

Meeting Assistant SHALL append project instructions to the summary agent instructions for projects bound to the meeting note frontmatter.

For each bound project folder under the configured projects folder, when an AGENTS.md file exists directly in that project folder, Meeting Assistant SHALL append the project name and the AGENTS.md content under a project instructions section.

When no bound projects have AGENTS.md, Meeting Assistant SHALL not append the project instructions section.

Scenario: Project AGENTS files are appended

• GIVEN the meeting note frontmatter lists projects Alpha and Beta
• AND both configured project folders contain AGENTS.md
• WHEN the summary agent is created
• THEN its instructions include ---, projects:, # Alpha, Alpha's AGENTS.md, # Beta, and Beta's AGENTS.md

Scenario: Projects without AGENTS files are skipped

• GIVEN the meeting note frontmatter lists project Alpha
• AND the configured project folder does not contain AGENTS.md
• WHEN the summary agent is created
• THEN no empty project instruction entry is appended for Alpha

Requirement: Summary note includes a one-line blurb

The write_summary tool SHALL require a one-line summary parameter in addition to the markdown summary body.

Meeting Assistant SHALL reject write_summary calls when the one-line summary contains line breaks.

Meeting Assistant SHALL write the one-line summary into summary note frontmatter as oneliner.

The summary-agent instructions SHALL tell the agent to keep the one-line summary very short, blurb-like, and concise, with complete sentences not required.

Scenario: Summary one-liner is written to frontmatter

• WHEN the summary agent calls write_summary with markdown content and one-line summary Architecture risks and next steps
• THEN Meeting Assistant writes oneliner: Architecture risks and next steps to the summary note frontmatter

Scenario: Multiline one-liner is rejected

• WHEN the summary agent calls write_summary with a one-line summary containing a line break
• THEN Meeting Assistant refuses the write
• AND does not mark the summary as written