# Meeting Assistant Meeting Assistant is a .NET 10 server application for capturing and enriching meetings. Its core purpose is to work with any kind of meeting, including in-person meetings. Integrations with platforms such as Teams, Zoom, or similar tools may augment the experience later, but the system must not depend on those product APIs for its primary meeting flow. The application is intended to: - create an Obsidian markdown note before transcription starts - keep meeting metadata, user notes, detected context, and links to generated output in that note - toggle recording/transcription mode through a configurable global hotkey - switch the active transcription profile during a meeting with a profile-specific toggle hotkey - abort and discard an active recording through a configurable global hotkey - show a Windows taskbar notification icon with state-aware recording controls - capture active-window screenshots through a configurable global hotkey - capture microphone input and computer output into one transcription stream - transcribe meetings with speaker attribution - use a configurable speech recognition pipeline, with local Whisper as the first version 1 fallback - use FunASR or local Whisper plus pyannote as configurable speaker-attribution paths - write transcript markdown files into the configured vault folder - generate summaries, decisions, and next steps - build and maintain a project knowledge base from meetings and future context sources - use Microsoft Agent Framework-based agents to reason over meeting and project context - give agents tools for project lookup, keyword lookup, retrieval-augmented generation, and project file updates in existing projects ## Repository Status This repository currently contains the initial .NET 10 application skeleton, a health endpoint, and the first OpenSpec change for version 1. ## Local Development Restore, build, and test: ```powershell dotnet restore MeetingAssistant.slnx dotnet build MeetingAssistant.slnx dotnet test MeetingAssistant.slnx ``` Run the server: ```powershell dotnet run --project MeetingAssistant ``` ## Local Autostart and Restart For the local Windows workstation, use the snippets restart helper instead of running the app directly when you want a durable background instance: ```powershell powershell -ExecutionPolicy Bypass -File C:\Manuel\snippets\restart-meeting-assistant.ps1 ``` The restart helper publishes `MeetingAssistant\MeetingAssistant.csproj` as `net10.0-windows` into a fresh temp folder under `C:\Manuel\meeting-assistant\tmp\meeting-assistant-runtime`. Only after that publish succeeds does it stop any active Meeting Assistant instance on port `5090` and start the newly published executable detached from the script. The detached starter writes logs and the last process id to: ```text %LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.out.log %LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.err.log %LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.pid ``` This script is also the intended autostart target. Add a Windows Startup shortcut to `C:\Manuel\snippets\restart-meeting-assistant.ps1` when the app should rebuild and restart automatically after login. The initial endpoints are: ```text GET /health GET /recording/status POST /recording/toggle POST /recording/start POST /recording/stop POST /recording/abort POST /asr/transcribe-file POST /asr/diarize-file POST /diagnostics/workflow/reload POST /meetings/current/summary/run POST /meetings/summary/retry GET /meetings/summary/retry?summaryPath=... ``` ## Configuration Meeting Assistant uses normal .NET configuration under the `MeetingAssistant` section: ```json { "MeetingAssistant": { "Hotkey": { "Toggle": "Ctrl+Alt+M", "Abort": "Ctrl+Alt+Z" }, "Vault": { "TranscriptsFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Transcripts", "MeetingNotesFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Notes", "AssistantContextFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Assistant Context", "SummariesFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Summaries", "ProjectsFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Projects", "DictationWordsPath": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\dictation-words.md" }, "Recording": { "TranscriptionProvider": "funasr", "SampleRate": 16000, "Channels": 1, "StopProcessingTimeout": "00:10:00", "MaxMetadataAttendeeImportCount": 30, "TemporaryRecordingsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Recordings" }, "FunAsr": { "Endpoint": "ws://127.0.0.1:10095", "Mode": "2pass", "ChunkSize": [5, 10, 5], "ChunkInterval": 10, "EncoderChunkLookBack": 4, "DecoderChunkLookBack": 1, "Itn": true, "EmitOnlinePartials": false, "WavName": "meeting", "FinalResultTimeout": "00:00:10", "Backend": { "Enabled": true, "DockerCommand": "docker", "Image": "registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.13", "ContainerName": "meeting-assistant-funasr", "ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\FunASR\\models", "ContainerPort": 10095, "Privileged": true, "CommandTimeout": "00:15:00", "StartupTimeout": "00:10:00", "StopOnDispose": true, "ServerCommand": "cd /workspace/FunASR/runtime && bash run_server_2pass.sh --download-model-dir /workspace/models --vad-dir damo/speech_fsmn_vad_zh-cn-16k-common-onnx --model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-onnx --online-model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online-onnx --punc-dir damo/punc_ct-transformer_zh-cn-common-vad_realtime-vocab272727-onnx --itn-dir thuduj12/fst_itn_zh --certfile 0 --hotword /workspace/models/hotwords.txt && tail -f /dev/null" }, "Diarization": { "Enabled": true, "AsrModel": "paraformer-zh", "VadModel": "fsmn-vad", "PunctuationModel": "ct-punc", "SpeakerModel": "cam++", "BatchSizeSeconds": 300, "BatchSizeThresholdSeconds": 60, "CommandTimeout": "00:30:00" } }, "WhisperLocal": { "ModelPath": "models\\ggml-base.bin", "Language": "auto", "WindowSeconds": 15, "Diarization": { "Enabled": true, "DockerCommand": "docker", "BaseImage": "python:3.11-slim", "Image": "meeting-assistant-pyannote:local", "BuildImage": true, "ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Pyannote\\models", "Model": "pyannote/speaker-diarization-3.1", "AnnotationSource": "SpeakerDiarization", "AlignmentMode": "PyannoteTurns", "TokenEnv": "HF_TOKEN", "CommandTimeout": "00:30:00" } }, "AzureSpeech": { "Endpoint": "https://germanywestcentral.api.cognitive.microsoft.com/", "Region": "germanywestcentral", "Language": "en-US", "KeyEnv": "AZURE_SPEECH_KEY", "RecognitionStopTimeout": "00:00:10", "DiarizeIntermediateResults": true }, "SpeakerIdentification": { "MinimumSampleSpeechDuration": "00:00:30", "MaximumSampleSegmentGap": "00:00:01", "PyannoteValidation": { "Enabled": false, "MinimumSingleSpeakerCoverage": 0.9, "MinimumMatchingKnownSnippetRatio": 0.5, "Diarization": { "Enabled": true, "DockerCommand": "docker", "Image": "meeting-assistant-pyannote:local", "BuildImage": true, "ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Pyannote\\models", "Model": "pyannote/speaker-diarization-3.1", "TokenEnv": "HF_TOKEN", "CommandTimeout": "00:30:00" } } }, "Automation": { "RulesPath": "meeting-rules.local.yaml" }, "WorkflowRulesEditor": { "Endpoint": "", "KeyEnv": "", "Model": "" }, "Screenshots": { "Hotkey": "Ctrl+Alt+S", "AttachmentsFolder": "Attachments", "Ocr": { "Enabled": false, "Endpoint": "", "KeyEnv": "LITELLM_API_KEY", "Model": "", "Timeout": "00:02:00", "Prompt": "This screenshot was captured during a meeting..." } }, "Agent": { "Endpoint": "https://litellm.schweigert.cloud", "KeyEnv": "LITELLM_API_KEY", "Model": "chatgpt/gpt-5.5", "EnableThinking": true, "ReasoningEffort": "Medium", "ReconnectionAttempts": 2, "ReconnectionDelay": "00:00:02", "ContextWindowTokens": 128000, "MaxOutputTokens": 8192, "EnableCompaction": true, "CompactionRemainingRatio": 0.10, "ResponsesCompactPath": "responses/compact" }, "Api": { "PublicBaseUrl": "http://localhost:5090" } } } ``` `funasr` streams 16 kHz PCM audio to the configured FunASR WebSocket endpoint. When `FunAsr:Backend:Enabled` is true, Meeting Assistant manages a local Docker-backed FunASR runtime: on application startup it begins a non-blocking warm-up that pulls the configured online streaming image, prepares the mounted model and hotword folder, replaces any stale container with the configured name, starts the two-pass WebSocket server mapped to the endpoint port, keeps the container alive after the FunASR startup script backgrounds the server process, and stops the container when the app shuts down. Recording can start while the backend is still warming up; captured audio is queued and then streamed once the WebSocket backend is healthy. Docker commands are bounded by `CommandTimeout`, and backend WebSocket readiness is bounded by `StartupTimeout`. FunASR response fields such as `spk_name`, `spk`, and sentence-level speaker metadata are preserved in transcript segments; missing speaker fields are written as `Unknown`. During recording, Meeting Assistant writes the mixed PCM stream to a temporary WAV under `TemporaryRecordingsFolder` only so the final diarization pass has audio to process. After capture stops and the streaming ASR provider drains the already-captured audio, FunASR Python `AutoModel` can run with VAD, punctuation, and CAMPPlus (`spk_model="cam++"`) over that temporary WAV. If sentence-level speaker labels are returned, the transcript markdown is rewritten with the final speaker-attributed segments. If final diarization is disabled or returns no segments, the live streaming transcript is kept. Temporary WAV files are deleted after the run completes, and stale temporary recordings from interrupted runs are deleted when the application starts. `whisper-local` uses Whisper.NET with a local ggml model and remains available by setting `Recording:TranscriptionProvider` to `whisper-local`. The model file is not committed to this repository; place it at the configured `ModelPath` before using live transcription. When `WhisperLocal:Diarization:Enabled` is true, the final post-processing pass runs pyannote in Docker over the temporary WAV and reads the Hugging Face token from `WhisperLocal:Diarization:Token` or `TokenEnv` (`HF_TOKEN` by default). `AnnotationSource` selects pyannote's `speaker_diarization` or `exclusive_speaker_diarization` output. `AlignmentMode` selects whether Meeting Assistant assigns the best-overlap pyannote speaker label to each Whisper segment or rebuilds the finished transcript as pyannote speaker-turn segments with matching Whisper text. When `BuildImage` is true, Meeting Assistant builds the configured local pyannote Docker image if Docker does not already have it, so Python packages are installed once instead of during every diarization call. The configured `ModelsFolder` is mounted as the persistent pyannote model cache and stores Hugging Face, torch, and pip cache artifacts so model downloads are reused across runs. If pyannote is disabled, has no token, or returns no turns, the live Whisper transcript is kept. `azure-speech` streams captured PCM audio to Azure AI Speech using the Speech SDK `ConversationTranscriber`. Configure `AzureSpeech:Endpoint`, `Region`, `Language`, and either `Key` or `KeyEnv`; this repository uses `AZURE_SPEECH_KEY` by default and does not store the key in appsettings. `DiarizeIntermediateResults` enables diarization for intermediate and final conversation transcription results. Azure returns generic speaker IDs such as `Guest-1` and `Guest-2`, which Meeting Assistant writes directly into live transcript segments. `RecognitionStopTimeout` bounds SDK shutdown after the captured audio stream has been closed, which keeps short diagnostic runs from waiting indefinitely for final service events. Since Azure Speech emits diarized live transcript segments, the Azure pipeline keeps those segments as the finished transcript instead of running pyannote. Speaker identity matching keeps candidate samples only after a diarized speaker has at least `SpeakerIdentification:MinimumSampleSpeechDuration` of continuous speech, defaulting to 30 seconds. Adjacent same-speaker segments may be combined when the gap is no larger than `MaximumSampleSegmentGap`, but a different speaker resets the pending span. `SpeakerIdentification:PyannoteValidation` is an optional secondary confidence layer. When enabled, pyannote rejects multi-speaker samples and must confirm an Azure-confirmed identity match before Meeting Assistant accepts it. It uses the same Docker-based pyannote runtime shape as local Whisper finalization and reads its Hugging Face token from the nested diarization options. ASR backends are exposed downstream as speech recognition pipelines. A pipeline can initialize, wait for readiness, accept audio chunks, emit live transcript segments, and produce a finished transcript after capture completes. The configured implementation is selected from `Recording:TranscriptionProvider` and registered through DI so recording and diagnostic endpoints do not need to know whether the backend uses FunASR, Whisper.NET, Azure Speech, pyannote, or a later provider. `POST /asr/transcribe-file` is a local diagnostic endpoint for the configured ASR backend. Send `{ "path": "C:\\path\\to\\sample.wav" }` with a 16-bit PCM WAV file to stream it through the configured speech recognition pipeline and return the emitted live transcript segments. For real meeting recordings, Meeting Assistant derives the optional finished-transcript `numSpeakers` hint from the meeting note `attendees` frontmatter. The hint is set to the attendee count only when at least two attendees are listed; empty and single-person attendee lists are ignored. For pyannote-backed finalization, that hint is passed to pyannote as `num_speakers`. `POST /asr/diarize-file` streams a local 16-bit PCM WAV through the configured speech recognition pipeline and returns the finished transcript. This is useful for checking the FunASR CAMPPlus path or the local Whisper plus pyannote path without starting a live recording. The request can include an optional speaker-count hint: `{ "path": "C:\\path\\to\\sample.wav", "numSpeakers": 5 }`. `DictationWordsPath` is reserved for providers that can accept vocabulary hints. It points to a markdown word list containing unusual project, person, product, or domain terms that should be biased during transcription when the provider supports it. Meeting notes link to separate transcript, assistant context, and summary markdown artifacts using the configured vault folders. `Hotkey:Abort` configures a global discard shortcut. The default is `Ctrl+Alt+Z`. Aborting an active recording stops capture/transcription, deletes the meeting note, transcript, assistant context, summary file if present, and linked screenshot attachments for that run, and skips automatic summary generation. Launch profile toggle hotkeys start a meeting when idle. When a different launch profile toggle is pressed during an active meeting, Meeting Assistant keeps the same meeting note, transcript, assistant context, and metadata; drains the current speech recognition pipeline without summarizing; appends a transcript marker; clears run-local speaker mappings; and continues transcription through the target profile. Pressing the same active profile toggle still stops the meeting normally. On Windows, Meeting Assistant shows a taskbar notification icon. The icon indicates idle, recording, or post-recording summarizing/processing state. Its right-click menu can start a recording for any configured launch profile, stop and transcribe the active recording, cancel and discard the active recording, or switch an active recording to another configured launch profile. A newer active recording takes priority in the icon state even if an older stopped run is still summarizing. Meeting note, transcript, assistant context, and summary artifacts use frontmatter links so they backlink to each other. Meeting note frontmatter includes `start_time` when recording starts and `end_time` when transcription processing finishes. Transcript and summary frontmatter copy the meeting title, start time, and end time from the meeting note; if the meeting has no title, the summary agent can provide one when writing the summary. `Recording:MaxMetadataAttendeeImportCount` limits how many attendees Outlook metadata enrichment imports into meeting-note frontmatter. The default is `30`; when an appointment has more attendees than that, Meeting Assistant still imports title, agenda, and scheduled end time, but leaves attendees empty because large invites are usually presentation-style meetings. Assistant context notes keep their artifact links in frontmatter and expose a lifecycle `state`: `collecting metadata`, `transcribing`, `speaker recognition`, `summarizing`, `finished`, or `error`. Meeting Assistant updates this state as metadata lookup, recording, final speaker recognition, and summary generation progress. `ProjectsFolder` contains project knowledge. Each direct subfolder is treated as one project, and a meeting binds projects by listing those subfolder names in the meeting note `projects` frontmatter. `Automation:RulesPath` points to an optional local YAML rules file. The default `meeting-rules.local.yaml` is ignored by git. Rules can trigger on meeting creation, assistant-context state transitions, or identified speakers; conditions are evaluated with NCalc-style expressions and step values can use Razor syntax against `Model.Meeting`, `Model.Event`, and `Model.Speaker`. The initial supported steps are `add_attendee`, `remove_attendee`, `set_property`, `add_context`, and `add_project`. See `docs/meeting-workflow-engine.md` for the detailed YAML format, supported variables, examples, and extension notes. `POST /diagnostics/workflow/reload` reloads application configuration so workflow automation settings such as `Automation:RulesPath` can be changed without restarting the application. A meeting run that already captured its options keeps using those options. `Screenshots:Hotkey` configures a global hotkey that captures the currently active window during an active meeting. Screenshots are written under `Screenshots:AttachmentsFolder`, which defaults to an `Attachments` folder beside the assistant context note, and each capture appends a timestamped markdown image link to the assistant context. Launch profiles can override the screenshot hotkey with `LaunchProfiles:{profile}:Screenshots:Hotkey`; only explicitly configured profile hotkeys are registered, so profile-specific hotkeys do not silently inherit and collide with the default profile. `Screenshots:Ocr` optionally enables vision extraction for screenshots. When `Enabled` is true, Meeting Assistant sends the screenshot and prompt to an OpenAI-compatible Responses endpoint and replaces the screenshot OCR placeholder with the model output. Blank `Endpoint` or `Model` values fall back to the summary `Agent` endpoint and model, which keeps local OCR on the same LiteLLM backend as summarization. `Key` or `KeyEnv` can be set specifically for OCR; otherwise the summary agent key configuration is used. The default prompt asks the model to say whether visible participants are clearly the exact meeting participants or only a partial result, and to return pixel crop coordinates when it can confidently isolate only the presentation or shared-screen content; valid crop boxes are saved as `*-cropped.png` beside the original screenshot and linked before the OCR block. Automatic summarization waits for pending screenshot OCR work to complete or hit `Timeout` before the assistant context moves to `summarizing`. `Agent` configures the Microsoft Agent Framework summary pipeline. `Key` may be set directly for local testing, but `KeyEnv` is preferred; by default the API key is read from `LITELLM_API_KEY`. After transcription has fully finished, Meeting Assistant automatically runs the summary pipeline for the meeting. The summary agent writes the full markdown summary through `write_summary` and must provide a required `oneliner` value, which is stored in summary frontmatter and must not contain line breaks. The summary agent can add and remove meeting-note attendees when transcript or OCR evidence is clear; partial participant evidence from screenshots should not be used to remove invitees by itself. It can also override a transcript speaker label with a named speaker only when the evidence is very certain, such as a user note, OCR at a correlating meeting timestamp, or a clear context cue. If that named speaker already exists in the transcript, `override_speaker` refuses by default; the agent must pass `merge=true` only when it is certain both labels are the same identity. When it is certain that a speaker identity was wrongfully matched, it can delete that identity, relabeling transcript speaker labels to `Removed-` and removing the local speaker identity database row before future matching. Final speaker identity learning and candidate updates run after the summary pipeline finishes so they use the summary-refined attendee list and any recorded speaker identity changes. `ReconnectionAttempts` and `ReconnectionDelay` control retries for transient model endpoint failures such as LiteLLM token refresh or 5xx responses. If summary generation still fails, Meeting Assistant overwrites the configured summary note with a markdown failure report containing the exception details and linked meeting artifacts. `ContextWindowTokens`, `MaxOutputTokens`, `EnableCompaction`, and `CompactionRemainingRatio` configure summary-agent context monitoring. Meeting Assistant estimates the outgoing Responses payload size, logs the estimated token count and remaining context, and compacts the conversation when only the configured remaining ratio is left. It first attempts `POST /v1/{ResponsesCompactPath}` on the configured OpenAI-compatible endpoint. If that endpoint is unavailable or returns invalid data, it falls back to Microsoft Agent Framework compaction: collapse old tool results, summarize older message groups, preserve only the last four user turns if needed, and finally drop oldest groups until the request is back under budget. The summary endpoints are: ```text POST /meetings/current/summary/run POST /meetings/summary/retry GET /meetings/summary/retry?summaryPath=... ``` `POST /meetings/summary/retry` accepts `{ "summaryPath": "20260519-124110-summary.md" }` or an absolute path under the configured summaries folder. It resolves the linked meeting note from frontmatter and reruns the summary pipeline, overwriting the existing summary file with either the new summary or a new failure report. Failure summary notes include a clickable retry link using `Api:PublicBaseUrl`, for example `http://localhost:5090/meetings/summary/retry?summaryPath=20260519-124110-summary.md`. The GET endpoint exists for Obsidian/browser clicks and performs the same overwrite behavior as the POST endpoint. Meeting note bodies stay reserved for user-authored notes. The summary agent currently has meeting-scoped tools to read the transcript, full meeting note including frontmatter, assistant context, user notes, glossary, and project files. Every read tool that returns file-like content can read the whole content or a clamped inclusive 1-based line range with `from` and `to`, so the agent can inspect large transcripts incrementally. The assistant context note is the agent's notebook and user-visible context window; the agent can write internal notes, discovered context, missing tool requests, suggested improvements, or follow-up task ideas there using line-based overwrite, replace, and insert modes. Project lookup and search tools are constrained to the projects listed in the meeting note frontmatter: When assistant context contains cropped screenshot links produced by OCR, the summary instructions tell the agent to include only the most relevant cropped screenshots in the summary and markdown-link them near the related summary text. - `read_meetingnote` - `list_projects` - `list_projectfiles` - `read_projectfile` - `write_projectfile` - `search` - `write_context` - `add_attendee` - `remove_attendee` - `override_speaker` - `delete_identity` `search` accepts ripgrep syntax and returns matches as `filename:line text`. If one project is searched, paths are relative to that project; if multiple projects are searched, paths include the project folder name. `write_projectfile` writes inside an existing project folder. With no line arguments it overwrites or creates the file. With `from` and `to` it replaces that inclusive 1-based line range. With `insert` it inserts content at that 1-based line position. ## Spec Workflow This repository is OpenSpec-driven. Before changing behavior, read: 1. `README.md` 2. `openspec/config.yaml` 3. existing specs in `openspec/specs` 4. active change specs in `openspec/changes/*/specs` The initial active change is: ```text openspec/changes/define-meeting-assistant-v1 ``` Validate it with: ```powershell openspec validate define-meeting-assistant-v1 --strict ```