Its core purpose is to work with any kind of meeting, including in-person meetings. Integrations with platforms such as Teams, Zoom, or similar tools may augment the experience later, but the system must not depend on those product APIs for its primary meeting flow.

The application is intended to:

create an Obsidian markdown note before transcription starts
keep meeting metadata, user notes, detected context, and links to generated output in that note
toggle recording/transcription mode through a configurable global hotkey
capture microphone input and computer output into one transcription stream
transcribe meetings with speaker attribution
use a configurable speech recognition pipeline, with local Whisper as the first version 1 fallback
use FunASR or local Whisper plus pyannote as configurable speaker-attribution paths
write transcript markdown files into the configured vault folder
generate summaries, decisions, and next steps
build and maintain a project knowledge base from meetings and future context sources
use Microsoft Agent Framework-based agents to reason over meeting and project context
give agents tools for project lookup, keyword lookup, retrieval-augmented generation, and project file updates in existing projects

Repository Status

This repository currently contains the initial .NET 10 application skeleton, a health endpoint, and the first OpenSpec change for version 1.

Local Development

Restore, build, and test:

dotnet restore MeetingAssistant.slnx
dotnet build MeetingAssistant.slnx
dotnet test MeetingAssistant.slnx

Run the server:

dotnet run --project MeetingAssistant

Local Autostart and Restart

For the local Windows workstation, use the snippets restart helper instead of running the app directly when you want a durable background instance:

powershell -ExecutionPolicy Bypass -File C:\Manuel\snippets\restart-meeting-assistant.ps1

The restart helper publishes MeetingAssistant\MeetingAssistant.csproj as net10.0-windows into a fresh temp folder under C:\Manuel\meeting-assistant\tmp\meeting-assistant-runtime. Only after that publish succeeds does it stop any active Meeting Assistant instance on port 5090 and start the newly published executable detached from the script.

The detached starter writes logs and the last process id to:

%LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.out.log
%LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.err.log
%LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.pid

This script is also the intended autostart target. Add a Windows Startup shortcut to C:\Manuel\snippets\restart-meeting-assistant.ps1 when the app should rebuild and restart automatically after login.

The initial endpoints are:

GET /health
GET /recording/status
POST /recording/toggle
POST /recording/start
POST /recording/stop
POST /asr/transcribe-file
POST /asr/diarize-file
POST /meetings/current/summary/run
POST /meetings/summary/retry
GET /meetings/summary/retry?summaryPath=...

Configuration

Meeting Assistant uses normal .NET configuration under the MeetingAssistant section:

{
  "MeetingAssistant": {
    "Hotkey": {
      "Toggle": "Ctrl+Alt+M"
    },
    "Vault": {
      "TranscriptsFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Transcripts",
      "MeetingNotesFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Notes",
      "AssistantContextFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Assistant Context",
      "SummariesFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Summaries",
      "ProjectsFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Projects",
      "DictationWordsPath": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\dictation-words.md"
    },
    "Recording": {
      "TranscriptionProvider": "funasr",
      "SampleRate": 16000,
      "Channels": 1,
      "StopProcessingTimeout": "00:10:00",
      "TemporaryRecordingsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Recordings"
    },
    "FunAsr": {
      "Endpoint": "ws://127.0.0.1:10095",
      "Mode": "2pass",
      "ChunkSize": [5, 10, 5],
      "ChunkInterval": 10,
      "EncoderChunkLookBack": 4,
      "DecoderChunkLookBack": 1,
      "Itn": true,
      "EmitOnlinePartials": false,
      "WavName": "meeting",
      "FinalResultTimeout": "00:00:10",
      "Backend": {
        "Enabled": true,
        "DockerCommand": "docker",
        "Image": "registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.13",
        "ContainerName": "meeting-assistant-funasr",
        "ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\FunASR\\models",
        "ContainerPort": 10095,
        "Privileged": true,
        "CommandTimeout": "00:15:00",
        "StartupTimeout": "00:10:00",
        "StopOnDispose": true,
        "ServerCommand": "cd /workspace/FunASR/runtime && bash run_server_2pass.sh --download-model-dir /workspace/models --vad-dir damo/speech_fsmn_vad_zh-cn-16k-common-onnx --model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-onnx --online-model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online-onnx --punc-dir damo/punc_ct-transformer_zh-cn-common-vad_realtime-vocab272727-onnx --itn-dir thuduj12/fst_itn_zh --certfile 0 --hotword /workspace/models/hotwords.txt && tail -f /dev/null"
      },
      "Diarization": {
        "Enabled": true,
        "AsrModel": "paraformer-zh",
        "VadModel": "fsmn-vad",
        "PunctuationModel": "ct-punc",
        "SpeakerModel": "cam++",
        "BatchSizeSeconds": 300,
        "BatchSizeThresholdSeconds": 60,
        "CommandTimeout": "00:30:00"
      }
    },
    "WhisperLocal": {
      "ModelPath": "models\\ggml-base.bin",
      "Language": "auto",
      "WindowSeconds": 15,
      "Diarization": {
        "Enabled": true,
        "DockerCommand": "docker",
        "BaseImage": "python:3.11-slim",
        "Image": "meeting-assistant-pyannote:local",
        "BuildImage": true,
        "ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Pyannote\\models",
        "Model": "pyannote/speaker-diarization-3.1",
        "AnnotationSource": "SpeakerDiarization",
        "AlignmentMode": "PyannoteTurns",
        "TokenEnv": "HF_TOKEN",
        "CommandTimeout": "00:30:00"
      }
    },
    "AzureSpeech": {
      "Endpoint": "https://germanywestcentral.api.cognitive.microsoft.com/",
      "Region": "germanywestcentral",
      "Language": "en-US",
      "KeyEnv": "AZURE_SPEECH_KEY",
      "RecognitionStopTimeout": "00:00:10",
      "DiarizeIntermediateResults": true
    },
    "Automation": {
      "RulesPath": "meeting-rules.local.yaml"
    },
    "Agent": {
      "Endpoint": "https://litellm.schweigert.cloud",
      "KeyEnv": "LITELLM_API_KEY",
      "Model": "copilot-gpt-5.5",
      "EnableThinking": true,
      "ReasoningEffort": "Medium",
      "ReconnectionAttempts": 2,
      "ReconnectionDelay": "00:00:02",
      "ContextWindowTokens": 128000,
      "MaxOutputTokens": 8192,
      "EnableCompaction": true,
      "CompactionRemainingRatio": 0.10,
      "ResponsesCompactPath": "responses/compact"
    },
    "Api": {
      "PublicBaseUrl": "http://localhost:5090"
    }
  }
}

funasr streams 16 kHz PCM audio to the configured FunASR WebSocket endpoint. When FunAsr:Backend:Enabled is true, Meeting Assistant manages a local Docker-backed FunASR runtime: on application startup it begins a non-blocking warm-up that pulls the configured online streaming image, prepares the mounted model and hotword folder, replaces any stale container with the configured name, starts the two-pass WebSocket server mapped to the endpoint port, keeps the container alive after the FunASR startup script backgrounds the server process, and stops the container when the app shuts down. Recording can start while the backend is still warming up; captured audio is queued and then streamed once the WebSocket backend is healthy. Docker commands are bounded by CommandTimeout, and backend WebSocket readiness is bounded by StartupTimeout. FunASR response fields such as spk_name, spk, and sentence-level speaker metadata are preserved in transcript segments; missing speaker fields are written as Unknown.

During recording, Meeting Assistant writes the mixed PCM stream to a temporary WAV under TemporaryRecordingsFolder only so the final diarization pass has audio to process. After capture stops and the streaming ASR provider drains the already-captured audio, FunASR Python AutoModel can run with VAD, punctuation, and CAMPPlus (spk_model="cam++") over that temporary WAV. If sentence-level speaker labels are returned, the transcript markdown is rewritten with the final speaker-attributed segments. If final diarization is disabled or returns no segments, the live streaming transcript is kept. Temporary WAV files are deleted after the run completes, and stale temporary recordings from interrupted runs are deleted when the application starts.

whisper-local uses Whisper.NET with a local ggml model and remains available by setting Recording:TranscriptionProvider to whisper-local. The model file is not committed to this repository; place it at the configured ModelPath before using live transcription. When WhisperLocal:Diarization:Enabled is true, the final post-processing pass runs pyannote in Docker over the temporary WAV and reads the Hugging Face token from WhisperLocal:Diarization:Token or TokenEnv (HF_TOKEN by default). AnnotationSource selects pyannote's speaker_diarization or exclusive_speaker_diarization output. AlignmentMode selects whether Meeting Assistant assigns the best-overlap pyannote speaker label to each Whisper segment or rebuilds the finished transcript as pyannote speaker-turn segments with matching Whisper text. When BuildImage is true, Meeting Assistant builds the configured local pyannote Docker image if Docker does not already have it, so Python packages are installed once instead of during every diarization call. The configured ModelsFolder is mounted as the persistent pyannote model cache and stores Hugging Face, torch, and pip cache artifacts so model downloads are reused across runs. If pyannote is disabled, has no token, or returns no turns, the live Whisper transcript is kept.

azure-speech streams captured PCM audio to Azure AI Speech using the Speech SDK ConversationTranscriber. Configure AzureSpeech:Endpoint, Region, Language, and either Key or KeyEnv; this repository uses AZURE_SPEECH_KEY by default and does not store the key in appsettings. DiarizeIntermediateResults enables diarization for intermediate and final conversation transcription results. Azure returns generic speaker IDs such as Guest-1 and Guest-2, which Meeting Assistant writes directly into live transcript segments. RecognitionStopTimeout bounds SDK shutdown after the captured audio stream has been closed, which keeps short diagnostic runs from waiting indefinitely for final service events. Since Azure Speech emits diarized live transcript segments, the Azure pipeline keeps those segments as the finished transcript instead of running pyannote.

ASR backends are exposed downstream as speech recognition pipelines. A pipeline can initialize, wait for readiness, accept audio chunks, emit live transcript segments, and produce a finished transcript after capture completes. The configured implementation is selected from Recording:TranscriptionProvider and registered through DI so recording and diagnostic endpoints do not need to know whether the backend uses FunASR, Whisper.NET, Azure Speech, pyannote, or a later provider.

POST /asr/transcribe-file is a local diagnostic endpoint for the configured ASR backend. Send { "path": "C:\\path\\to\\sample.wav" } with a 16-bit PCM WAV file to stream it through the configured speech recognition pipeline and return the emitted live transcript segments.

For real meeting recordings, Meeting Assistant derives the optional finished-transcript numSpeakers hint from the meeting note attendees frontmatter. The hint is set to the attendee count only when at least two attendees are listed; empty and single-person attendee lists are ignored. For pyannote-backed finalization, that hint is passed to pyannote as num_speakers.

POST /asr/diarize-file streams a local 16-bit PCM WAV through the configured speech recognition pipeline and returns the finished transcript. This is useful for checking the FunASR CAMPPlus path or the local Whisper plus pyannote path without starting a live recording. The request can include an optional speaker-count hint: { "path": "C:\\path\\to\\sample.wav", "numSpeakers": 5 }.

DictationWordsPath is reserved for providers that can accept vocabulary hints. It points to a markdown word list containing unusual project, person, product, or domain terms that should be biased during transcription when the provider supports it.

Meeting notes link to separate transcript, assistant context, and summary markdown artifacts using the configured vault folders.

Meeting note, transcript, assistant context, and summary artifacts use frontmatter links so they backlink to each other. Meeting note frontmatter includes start_time when recording starts and end_time when transcription processing finishes. Transcript and summary frontmatter copy the meeting title, start time, and end time from the meeting note; if the meeting has no title, the summary agent can provide one when writing the summary.

Assistant context notes keep their artifact links in frontmatter and expose a lifecycle state: collecting metadata, transcribing, speaker recognition, summarizing, finished, or error. Meeting Assistant updates this state as metadata lookup, recording, final speaker recognition, and summary generation progress.

ProjectsFolder contains project knowledge. Each direct subfolder is treated as one project, and a meeting binds projects by listing those subfolder names in the meeting note projects frontmatter.

Automation:RulesPath points to an optional local YAML rules file. The default meeting-rules.local.yaml is ignored by git. Rules can trigger on meeting creation, assistant-context state transitions, or identified speakers; conditions are evaluated with NCalc-style expressions and step values can use Razor syntax against Model.Meeting, Model.Event, and Model.Speaker. The initial supported steps are add_attendee, remove_attendee, set_property, add_context, and add_project.

See docs/meeting-workflow-engine.md for the detailed YAML format, supported variables, examples, and extension notes.

Agent configures the Microsoft Agent Framework summary pipeline. Key may be set directly for local testing, but KeyEnv is preferred; by default the API key is read from LITELLM_API_KEY. After transcription has fully finished, Meeting Assistant automatically runs the summary pipeline for the meeting. ReconnectionAttempts and ReconnectionDelay control retries for transient model endpoint failures such as LiteLLM token refresh or 5xx responses. If summary generation still fails, Meeting Assistant overwrites the configured summary note with a markdown failure report containing the exception details and linked meeting artifacts.

ContextWindowTokens, MaxOutputTokens, EnableCompaction, and CompactionRemainingRatio configure summary-agent context monitoring. Meeting Assistant estimates the outgoing Responses payload size, logs the estimated token count and remaining context, and compacts the conversation when only the configured remaining ratio is left. It first attempts POST /v1/{ResponsesCompactPath} on the configured OpenAI-compatible endpoint. If that endpoint is unavailable or returns invalid data, it falls back to Microsoft Agent Framework compaction: collapse old tool results, summarize older message groups, preserve only the last four user turns if needed, and finally drop oldest groups until the request is back under budget.

The summary endpoints are:

POST /meetings/current/summary/run
POST /meetings/summary/retry
GET /meetings/summary/retry?summaryPath=...

POST /meetings/summary/retry accepts { "summaryPath": "20260519-124110-summary.md" } or an absolute path under the configured summaries folder. It resolves the linked meeting note from frontmatter and reruns the summary pipeline, overwriting the existing summary file with either the new summary or a new failure report.

Failure summary notes include a clickable retry link using Api:PublicBaseUrl, for example http://localhost:5090/meetings/summary/retry?summaryPath=20260519-124110-summary.md. The GET endpoint exists for Obsidian/browser clicks and performs the same overwrite behavior as the POST endpoint. Meeting note bodies stay reserved for user-authored notes.

The summary agent currently has meeting-scoped tools to read the transcript, full meeting note including frontmatter, assistant context, user notes, glossary, and project files. Every read tool that returns file-like content can read the whole content or a clamped inclusive 1-based line range with from and to, so the agent can inspect large transcripts incrementally. The assistant context note is the agent's notebook and user-visible context window; the agent can write internal notes, discovered context, missing tool requests, suggested improvements, or follow-up task ideas there using line-based overwrite, replace, and insert modes. Project lookup and search tools are constrained to the projects listed in the meeting note frontmatter:

read_meetingnote
list_projects
list_projectfiles
read_projectfile
write_projectfile
search
write_context

search accepts ripgrep syntax and returns matches as filename:line text. If one project is searched, paths are relative to that project; if multiple projects are searched, paths include the project folder name.

write_projectfile writes inside an existing project folder. With no line arguments it overwrites or creates the file. With from and to it replaces that inclusive 1-based line range. With insert it inserts content at that 1-based line position.

Spec Workflow

This repository is OpenSpec-driven. Before changing behavior, read:

README.md
openspec/config.yaml
existing specs in openspec/specs
active change specs in openspec/changes/*/specs

The initial active change is:

openspec/changes/define-meeting-assistant-v1

Validate it with:

openspec validate define-meeting-assistant-v1 --strict