Manuel/meeting-assistant
Public Access
1
0
Fork 0
Code Issues Pull Requests 4 Actions Packages Projects Releases Wiki Activity Open in Coder

Implement meeting assistant v1

Browse Source
This commit is contained in:
codex
2026-05-20 02:06:16 +02:00
parent 90df1edc03
commit 0297bcc0f6
120 changed files with 11883 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+180 -3
View File
@@ -7,12 +7,17 @@ Its core purpose is to work with any kind of meeting, including in-person meetin
The application is intended to:
- create an Obsidian markdown note before transcription starts
- keep meeting metadata, user notes, detected context, and generated output in that note
- keep meeting metadata, user notes, detected context, and links to generated output in that note
- toggle recording/transcription mode through a configurable global hotkey
- capture microphone input and computer output into one transcription stream
- transcribe meetings with speaker attribution
- use a configurable speech recognition pipeline, with local Whisper as the first version 1 fallback
- use FunASR or local Whisper plus pyannote as configurable speaker-attribution paths
- write transcript markdown files into the configured vault folder
- generate summaries, decisions, and next steps
- build and maintain a project knowledge base from meetings and future context sources
- use Microsoft Agent Framework-based agents to reason over meeting and project context
- give agents tools for project lookup, keyword lookup, retrieval-augmented generation, and controlled project file updates when the user or project context allows it
- give agents tools for project lookup, keyword lookup, retrieval-augmented generation, and project file updates in existing projects
## Repository Status
@@ -34,12 +39,184 @@ Run the server:
dotnet run --project MeetingAssistant
```
The initial health endpoint is:
The initial endpoints are:
```text
GET /health
GET /recording/status
POST /recording/toggle
POST /recording/start
POST /recording/stop
POST /asr/transcribe-file
POST /asr/diarize-file
POST /meetings/current/summary/run
POST /meetings/summary/retry
GET /meetings/summary/retry?summaryPath=...
```
## Configuration
Meeting Assistant uses normal .NET configuration under the `MeetingAssistant` section:
```json
{
"MeetingAssistant": {
"Hotkey": {
"Toggle": "Ctrl+Alt+M"
},
"Vault": {
"TranscriptsFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Transcripts",
"MeetingNotesFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Notes",
"AssistantContextFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Assistant Context",
"SummariesFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\Summaries",
"ProjectsFolder": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Projects",
"DictationWordsPath": "%USERPROFILE%\\OpenCloud\\Persönlich\\Vault\\Meetings\\dictation-words.md"
},
"Recording": {
"TranscriptionProvider": "funasr",
"SampleRate": 16000,
"Channels": 1,
"StopProcessingTimeout": "00:10:00",
"TemporaryRecordingsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Recordings"
},
"FunAsr": {
"Endpoint": "ws://127.0.0.1:10095",
"Mode": "2pass",
"ChunkSize": [5, 10, 5],
"ChunkInterval": 10,
"EncoderChunkLookBack": 4,
"DecoderChunkLookBack": 1,
"Itn": true,
"EmitOnlinePartials": false,
"WavName": "meeting",
"FinalResultTimeout": "00:00:10",
"Backend": {
"Enabled": true,
"DockerCommand": "docker",
"Image": "registry.cn-hangzhou.aliyuncs.com/funasr_repo/funasr:funasr-runtime-sdk-online-cpu-0.1.13",
"ContainerName": "meeting-assistant-funasr",
"ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\FunASR\\models",
"ContainerPort": 10095,
"Privileged": true,
"CommandTimeout": "00:15:00",
"StartupTimeout": "00:10:00",
"StopOnDispose": true,
"ServerCommand": "cd /workspace/FunASR/runtime && bash run_server_2pass.sh --download-model-dir /workspace/models --vad-dir damo/speech_fsmn_vad_zh-cn-16k-common-onnx --model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-onnx --online-model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online-onnx --punc-dir damo/punc_ct-transformer_zh-cn-common-vad_realtime-vocab272727-onnx --itn-dir thuduj12/fst_itn_zh --certfile 0 --hotword /workspace/models/hotwords.txt && tail -f /dev/null"
},
"Diarization": {
"Enabled": true,
"AsrModel": "paraformer-zh",
"VadModel": "fsmn-vad",
"PunctuationModel": "ct-punc",
"SpeakerModel": "cam++",
"BatchSizeSeconds": 300,
"BatchSizeThresholdSeconds": 60,
"CommandTimeout": "00:30:00"
}
},
"WhisperLocal": {
"ModelPath": "models\\ggml-base.bin",
"Language": "auto",
"WindowSeconds": 15,
"Diarization": {
"Enabled": true,
"DockerCommand": "docker",
"BaseImage": "python:3.11-slim",
"Image": "meeting-assistant-pyannote:local",
"BuildImage": true,
"ModelsFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Pyannote\\models",
"Model": "pyannote/speaker-diarization-3.1",
"AnnotationSource": "SpeakerDiarization",
"AlignmentMode": "PyannoteTurns",
"TokenEnv": "HF_TOKEN",
"CommandTimeout": "00:30:00"
}
},
"AzureSpeech": {
"Endpoint": "https://germanywestcentral.api.cognitive.microsoft.com/",
"Region": "germanywestcentral",
"Language": "en-US",
"KeyEnv": "AZURE_SPEECH_KEY",
"RecognitionStopTimeout": "00:00:10",
"DiarizeIntermediateResults": true
},
"Agent": {
"Endpoint": "https://litellm.schweigert.cloud",
"KeyEnv": "LITELLM_API_KEY",
"Model": "copilot-gpt-5.5",
"EnableThinking": true,
"ReasoningEffort": "Medium",
"ReconnectionAttempts": 2,
"ReconnectionDelay": "00:00:02",
"ContextWindowTokens": 128000,
"MaxOutputTokens": 8192,
"EnableCompaction": true,
"CompactionRemainingRatio": 0.10,
"ResponsesCompactPath": "responses/compact"
},
"Api": {
"PublicBaseUrl": "http://localhost:5090"
}
}
}
```
`funasr` streams 16 kHz PCM audio to the configured FunASR WebSocket endpoint. When `FunAsr:Backend:Enabled` is true, Meeting Assistant manages a local Docker-backed FunASR runtime: on application startup it begins a non-blocking warm-up that pulls the configured online streaming image, prepares the mounted model and hotword folder, replaces any stale container with the configured name, starts the two-pass WebSocket server mapped to the endpoint port, keeps the container alive after the FunASR startup script backgrounds the server process, and stops the container when the app shuts down. Recording can start while the backend is still warming up; captured audio is queued and then streamed once the WebSocket backend is healthy. Docker commands are bounded by `CommandTimeout`, and backend WebSocket readiness is bounded by `StartupTimeout`. FunASR response fields such as `spk_name`, `spk`, and sentence-level speaker metadata are preserved in transcript segments; missing speaker fields are written as `Unknown`.
During recording, Meeting Assistant writes the mixed PCM stream to a temporary WAV under `TemporaryRecordingsFolder` only so the final diarization pass has audio to process. After capture stops and the streaming ASR provider drains the already-captured audio, FunASR Python `AutoModel` can run with VAD, punctuation, and CAMPPlus (`spk_model="cam++"`) over that temporary WAV. If sentence-level speaker labels are returned, the transcript markdown is rewritten with the final speaker-attributed segments. If final diarization is disabled or returns no segments, the live streaming transcript is kept. Temporary WAV files are deleted after the run completes, and stale temporary recordings from interrupted runs are deleted when the application starts.
`whisper-local` uses Whisper.NET with a local ggml model and remains available by setting `Recording:TranscriptionProvider` to `whisper-local`. The model file is not committed to this repository; place it at the configured `ModelPath` before using live transcription. When `WhisperLocal:Diarization:Enabled` is true, the final post-processing pass runs pyannote in Docker over the temporary WAV and reads the Hugging Face token from `WhisperLocal:Diarization:Token` or `TokenEnv` (`HF_TOKEN` by default). `AnnotationSource` selects pyannote's `speaker_diarization` or `exclusive_speaker_diarization` output. `AlignmentMode` selects whether Meeting Assistant assigns the best-overlap pyannote speaker label to each Whisper segment or rebuilds the finished transcript as pyannote speaker-turn segments with matching Whisper text. When `BuildImage` is true, Meeting Assistant builds the configured local pyannote Docker image if Docker does not already have it, so Python packages are installed once instead of during every diarization call. The configured `ModelsFolder` is mounted as the persistent pyannote model cache and stores Hugging Face, torch, and pip cache artifacts so model downloads are reused across runs. If pyannote is disabled, has no token, or returns no turns, the live Whisper transcript is kept.
`azure-speech` streams captured PCM audio to Azure AI Speech using the Speech SDK `ConversationTranscriber`. Configure `AzureSpeech:Endpoint`, `Region`, `Language`, and either `Key` or `KeyEnv`; this repository uses `AZURE_SPEECH_KEY` by default and does not store the key in appsettings. `DiarizeIntermediateResults` enables diarization for intermediate and final conversation transcription results. Azure returns generic speaker IDs such as `Guest-1` and `Guest-2`, which Meeting Assistant writes directly into live transcript segments. `RecognitionStopTimeout` bounds SDK shutdown after the captured audio stream has been closed, which keeps short diagnostic runs from waiting indefinitely for final service events. Since Azure Speech emits diarized live transcript segments, the Azure pipeline keeps those segments as the finished transcript instead of running pyannote.
ASR backends are exposed downstream as speech recognition pipelines. A pipeline can initialize, wait for readiness, accept audio chunks, emit live transcript segments, and produce a finished transcript after capture completes. The configured implementation is selected from `Recording:TranscriptionProvider` and registered through DI so recording and diagnostic endpoints do not need to know whether the backend uses FunASR, Whisper.NET, Azure Speech, pyannote, or a later provider.
`POST /asr/transcribe-file` is a local diagnostic endpoint for the configured ASR backend. Send `{ "path": "C:\\path\\to\\sample.wav" }` with a 16-bit PCM WAV file to stream it through the configured speech recognition pipeline and return the emitted live transcript segments.
For real meeting recordings, Meeting Assistant derives the optional finished-transcript `numSpeakers` hint from the meeting note `attendees` frontmatter. The hint is set to the attendee count only when at least two attendees are listed; empty and single-person attendee lists are ignored. For pyannote-backed finalization, that hint is passed to pyannote as `num_speakers`.
`POST /asr/diarize-file` streams a local 16-bit PCM WAV through the configured speech recognition pipeline and returns the finished transcript. This is useful for checking the FunASR CAMPPlus path or the local Whisper plus pyannote path without starting a live recording. The request can include an optional speaker-count hint: `{ "path": "C:\\path\\to\\sample.wav", "numSpeakers": 5 }`.
`DictationWordsPath` is reserved for providers that can accept vocabulary hints. It points to a markdown word list containing unusual project, person, product, or domain terms that should be biased during transcription when the provider supports it.
Meeting notes link to separate transcript, assistant context, and summary markdown artifacts using the configured vault folders.
Meeting note, transcript, assistant context, and summary artifacts use frontmatter links so they backlink to each other. Meeting note frontmatter includes `start_time` when recording starts and `end_time` when transcription processing finishes. Transcript and summary frontmatter copy the meeting title, start time, and end time from the meeting note; if the meeting has no title, the summary agent can provide one when writing the summary.
Assistant context notes keep their artifact links in frontmatter and expose a lifecycle `state`: `transcribing`, `speaker recognition`, `summarizing`, `finished`, or `error`. Meeting Assistant updates this state as recording, final speaker recognition, and summary generation progress.
`ProjectsFolder` contains project knowledge. Each direct subfolder is treated as one project, and a meeting binds projects by listing those subfolder names in the meeting note `projects` frontmatter.
`Agent` configures the Microsoft Agent Framework summary pipeline. `Key` may be set directly for local testing, but `KeyEnv` is preferred; by default the API key is read from `LITELLM_API_KEY`. After transcription has fully finished, Meeting Assistant automatically runs the summary pipeline for the meeting. `ReconnectionAttempts` and `ReconnectionDelay` control retries for transient model endpoint failures such as LiteLLM token refresh or 5xx responses. If summary generation still fails, Meeting Assistant overwrites the configured summary note with a markdown failure report containing the exception details and linked meeting artifacts.
`ContextWindowTokens`, `MaxOutputTokens`, `EnableCompaction`, and `CompactionRemainingRatio` configure summary-agent context monitoring. Meeting Assistant estimates the outgoing Responses payload size, logs the estimated token count and remaining context, and compacts the conversation when only the configured remaining ratio is left. It first attempts `POST /v1/{ResponsesCompactPath}` on the configured OpenAI-compatible endpoint. If that endpoint is unavailable or returns invalid data, it falls back to Microsoft Agent Framework compaction: collapse old tool results, summarize older message groups, preserve only the last four user turns if needed, and finally drop oldest groups until the request is back under budget.
The summary endpoints are:
```text
POST /meetings/current/summary/run
POST /meetings/summary/retry
GET /meetings/summary/retry?summaryPath=...
```
`POST /meetings/summary/retry` accepts `{ "summaryPath": "20260519-124110-summary.md" }` or an absolute path under the configured summaries folder. It resolves the linked meeting note from frontmatter and reruns the summary pipeline, overwriting the existing summary file with either the new summary or a new failure report.
Failure summary notes include a clickable retry link using `Api:PublicBaseUrl`, for example `http://localhost:5090/meetings/summary/retry?summaryPath=20260519-124110-summary.md`. The GET endpoint exists for Obsidian/browser clicks and performs the same overwrite behavior as the POST endpoint. Meeting note bodies stay reserved for user-authored notes.
The summary agent currently has meeting-scoped tools to read the transcript, full meeting note including frontmatter, assistant context, user notes, glossary, and project files. Every read tool that returns file-like content can read the whole content or a clamped inclusive 1-based line range with `from` and `to`, so the agent can inspect large transcripts incrementally. The assistant context note is the agent's notebook and user-visible context window; the agent can write internal notes, discovered context, missing tool requests, suggested improvements, or follow-up task ideas there using line-based overwrite, replace, and insert modes. Project lookup and search tools are constrained to the projects listed in the meeting note frontmatter:
- `read_meetingnote`
- `list_projects`
- `list_projectfiles`
- `read_projectfile`
- `write_projectfile`
- `search`
- `write_context`
`search` accepts ripgrep syntax and returns matches as `filename:line text`. If one project is searched, paths are relative to that project; if multiple projects are searched, paths include the project folder name.
`write_projectfile` writes inside an existing project folder. With no line arguments it overwrites or creates the file. With `from` and `to` it replaces that inclusive 1-based line range. With `insert` it inserts content at that 1-based line position.
## Spec Workflow
This repository is OpenSpec-driven. Before changing behavior, read: