Compare commits

...
Author SHA1 Message Date
dh 4e270ae0ea ci: accept QEMU paused prelaunch state and retire one-time cleanup
PR and Push Build/Test / portable-build-and-test (push) Canceled after 0s
PR and Push Build/Test / build-and-test (push) Canceled after 3m50s
2026-10-03 20:17:30 +02:00
dh abe907ce70 ci: replace inherited probe volume with read-only tmpfs
PR and Push Build/Test / portable-build-and-test (push) Canceled after 0s
PR and Push Build/Test / build-and-test (push) Canceled after 1m7s
2026-10-03 19:58:21 +02:00
dh 64d257468d ci: diagnose existing Docker-host KVM without provisioning
PR and Push Build/Test / portable-build-and-test (push) Canceled after 0s
PR and Push Build/Test / build-and-test (push) Canceled after 25s
2026-10-03 19:41:51 +02:00
dh 40281b57a5 Isolate measured UID watchdog failure in the native TCG diagnostic
PR and Push Build/Test / portable-build-and-test (push) Canceled after 0s
PR and Push Build/Test / build-and-test (push) Canceled after 1m33s
2026-10-03 18:58:10 +02:00
dh c92e62bdf5 Reduce native readiness probe overhead and preserve screenshot evidence
PR and Push Build/Test / portable-build-and-test (push) Canceled after 0s
PR and Push Build/Test / build-and-test (push) Canceled after 1m15s
2026-10-03 18:05:15 +02:00
dh b40234d3b5 ci: capture native recovery startup context before platform probe
PR and Push Build/Test / portable-build-and-test (push) Canceled after 0s
PR and Push Build/Test / build-and-test (push) Canceled after 3m24s
2026-10-03 16:38:34 +02:00
dh 0a30a1ca5a ci: retain current build fixes in the readonly native diagnostic 2026-10-03 16:22:03 +02:00
dh 1b19b08f2e build: use the tray core dependency without unused WinUI tooling
PR and Push Build/Test / build-and-test (push) Successful in 18m13s
PR and Push Build/Test / portable-build-and-test (push) Successful in 6m26s
2026-10-03 15:33:02 +02:00
dh 6b1896660f ci: derive the macOS probe from the pinned QEMU filesystem image 2026-10-03 15:01:00 +02:00
dh 05e23857dd ci: provision SDK code-signing roots in the disposable Wine prefix 2026-10-03 15:00:48 +02:00
dh 9a91a81992 ci: probe native macOS Recovery readiness on the existing Ubuntu runner 2026-10-03 14:18:50 +02:00
dh 83726a2233 ci: use Wine 11 for .NET 10 targeting-pack enumeration 2026-10-03 14:17:44 +02:00
dh 4840b8e3be ci: diagnose Windows SDK pack resolution on the actual runner 2026-10-03 13:51:49 +02:00
dh 445bff99ce docs: record native guest CI attempt and remaining verification gaps 2026-10-03 13:08:31 +02:00
dh 1164c26846 ci: validate macOS managed behavior on Ubuntu and stabilize shutdown 2026-10-03 11:30:29 +02:00
dh bd35ebc4c8 docs: record macOS main-sync review and real recording verification 2026-10-03 10:30:02 +02:00
dh fc0edd812d Merge latest main into macOS support and preserve portable integrations 2026-10-03 10:16:09 +02:00
codex 7b2bcd3631 Preserve recovery transcripts and add user-approved agent requests
Append successful offline recovery with a duplicate-warning marker and retain failed replay sources. Offer newly available microphones through confirmation notifications.

Add project discovery metadata and blocking association approval, plus nonblocking workflow requests that start a separate settings-agent conversation. Include regression tests, OpenSpec changes, and verified local Release startup.
2026-09-25 11:58:34 +02:00
dh bf6e1e7560 Merge remote-tracking branch 'refs/remotes/fork/codex/macos-support' into codex/macos-support
# Conflicts:
#	MeetingAssistant/Program.cs
#	README.md
2026-09-20 12:44:03 +02:00
codex 723298e720 Merge main into macOS support and repair integration fixtures 2026-09-18 10:56:40 +02:00
codex e011e36ef7 Merge pull request 'chore(deps): update dependency microsoft.agents.ai.openai to 1.21.0' (#53) from renovate/microsoft.agents.ai.openai-1.x into main 2026-09-18 09:59:28 +02:00
codex e17fb8b225 Merge pull request 'chore(deps): update microsoftspeechversion to 1.51.2' (#48) from renovate/microsoftspeechversion into main 2026-09-18 09:58:38 +02:00
codex c01213278e Merge pull request 'chore(deps): update dependency microsoft.entityframeworkcore.sqlite to 10.0.12' (#51) from renovate/microsoft.entityframeworkcore.sqlite-10.x into main 2026-09-18 09:57:58 +02:00
codex 7814f1c7c4 Merge pull request 'chore(deps): update dependency naudio to v3' (#45) from renovate/naudio-3.x into main 2026-09-18 09:57:23 +02:00
codex 88c3b21b2a Merge pull request 'chore(deps): update dependency naudio to 2.4.0' (#49) from renovate/naudio-2.x into main 2026-09-18 09:56:40 +02:00
codex b249e5d977 Merge branch 'renovate/microsoftspeechversion' into renovate/microsoft.agents.ai.openai-1.x 2026-09-17 13:36:23 +02:00
codex 3e4bf88931 Merge branch 'renovate/microsoft.entityframeworkcore.sqlite-10.x' into renovate/microsoftspeechversion 2026-09-17 13:36:22 +02:00
codex 5e7c9f64d0 Merge remote-tracking branch 'origin/renovate/naudio-3.x' into renovate/microsoft.entityframeworkcore.sqlite-10.x
# Conflicts:
#	MeetingAssistant/MeetingAssistant.csproj
2026-09-17 13:36:21 +02:00
codex fbd5a0852f Merge remote-tracking branch 'origin/renovate/naudio-2.x' into renovate/naudio-3.x
# Conflicts:
#	MeetingAssistant/MeetingAssistant.csproj
2026-09-17 13:34:55 +02:00
codex 568bedb77a fix: isolate Windows sample playback for NAudio 3 2026-09-17 13:34:47 +02:00
codex f21c56b1c7 Merge remote-tracking branch 'origin/main' into renovate/naudio-3.x
# Conflicts:
#	MeetingAssistant/MeetingAssistant.csproj
2026-09-17 13:30:47 +02:00
codex c56de2b7ab Merge remote-tracking branch 'origin/renovate/microsoftspeechversion' into renovate/microsoftspeechversion 2026-09-17 13:28:40 +02:00
codex 40d10d634a fix: reference WinMM explicitly for shared sample playback 2026-09-17 13:27:49 +02:00
codex 264a3a823f Merge remote-tracking branch 'origin/main' into renovate/naudio-2.x
# Conflicts:
#	MeetingAssistant/MeetingAssistant.csproj
2026-09-17 13:26:04 +02:00
codex 072ebf38a3 Merge remote-tracking branch 'origin/main' into renovate/microsoft.entityframeworkcore.sqlite-10.x 2026-09-17 13:24:45 +02:00
codex 4d66de943e fix: use Responses client options after Agent Framework update 2026-09-17 13:23:57 +02:00
codex 150f11b6a7 Merge remote-tracking branch 'origin/main' into renovate/microsoft.agents.ai.openai-1.x 2026-09-17 13:21:45 +02:00
codex d7db64bec7 Merge pull request 'chore(deps): update dependency microsoft.net.test.sdk to 18.10.1' (#56) from renovate/microsoft.net.test.sdk-18.x into main 2026-09-17 10:50:52 +02:00
codex cdeef1967e Merge pull request 'chore(deps): update dependency ncalcsync to 7.2.0' (#55) from renovate/ncalcsync-7.x into main 2026-09-17 10:49:02 +02:00
codex aeb0cfe42f Merge pull request 'chore(deps): update dependency system.drawing.common to 10.0.12' (#52) from renovate/system.drawing.common-10.x into main 2026-09-17 10:46:41 +02:00
renovate-bot ef0c0faba8 chore(deps): update dependency microsoft.net.test.sdk to 18.10.1 2026-09-17 02:45:15 +00:00
codex 93eb3d4e98 fix: prevent profile-switch transcription deadlock 2026-09-16 15:00:49 +02:00
dh d1d4b53ee8 ci: validate portable tests on macos 2026-09-16 08:30:56 +02:00
Manuel d93d758a8f Merge branch 'main' into renovate/naudio-3.x 2026-09-15 09:51:19 +02:00
Manuel 48ad4fb930 Merge branch 'main' into renovate/microsoftspeechversion 2026-09-15 09:51:06 +02:00
Manuel cb5a300b44 Merge branch 'main' into renovate/microsoft.entityframeworkcore.sqlite-10.x 2026-09-15 09:50:42 +02:00
Manuel b6acba8402 Merge branch 'main' into renovate/system.drawing.common-10.x 2026-09-15 09:50:31 +02:00
Manuel a82e2fff03 Merge branch 'main' into renovate/microsoft.agents.ai.openai-1.x 2026-09-15 09:50:21 +02:00
Manuel 336d940acd Merge branch 'main' into renovate/naudio-2.x 2026-09-15 09:50:03 +02:00
codex cbf854fb2a Merge pull request 'chore(deps): update dependency microsoft.net.test.sdk to 18.10.0' (#54) from renovate/microsoft.net.test.sdk-18.x into main 2026-09-14 08:55:43 +02:00
renovate-bot bacef060fe chore(deps): update dependency ncalcsync to 7.2.0 2026-09-14 02:41:00 +00:00
renovate-bot a928382e51 chore(deps): update dependency microsoft.net.test.sdk to 18.10.0 2026-09-13 02:40:52 +00:00
renovate-bot e23d04a38c chore(deps): update dependency naudio to v3 2026-09-12 02:46:02 +00:00
renovate-bot c6a50d307e chore(deps): update dependency naudio to 2.4.0 2026-09-12 02:45:52 +00:00
renovate-bot 436fa6c149 chore(deps): update dependency microsoft.agents.ai.openai to 1.21.0 2026-09-12 02:45:39 +00:00
renovate-bot 5dce5fd350 chore(deps): update microsoftspeechversion to 1.51.2 2026-09-12 02:45:33 +00:00
renovate-bot c37654a7d9 chore(deps): update dependency system.drawing.common to 10.0.12 2026-09-12 02:45:27 +00:00
renovate-bot cc9044e091 chore(deps): update dependency microsoft.entityframeworkcore.sqlite to 10.0.12 2026-09-12 02:45:18 +00:00
dh 0e2aa3830f fix(macos): launch service with stable privacy identity 2026-09-11 20:59:56 +02:00
dh d77187e9ec fix(macos): bundle audio capture for launchd privacy 2026-09-11 19:27:21 +02:00
dh 6adf6f726b Merge remote-tracking branch 'origin/main' into codex/macos-support
# Conflicts:
#	MeetingAssistant/Program.cs
#	README.md
#	docs/meeting-assistant-configuration.md
2026-09-11 19:16:53 +02:00
codex f86af983e8 feat: add local Resemblyzer speaker recognition 2026-09-11 13:49:27 +02:00
codex 43fc8aaec0 Merge pull request 'chore(deps): update dependency xunit.runner.visualstudio to v4' (#46) from renovate/xunit.runner.visualstudio-4.x into main 2026-09-10 10:36:56 +02:00
codex 8b543a2161 Merge pull request 'chore(deps): update dependency ncalcsync to 7.1.0' (#40) from renovate/ncalcsync-7.x into main 2026-09-10 10:36:48 +02:00
codex 58896073c0 Merge pull request 'chore(deps): update dependency microsoft.aspnetcore.mvc.testing to 10.0.12' (#50) from renovate/microsoft.aspnetcore.mvc.testing-10.x into main 2026-09-09 13:15:13 +02:00
renovate-bot 945543c078 chore(deps): update dependency microsoft.aspnetcore.mvc.testing to 10.0.12 2026-09-09 02:40:27 +00:00
renovate-bot 3e1f447b20 chore(deps): update dependency xunit.runner.visualstudio to v4 2026-09-04 02:42:29 +00:00
renovate-bot 497390c2ec chore(deps): update dependency ncalcsync to 7.1.0 2026-09-04 02:42:18 +00:00
codex b1365b202d fix: unify pyannote validation toggle 2026-09-02 13:46:51 +02:00
codex 70ffaa7357 feat: add transcription pause controls 2026-09-02 12:22:11 +02:00
dh 84dc696ce9 fix(macos): preserve audio when transcription fails 2026-09-01 11:55:15 +02:00
codex 714da456df Merge pull request 'chore(deps): update dependency microsoft.agents.ai.openai to 1.20.0' (#47) from renovate/microsoft.agents.ai.openai-1.x into main 2026-09-01 09:32:08 +02:00
renovate-bot fa67f30c49 chore(deps): update dependency microsoft.agents.ai.openai to 1.20.0 2026-09-01 02:50:34 +00:00
dh 68db6955a2 Merge remote-tracking branch 'origin/main' into codex/macos-support
# Conflicts:
#	README.md
2026-08-31 18:49:49 +02:00
codex dc37719d46 Merge pull request 'chore(deps): update dependency microsoft.aspnetcore.mvc.testing to 10.0.11' (#41) from renovate/microsoft.aspnetcore.mvc.testing-10.x into main 2026-08-18 13:13:40 +02:00
renovate-bot ee453e170e chore(deps): update dependency microsoft.aspnetcore.mvc.testing to 10.0.11 2026-08-18 02:40:54 +00:00
codex d51463321b Merge pull request 'chore(deps): update dependency microsoft.net.test.sdk to 18.9.0' (#44) from renovate/microsoft.net.test.sdk-18.x into main 2026-08-17 09:05:30 +02:00
renovate-bot 41ae47fbfe chore(deps): update dependency microsoft.net.test.sdk to 18.9.0 2026-08-16 02:49:14 +00:00
codex f7ad2ce41a Merge pull request 'chore(deps): update dependency system.drawing.common to 10.0.11' (#43) from renovate/system.drawing.common-10.x into main 2026-08-13 10:53:04 +02:00
renovate-bot bc76151926 chore(deps): update dependency system.drawing.common to 10.0.11 2026-08-13 02:41:36 +00:00
codex af9f5b34dc Merge pull request 'chore(deps): update dependency microsoft.entityframeworkcore.sqlite to 10.0.11' (#42) from renovate/microsoft.entityframeworkcore.sqlite-10.x into main 2026-08-12 13:19:26 +02:00
renovate-bot f3489f8e95 chore(deps): update dependency microsoft.entityframeworkcore.sqlite to 10.0.11 2026-08-12 02:49:08 +00:00
codex 39591eef12 docs: refresh Meeting Assistant runtime boundaries 2026-08-11 09:15:16 +02:00
148 changed files with 11940 additions and 735 deletions
@@ -0,0 +1,36 @@
name: Existing Docker-host KVM diagnostic
on:
workflow_dispatch:
jobs:
existing-kvm-diagnostic:
runs-on: ubuntu-latest
timeout-minutes: 5
env:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "1"
DOTNET_NOLOGO: "1"
steps:
- name: Checkout diagnostic source
uses: actions/checkout@v7
- name: Setup .NET for the diagnostic helper
uses: actions/setup-dotnet@v6
with:
dotnet-version: "10.0.x"
- name: Test only the existing Docker-host KVM device
run: dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --run --output artifacts/existing-kvm
- name: Always retry cleanup of this diagnostic's owned container
if: always()
run: dotnet run --file tools/ci/ExistingKvmDiagnostic.cs -- --cleanup --output artifacts/existing-kvm
- name: Preserve KVM diagnostic evidence
if: always()
uses: actions/upload-artifact@v3
with:
name: existing-docker-host-kvm-diagnostic
path: artifacts/existing-kvm/
if-no-files-found: error
retention-days: 7
@@ -0,0 +1,36 @@
name: Native macOS Recovery diagnostic on Ubuntu
on:
workflow_dispatch:
jobs:
macos-native-diagnostic:
runs-on: ubuntu-latest
timeout-minutes: 45
env:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "1"
DOTNET_NOLOGO: "1"
steps:
- name: Checkout diagnostic source
uses: actions/checkout@v7
- name: Setup .NET for the diagnostic helper
uses: actions/setup-dotnet@v6
with:
dotnet-version: "10.0.x"
- name: Probe native macOS Recovery with existing Docker resources
run: dotnet run --file tools/ci/MacOsNativeDiagnostic.cs -- --run --output artifacts/native-macos
- name: Always clean up only this diagnostic's owned resources
if: always()
run: dotnet run --file tools/ci/MacOsNativeDiagnostic.cs -- --cleanup --output artifacts/native-macos
- name: Preserve native diagnostic evidence
if: always()
uses: actions/upload-artifact@v3
with:
name: native-macos-recovery-diagnostic
path: artifacts/native-macos/
if-no-files-found: error
retention-days: 7
+110 -20
View File
@@ -24,37 +24,33 @@ jobs:
- name: Install Wine
run: |
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install -y --no-install-recommends wine64 wine32 winbind unzip
if [ -d /usr/lib/wine ]; then
echo "/usr/lib/wine" >> "${GITHUB_PATH}"
export PATH="${PATH}:/usr/lib/wine"
fi
if command -v wine >/dev/null 2>&1; then
WINE_BIN="$(command -v wine)"
elif command -v wine64 >/dev/null 2>&1; then
WINE_BIN="$(command -v wine64)"
elif [ -x /usr/lib/wine/wine64 ]; then
WINE_BIN="/usr/lib/wine/wine64"
elif [ -x /usr/lib/wine/wine ]; then
WINE_BIN="/usr/lib/wine/wine"
else
echo "No wine binary found after installation."
ls -la /usr/lib/wine || true
exit 1
fi
sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg winbind unzip
sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://dl.winehq.org/wine-builds/winehq.key \
| gpg --dearmor \
| sudo tee /etc/apt/keyrings/winehq-archive.key >/dev/null
. /etc/os-release
curl -fsSL "https://dl.winehq.org/wine-builds/ubuntu/dists/${VERSION_CODENAME}/winehq-${VERSION_CODENAME}.sources" \
| sudo tee /etc/apt/sources.list.d/winehq.sources >/dev/null
sudo apt-get update
# Wine 9 cannot enumerate the DOS_DOT patterns used by .NET 10's SDK pack lookup.
sudo apt-get install -y --no-install-recommends "winehq-stable=11.0.0.0~${VERSION_CODENAME}-1"
WINE_BIN="/opt/wine-stable/bin/wine"
test -x "${WINE_BIN}"
echo "WINE_BIN=${WINE_BIN}" >> "${GITHUB_ENV}"
"${WINE_BIN}" --version
- name: Restore (Windows target)
- name: Restore (portable tests for Windows/Wine)
run: |
dotnet restore MeetingAssistant.Tests/MeetingAssistant.Tests.csproj \
-p:EnableWindowsTargeting=true \
-r win-x64
- name: Build (Windows target)
- name: Build (portable tests for Windows/Wine)
run: |
dotnet build MeetingAssistant.Tests/MeetingAssistant.Tests.csproj \
-c Release \
@@ -77,16 +73,110 @@ jobs:
echo "WIN_DOTNET_DIR=${WIN_DOTNET_DIR}" >> "${GITHUB_ENV}"
"${WINE_BIN}" "${WIN_DOTNET_DIR}/dotnet.exe" --info
- name: Provision NuGet signature trust inside Wine
run: |
SDK_VERSION="$(dotnet --version)"
SDK_TRUST_ROOT="${DOTNET_ROOT}/sdk/${SDK_VERSION}/trustedroots"
dotnet publish scripts/wine-sdk-trust.cs \
-c Release \
-p:UseAppHost=false \
-p:PublishAot=false \
-o "${RUNNER_TEMP}/wine-sdk-trust" \
--nologo
"${WINE_BIN}" "${WIN_DOTNET_DIR}/dotnet.exe" \
"Z:${RUNNER_TEMP}/wine-sdk-trust/WineSdkTrust.dll" --import \
"Z:${SDK_TRUST_ROOT}/codesignctl.pem" \
"Z:${SDK_TRUST_ROOT}/timestampctl.pem"
- name: Diagnose Windows SDK targeting-pack resolution
run: |
find "${WIN_DOTNET_DIR}/packs" -maxdepth 4 -name PackageOverrides.txt -print
"${WINE_BIN}" "${WIN_DOTNET_DIR}/dotnet.exe" msbuild MeetingAssistant/MeetingAssistant.csproj \
-p:EnableWindowsTargeting=true \
-p:TargetFramework=net10.0 \
-getProperty:NetCoreRoot,NetCoreTargetingPackRoot,PrunePackageDataRoot,PrunePackageTargetingPackRoots,MSBuildSDKsPath,DOTNET_MSBUILD_SDK_RESOLVER_CLI_DIR
"${WINE_BIN}" "${WIN_DOTNET_DIR}/dotnet.exe" msbuild MeetingAssistant/MeetingAssistant.csproj \
-p:EnableWindowsTargeting=true \
-p:TargetFramework=net10.0 \
-t:AddPrunePackageReferences \
-v:normal
- name: Build Windows desktop target via Wine
run: |
rm -f MeetingAssistant/bin/Release/net10.0-windows10.0.19041.0/win-x64/MeetingAssistant.dll
"${WINE_BIN}" "${WIN_DOTNET_DIR}/dotnet.exe" build MeetingAssistant/MeetingAssistant.csproj \
-c Release \
-f net10.0-windows10.0.19041.0 \
-r win-x64 \
-p:EnableWindowsTargeting=true \
--nologo
test -s MeetingAssistant/bin/Release/net10.0-windows10.0.19041.0/win-x64/MeetingAssistant.dll
- name: Run tests via Wine (Windows dotnet host)
run: |
rm -f artifacts/tests/wine.trx
"${WINE_BIN}" "${WIN_DOTNET_DIR}/dotnet.exe" test MeetingAssistant.Tests/MeetingAssistant.Tests.csproj \
-c Release \
-r win-x64 \
--no-build \
--logger "trx;LogFileName=wine.trx" \
--results-directory artifacts/tests \
--nologo
test -s artifacts/tests/wine.trx
- name: Show test output folder on failure
if: failure()
run: |
find artifacts/tests -type f -print || true
find MeetingAssistant.Tests -type d -name TestResults -print || true
find MeetingAssistant.Tests -type f -path "*/TestResults/*" -maxdepth 5 -print || true
portable-build-and-test:
runs-on: ubuntu-latest
env:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "1"
DOTNET_NOLOGO: "1"
TZ: "Europe/Berlin"
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Setup .NET
uses: actions/setup-dotnet@v6
with:
dotnet-version: "10.0.x"
- name: Restore (portable target)
run: |
dotnet restore MeetingAssistant.Tests/MeetingAssistant.Tests.csproj \
-p:EnableWindowsTargeting=true
- name: Build (portable target including macOS managed adapters)
run: |
dotnet build MeetingAssistant.Tests/MeetingAssistant.Tests.csproj \
-c Release \
-f net10.0 \
--no-restore \
-p:EnableWindowsTargeting=true \
--nologo
- name: Run portable and macOS managed behavior tests on Ubuntu
run: |
rm -f artifacts/tests/portable.trx
dotnet test MeetingAssistant.Tests/MeetingAssistant.Tests.csproj \
-c Release \
-f net10.0 \
--no-build \
-p:EnableWindowsTargeting=true \
--logger "trx;LogFileName=portable.trx" \
--results-directory artifacts/tests \
--nologo
test -s artifacts/tests/portable.trx
- name: Show test output folder on failure
if: failure()
run: |
find artifacts/tests -type f -print || true
find MeetingAssistant.Tests -type d -name TestResults -print || true
find MeetingAssistant.Tests -type f -path "*/TestResults/*" -maxdepth 5 -print || true
@@ -110,6 +110,11 @@ public sealed class AzureSpeechStreamingTranscriptionProviderTests
[Fact]
public void ResolveKeyFallsBackToUserEnvironment()
{
if (!OperatingSystem.IsWindows())
{
return;
}
var keyEnv = $"MEETING_ASSISTANT_AZURE_KEY_{Guid.NewGuid():N}";
try
{
@@ -1,6 +1,7 @@
using System.Net;
using System.Net.Http.Json;
using MeetingAssistant.Recording;
using MeetingAssistant.Workflow;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
@@ -54,6 +55,17 @@ public sealed class HealthEndpointTests : IClassFixture<WebApplicationFactory<Pr
Assert.Equal(HostOperatingSystem.DetectCurrent(), first);
}
[Fact]
public async Task NeutralBuildReportsSpeakerPlaybackUnavailable()
{
var playback = factory.Services.GetRequiredService<IWorkflowRulesEditorSamplePlaybackQueue>();
var result = await playback.QueueAsync(1, [1, 2]);
Assert.Contains("Windows build", result);
Assert.DoesNotContain("Queued", result);
}
[Fact]
public async Task ApplicationStartupDeletesStaleTemporaryRecordings()
{
@@ -58,6 +58,64 @@ public sealed class LaunchProfileOptionsProviderTests
Assert.Equal("Ctrl+Alt+L", profile.Options.Hotkey.Toggle);
}
[Fact]
public void CheckedInConfigurationKeepsResemblyzerOptInWithDocumentedDefaults()
{
var profile = CreateProviderFromAppsettings().GetRequiredProfile(null);
Assert.False(profile.Options.SpeakerIdentification.Resemblyzer.Enabled);
Assert.Equal(5, profile.Options.SpeakerIdentification.Resemblyzer.RequiredVectorsPerSpeaker);
Assert.Equal(1000, profile.Options.SpeakerIdentification.Resemblyzer.MaxVectorsPerIdentity);
Assert.Equal(20, profile.Options.SpeakerIdentification.Resemblyzer.OutlierPruningMinimumVectors);
Assert.Equal(0.75, profile.Options.SpeakerIdentification.Resemblyzer.OutlierPruningNeighborSimilarity);
Assert.Equal(3, profile.Options.SpeakerIdentification.Resemblyzer.OutlierPruningMinimumNeighbors);
Assert.Equal(0.60, profile.Options.SpeakerIdentification.Resemblyzer.OutlierPruningMinimumClusterRatio);
}
[Fact]
public void InvalidResemblyzerOutlierPruningSettingsAreRejected()
{
var provider = CreateProvider(new Dictionary<string, string?>
{
["MeetingAssistant:SpeakerIdentification:Resemblyzer:OutlierPruningMinimumClusterRatio"] = "1.1"
});
var exception = Assert.Throws<InvalidOperationException>(
() => provider.GetRequiredProfile(null));
Assert.Contains("OutlierPruningMinimumClusterRatio", exception.Message);
}
[Fact]
public void NonFiniteResemblyzerSimilaritySettingsAreRejected()
{
var provider = CreateProvider(new Dictionary<string, string?>
{
["MeetingAssistant:SpeakerIdentification:Resemblyzer:MinimumIdentitySimilarity"] = "NaN"
});
var exception = Assert.Throws<InvalidOperationException>(
() => provider.GetRequiredProfile(null));
Assert.Contains("MinimumIdentitySimilarity", exception.Message);
}
[Fact]
public void ContradictorySpeakerSampleDurationsAreRejected()
{
var provider = CreateProvider(new Dictionary<string, string?>
{
["MeetingAssistant:SpeakerIdentification:MinimumSampleSpeechDuration"] = "00:01:01",
["MeetingAssistant:SpeakerIdentification:MaximumSampleDuration"] = "00:01:00"
});
var exception = Assert.Throws<InvalidOperationException>(
() => provider.GetRequiredProfile(null));
Assert.Contains("MinimumSampleSpeechDuration", exception.Message);
Assert.Contains("MaximumSampleDuration", exception.Message);
}
[Fact]
public void DuplicateProfileHotkeysAreRejected()
{
@@ -1,21 +1,36 @@
using MeetingAssistant.LaunchProfiles;
using MeetingAssistant.MacOs;
using MeetingAssistant.Workflow;
using MeetingAssistant.Notifications;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Tests;
public sealed class MacOsDesktopControlManifestTests
{
[Fact]
public void AutomaticWorkflowChangeCannotSilentlyLoseItsPromptOnMacOs()
{
IWorkflowRulesEditorWindowService service = new MacOsWorkflowRulesEditorWindowService(
Options.Create(new MeetingAssistantOptions()));
Assert.Throws<PlatformNotSupportedException>(() => service.ShowWithPrompt("Update meeting rules"));
}
[Fact]
public async Task MacOsApplicationDeclinesUnavailableAutomaticApprovals()
{
using var factory = new WebApplicationFactory<Program>();
var approval = factory.Services.GetRequiredService<AgentApprovalService>();
Assert.False(await approval.ConfirmAsync("Test approval", "Do not apply a change", CancellationToken.None));
}
[Fact]
public void MacOsApplicationRegistersARealInteractiveAgentWindow()
{
if (!OperatingSystem.IsMacOS())
{
return;
}
using var factory = new WebApplicationFactory<Program>();
var service = factory.Services.GetRequiredService<IWorkflowRulesEditorWindowService>();
@@ -0,0 +1,12 @@
namespace MeetingAssistant.Tests;
public sealed class MacOsFactAttribute : FactAttribute
{
public MacOsFactAttribute()
{
if (!OperatingSystem.IsMacOS())
{
Skip = "Requires native macOS helpers and Apple system tools; run on macOS.";
}
}
}
@@ -5,9 +5,11 @@ using MeetingAssistant.Transcription;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using System.Net.Http.Json;
@@ -15,6 +17,83 @@ namespace MeetingAssistant.Tests;
public sealed class MacOsMeetingAudioSourceTests
{
[Fact]
public void MacOsServiceLauncherDefinesStablePrivacyIdentity()
{
var repositoryDirectory = new DirectoryInfo(AppContext.BaseDirectory);
while (repositoryDirectory is not null &&
!File.Exists(Path.Combine(repositoryDirectory.FullName, "MeetingAssistant", "MeetingAssistant.csproj")))
{
repositoryDirectory = repositoryDirectory.Parent;
}
Assert.NotNull(repositoryDirectory);
var projectDirectory = Path.Combine(repositoryDirectory.FullName, "MeetingAssistant");
var launcherDirectory = Path.Combine(projectDirectory, "Native", "MacOsLauncher");
var infoPlistPath = Path.Combine(launcherDirectory, "Info.plist");
var launcherSourcePath = Path.Combine(launcherDirectory, "main.swift");
Assert.True(File.Exists(infoPlistPath), $"Expected launcher Info.plist at '{infoPlistPath}'.");
Assert.True(File.Exists(launcherSourcePath), $"Expected launcher source at '{launcherSourcePath}'.");
var infoPlist = File.ReadAllText(infoPlistPath);
Assert.Contains("cloud.schweigert.meeting-assistant", infoPlist, StringComparison.Ordinal);
Assert.Contains("NSMicrophoneUsageDescription", infoPlist, StringComparison.Ordinal);
Assert.Contains("NSCalendarsFullAccessUsageDescription", infoPlist, StringComparison.Ordinal);
var launcherSource = File.ReadAllText(launcherSourcePath);
Assert.Contains("set -a", launcherSource, StringComparison.Ordinal);
Assert.Contains("Contents/Resources/app", launcherSource, StringComparison.Ordinal);
var projectFile = File.ReadAllText(Path.Combine(projectDirectory, "MeetingAssistant.csproj"));
Assert.Contains(
"designated =&gt; identifier &quot;cloud.schweigert.meeting-assistant&quot;",
projectFile,
StringComparison.Ordinal);
}
[MacOsFact]
public void NativeAudioCaptureIsPackagedAsSignedMacOsAppForPersistentPrivacyGrant()
{
var configuration = new DirectoryInfo(AppContext.BaseDirectory)
.Parent?.Name ?? "Debug";
var appPath = Path.GetFullPath(Path.Combine(
AppContext.BaseDirectory,
"..",
"..",
"..",
"..",
"MeetingAssistant",
"bin",
configuration,
"net10.0",
"Native",
"MeetingAssistantAudioCapture.app"));
var infoPlistPath = Path.Combine(appPath, "Contents", "Info.plist");
var helperPath = Path.Combine(appPath, "Contents", "MacOS", "macos-meeting-audio-capture");
Assert.True(Directory.Exists(appPath), $"Expected macOS audio capture app at '{appPath}'.");
Assert.True(File.Exists(infoPlistPath), $"Expected Info.plist at '{infoPlistPath}'.");
Assert.True(File.Exists(helperPath), $"Expected native helper at '{helperPath}'.");
var infoPlist = File.ReadAllText(infoPlistPath);
Assert.Contains("cloud.schweigert.meeting-assistant.audio-capture", infoPlist, StringComparison.Ordinal);
Assert.Contains("NSMicrophoneUsageDescription", infoPlist, StringComparison.Ordinal);
using var verification = System.Diagnostics.Process.Start(new System.Diagnostics.ProcessStartInfo
{
FileName = "/usr/bin/codesign",
ArgumentList = { "--verify", "--deep", "--strict", appPath },
RedirectStandardError = true,
UseShellExecute = false
});
Assert.NotNull(verification);
verification.WaitForExit();
Assert.True(
verification.ExitCode == 0,
$"Expected a valid app-bundle signature: {verification.StandardError.ReadToEnd()}");
}
[Fact]
public async Task MacOsMicrophoneStreamsNativePcmUsingRunFormat()
{
@@ -85,7 +164,7 @@ public sealed class MacOsMeetingAudioSourceTests
microphonePcm: Pcm16(2_000),
systemPcm: Pcm16(10_000));
var speechPipelines = new CapturingSpeechRecognitionPipelineFactory();
await using var factory = new WebApplicationFactory<Program>().WithWebHostBuilder(builder =>
var factory = new WebApplicationFactory<Program>().WithWebHostBuilder(builder =>
{
builder.ConfigureAppConfiguration((_, configuration) =>
{
@@ -105,6 +184,10 @@ public sealed class MacOsMeetingAudioSourceTests
});
builder.ConfigureTestServices(services =>
{
// The endpoint fixture supplies its own transcription pipeline and does not warm external pyannote models.
services.Remove(services.Single(descriptor =>
descriptor.ServiceType == typeof(IHostedService) &&
descriptor.ImplementationType == typeof(PyannoteDiarizationWarmupHostedService)));
services.RemoveAll<IMeetingAudioSource>();
services.RemoveAll<IMacOsAudioCaptureProcessFactory>();
PortableMeetingAudioRegistration.Add(services, isMacOs: true);
@@ -140,6 +223,9 @@ public sealed class MacOsMeetingAudioSourceTests
}
finally
{
await factory.DisposeAsync();
using var connection = new SqliteConnection($"Data Source={Path.Combine(testRoot, "speakers.db")}");
SqliteConnection.ClearPool(connection);
if (Directory.Exists(testRoot))
{
Directory.Delete(testRoot, recursive: true);
@@ -133,14 +133,9 @@ public sealed class MacOsMeetingIntegrationTests
Assert.Equal("meeting.png.cropped.png", croppedPath);
}
[Fact]
[MacOsFact]
public async Task MacOsCapabilityEndpointReportsEnabledRealProviders()
{
if (!OperatingSystem.IsMacOS())
{
return;
}
await using var factory = new WebApplicationFactory<Program>();
using var client = factory.CreateClient();
@@ -163,14 +158,9 @@ public sealed class MacOsMeetingIntegrationTests
Assert.True(report.WhisperLocalDiarizationEnabled);
}
[Fact]
[MacOsFact]
public async Task NativeHelperAdvertisesCalendarPromptAndScreenshotFeatures()
{
if (!OperatingSystem.IsMacOS())
{
return;
}
var runner = new ProcessCommandRunner(
NullLogger<ProcessCommandRunner>.Instance);
@@ -190,14 +180,9 @@ public sealed class MacOsMeetingIntegrationTests
.ToArray());
}
[Fact]
[MacOsFact]
public async Task NativeHelperCropsPngUsingOcrPixelCoordinates()
{
if (!OperatingSystem.IsMacOS())
{
return;
}
var testRoot = Path.Combine(
Path.GetTempPath(),
$"meeting-assistant-native-crop-{Guid.NewGuid():N}");
@@ -268,14 +253,9 @@ public sealed class MacOsMeetingIntegrationTests
Assert.Equal(TimeSpan.FromHours(23), client.CalendarTo - client.CalendarFrom);
}
[Fact]
[MacOsFact]
public async Task CalendarClientFallsBackToCalendarAutomationWhenEventKitIsDenied()
{
if (!OperatingSystem.IsMacOS())
{
return;
}
var runner = new SequenceCommandRunner(
new CommandResult(
MacOsIntegrationException.CalendarPermissionDeniedExitCode,
@@ -9,10 +9,10 @@
<ItemGroup>
<PackageReference Include="coverlet.collector" Version="10.0.1" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.10" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.8.1" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.12" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.10.1" />
<PackageReference Include="xunit" Version="2.9.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.5" />
<PackageReference Include="xunit.runner.visualstudio" Version="4.0.0" />
</ItemGroup>
<ItemGroup>
@@ -206,7 +206,7 @@ public sealed class MeetingNoteStoreTests
{
var link = MeetingNoteActionLinks.CreateSummaryRetryLink(
"http://localhost:5090/",
"C:\\Vault\\Meetings\\Summaries\\summary with spaces.md");
Path.Combine(Path.GetTempPath(), "summary with spaces.md"));
Assert.Equal(
"[Retry summary generation](http://localhost:5090/meetings/summary/retry?summaryPath=summary%20with%20spaces.md)",
@@ -7,6 +7,37 @@ namespace MeetingAssistant.Tests;
public sealed class MeetingSummaryInstructionBuilderTests
{
[Fact]
public async Task BuilderAddsBoundMetadataWithoutAgentsAndRequestGuidanceToCustomPrompt()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var projectsRoot = Path.Combine(root, "Projects");
var artifacts = CreateArtifacts(root);
await WriteMeetingNoteAsync(artifacts.MeetingNotePath, ["Alpha"]);
Directory.CreateDirectory(Path.Combine(projectsRoot, "Alpha"));
await File.WriteAllTextAsync(Path.Combine(projectsRoot, "Alpha", "PROJECT.md"),
"---\nname: Alpha Platform\ndescription: Builds meeting automation\n---\nInternal notes");
await WriteProjectAgentsAsync(projectsRoot, "Unbound", "Do not expose these instructions");
var builder = new MeetingSummaryInstructionBuilder(Options.Create(new MeetingAssistantOptions
{
Agent = new AgentOptions { InitialPrompt = "Custom base." },
Vault = new VaultOptions { ProjectsFolder = projectsRoot }
}));
var instructions = await builder.BuildAsync(artifacts, CancellationToken.None);
Assert.StartsWith("Custom base.", instructions);
Assert.Contains("# Alpha", instructions);
Assert.Contains("Alpha Platform", instructions);
Assert.Contains("Builds meeting automation", instructions);
Assert.Contains("PROJECT.md", instructions);
Assert.Contains("256", instructions);
Assert.Contains("request_project_association", instructions);
Assert.Contains("request_workflow_change", instructions);
Assert.DoesNotContain("Internal notes", instructions);
Assert.DoesNotContain("Do not expose", instructions);
}
[Fact]
public async Task BuilderUsesDefaultPromptWhenConfiguredPromptIsBlank()
{
@@ -75,7 +106,8 @@ public sealed class MeetingSummaryInstructionBuilderTests
var instructions = await builder.BuildAsync(artifacts, CancellationToken.None);
Assert.Equal("Custom summarizer instructions.", instructions);
Assert.StartsWith("Custom summarizer instructions.", instructions);
Assert.Contains("request_project_association", instructions);
}
[Fact]
@@ -96,21 +128,12 @@ public sealed class MeetingSummaryInstructionBuilderTests
var instructions = await builder.BuildAsync(artifacts, CancellationToken.None);
var expected = """
Base.
---
projects:
# Alpha
Alpha instructions.
# Beta
Beta instructions.
""";
Assert.Equal(NormalizeLineEndings(expected), NormalizeLineEndings(instructions));
Assert.StartsWith("Base.", instructions);
Assert.Contains("---\nprojects:", NormalizeLineEndings(instructions));
Assert.Contains("# Alpha", instructions);
Assert.Contains("Alpha instructions.", instructions);
Assert.Contains("# Beta", instructions);
Assert.Contains("Beta instructions.", instructions);
}
private static MeetingSessionArtifacts CreateArtifacts(string root)
@@ -50,6 +50,145 @@ public sealed class MicrophoneAudioSourceTests
await cancellation.CancelAsync();
}
[Fact]
public async Task AcceptedNewMicrophoneReplacesCaptureWithinTheSameStream()
{
var selection = new MicrophoneDeviceSelection();
var devices = new SelectableCaptureDevices(selection);
var prompts = new AcceptingPrompt();
var monitor = new MicrophoneSwitchMonitor(devices, prompts,
NullLogger<MicrophoneSwitchMonitor>.Instance, TimeSpan.FromMilliseconds(10), TimeSpan.FromMinutes(1));
var source = new MicrophoneAudioSource(devices, NullLogger<MicrophoneAudioSource>.Instance,
TimeSpan.Zero, monitor, selection);
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await using var chunks = source.CaptureAsync(new MeetingAssistantOptions(), lifetime.Token).GetAsyncEnumerator();
try
{
Assert.True(await chunks.MoveNextAsync());
Assert.Equal(1_000, BitConverter.ToInt16(chunks.Current.Pcm));
devices.Available = [new("old", "Built-in"), new("new", "USB headset")];
Assert.Equal("new", (await prompts.Shown.Task.WaitAsync(TimeSpan.FromSeconds(2))).Id);
var nextChunk = chunks.MoveNextAsync().AsTask();
Assert.False(nextChunk.IsCompleted);
Assert.Null(selection.SelectedDeviceId);
prompts.Response.SetResult(true);
Assert.True(await nextChunk.WaitAsync(TimeSpan.FromSeconds(2)));
Assert.Equal(2_000, BitConverter.ToInt16(chunks.Current.Pcm));
Assert.Equal("new", selection.SelectedDeviceId);
Assert.False(lifetime.IsCancellationRequested);
}
finally
{
await lifetime.CancelAsync();
}
}
[Fact]
public async Task EndingMeetingDuringCreationDisposesAbandonedCapture()
{
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(5));
var abandonedCapture = new DisposableAudioSource();
var factory = new CallbackCaptureFactory(() =>
{
lifetime.Cancel();
return abandonedCapture;
});
var source = new MicrophoneAudioSource(factory, NullLogger<MicrophoneAudioSource>.Instance);
await using var chunks = source.CaptureAsync(lifetime.Token).GetAsyncEnumerator();
Assert.False(await chunks.MoveNextAsync());
Assert.True(abandonedCapture.Disposed);
}
private sealed class CallbackCaptureFactory(Func<IMeetingAudioSource> create) : IMicrophoneCaptureSourceFactory
{
public IMeetingAudioSource CreateCapture(MeetingAssistantOptions options) => create();
}
[Fact]
public async Task AcceptedSwitchDuringCreationDisposesAbandonedCapture()
{
var selection = new MicrophoneDeviceSelection();
var devices = new SwitchDuringCreationDevices(selection);
var prompts = new AcceptingPrompt();
prompts.Response.SetResult(true);
var monitor = new MicrophoneSwitchMonitor(devices, prompts,
NullLogger<MicrophoneSwitchMonitor>.Instance, TimeSpan.FromMilliseconds(10), TimeSpan.FromMinutes(1));
var source = new MicrophoneAudioSource(devices, NullLogger<MicrophoneAudioSource>.Instance,
TimeSpan.Zero, monitor, selection);
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await using var chunks = source.CaptureAsync(new MeetingAssistantOptions(), lifetime.Token).GetAsyncEnumerator();
try
{
Assert.True(await chunks.MoveNextAsync());
Assert.Equal("new", selection.SelectedDeviceId);
Assert.Equal(2_000, BitConverter.ToInt16(chunks.Current.Pcm));
Assert.True(devices.AbandonedCapture.Disposed);
}
finally
{
await lifetime.CancelAsync();
}
}
private sealed class SwitchDuringCreationDevices(MicrophoneDeviceSelection selection)
: IMicrophoneDeviceProvider, IMicrophoneCaptureSourceFactory
{
private MicrophoneDevice[] available = [new("old", "Built-in")];
private int creations;
private volatile bool switchObserved;
public DisposableAudioSource AbandonedCapture { get; } = new();
public IReadOnlyList<MicrophoneDevice> GetAvailableMicrophones()
{
if (selection.SelectedDeviceId == "new") switchObserved = true;
return Volatile.Read(ref available);
}
public MicrophoneDeviceSnapshot GetMicrophoneSnapshot(MeetingAssistantOptions options)
=> new(GetAvailableMicrophones(), available[0]);
public IMeetingAudioSource CreateCapture(MeetingAssistantOptions options)
{
if (++creations > 1) return new ActiveAudioSource(Pcm16(2_000));
Volatile.Write(ref available, [new("old", "Built-in"), new("new", "USB headset")]);
// Let the acceptance callback finish before returning the already-opened source.
if (!SpinWait.SpinUntil(() => switchObserved, TimeSpan.FromSeconds(2)))
throw new TimeoutException("The new microphone was not selected during capture creation.");
return AbandonedCapture;
}
}
private sealed class DisposableAudioSource : IMeetingAudioSource, IDisposable
{
public bool Disposed { get; private set; }
public IAsyncEnumerable<AudioChunk> CaptureAsync(CancellationToken cancellationToken)
=> new ActiveAudioSource(Pcm16(1_000)).CaptureAsync(cancellationToken);
public void Dispose() => Disposed = true;
}
private sealed class AcceptingPrompt : IMicrophoneSwitchPromptService
{
public TaskCompletionSource<MicrophoneDevice> Shown { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public TaskCompletionSource<bool> Response { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public Task<bool> ConfirmSwitchAsync(MicrophoneDevice microphone, CancellationToken cancellationToken)
{
Shown.TrySetResult(microphone);
return Response.Task.WaitAsync(cancellationToken);
}
}
private sealed class SelectableCaptureDevices(MicrophoneDeviceSelection selection)
: IMicrophoneDeviceProvider, IMicrophoneCaptureSourceFactory
{
public volatile MicrophoneDevice[] Available = [new("old", "Built-in")];
public IReadOnlyList<MicrophoneDevice> GetAvailableMicrophones() => Available;
public MicrophoneDeviceSnapshot GetMicrophoneSnapshot(MeetingAssistantOptions options)
=> new(Available, selection.Resolve(options.Recording.MicrophoneDeviceId, Available[0], Available));
public IMeetingAudioSource CreateCapture(MeetingAssistantOptions options)
=> new ActiveAudioSource(Pcm16(GetMicrophoneSnapshot(options).Current!.Id == "new" ? (short)2_000 : (short)1_000));
}
private static byte[] Pcm16(short sample)
{
return BitConverter.GetBytes(sample);
@@ -0,0 +1,163 @@
using System.Threading.Channels;
using MeetingAssistant.Recording;
using Microsoft.Extensions.Logging.Abstractions;
namespace MeetingAssistant.Tests;
public sealed class MicrophoneSwitchMonitorTests
{
[Fact]
public async Task CancellationDuringConfirmationDeviceLookupCannotSwitchMicrophone()
{
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(5));
var devices = new MutableDevices();
var prompts = new ImmediatelyAcceptedPrompt(() => devices.BeforeRead = lifetime.Cancel);
var monitor = new MicrophoneSwitchMonitor(devices, prompts,
NullLogger<MicrophoneSwitchMonitor>.Instance, TimeSpan.FromMilliseconds(10), TimeSpan.FromMinutes(1));
var switched = false;
var monitoring = monitor.RunAsync((_, _) => switched = true, lifetime.Token);
devices.Available = [new("usb", "USB microphone")];
await monitoring.WaitAsync(TimeSpan.FromSeconds(2));
Assert.True(prompts.WasShown);
Assert.False(switched);
}
[Fact]
public async Task OffersOnlyNewDevicesOnceAndSwitchesOnlyAfterAcceptance()
{
var oldDevice = new MicrophoneDevice("old", "Built-in microphone");
var newDevice = new MicrophoneDevice("new", "USB headset");
var devices = new MutableDevices { Available = [oldDevice] };
var prompts = new ControlledPrompts();
var monitor = CreateMonitor(devices, prompts);
var switched = new TaskCompletionSource<MicrophoneDevice>(TaskCreationOptions.RunContinuationsAsynchronously);
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(5));
var monitoring = monitor.RunAsync((device, _) => switched.TrySetResult(device), lifetime.Token);
try
{
devices.Available = [oldDevice, newDevice];
var prompt = await prompts.NextAsync();
Assert.Equal(newDevice, prompt.Device);
Assert.False(switched.Task.IsCompleted);
await devices.WaitForPollsAsync();
Assert.False(prompts.HasMore);
prompt.Response.SetResult(true);
Assert.Equal(newDevice, await switched.Task.WaitAsync(TimeSpan.FromSeconds(2)));
await devices.WaitForPollsAsync();
Assert.False(prompts.HasMore);
}
finally
{
await lifetime.CancelAsync();
await monitoring;
}
}
[Theory]
[InlineData("declined")]
[InlineData("removed")]
[InlineData("expired")]
[InlineData("ended")]
public async Task InvalidOrDeclinedPromptsCannotChangeSelection(string outcome)
{
var device = new MicrophoneDevice("usb", "USB microphone");
var devices = new MutableDevices();
var prompts = new ControlledPrompts();
var monitor = CreateMonitor(devices, prompts,
outcome == "expired" ? TimeSpan.FromMilliseconds(30) : null);
var switched = false;
using var lifetime = new CancellationTokenSource(TimeSpan.FromSeconds(5));
var monitoring = monitor.RunAsync((_, _) => switched = true, lifetime.Token);
try
{
devices.Available = [device];
var prompt = await prompts.NextAsync();
if (outcome == "declined")
{
prompt.Response.SetResult(false);
await devices.WaitForPollsAsync();
}
else
{
if (outcome == "removed") devices.Available = [];
if (outcome == "ended") await lifetime.CancelAsync();
var canceled = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
using var registration = prompt.Cancellation.Register(() => canceled.TrySetResult());
await canceled.Task.WaitAsync(TimeSpan.FromSeconds(2));
prompt.Response.TrySetResult(true);
}
await lifetime.CancelAsync();
await monitoring;
Assert.False(switched);
}
finally
{
await lifetime.CancelAsync();
await monitoring;
}
}
private static MicrophoneSwitchMonitor CreateMonitor(MutableDevices devices, ControlledPrompts prompts,
TimeSpan? promptLifetime = null) => new(devices, prompts,
NullLogger<MicrophoneSwitchMonitor>.Instance, TimeSpan.FromMilliseconds(10),
promptLifetime ?? TimeSpan.FromMinutes(1));
private sealed class MutableDevices : IMicrophoneDeviceProvider
{
private readonly Channel<bool> polls = Channel.CreateUnbounded<bool>();
public volatile MicrophoneDevice[] Available = [];
public Action? BeforeRead { get; set; }
public IReadOnlyList<MicrophoneDevice> GetAvailableMicrophones()
{
BeforeRead?.Invoke();
polls.Writer.TryWrite(true);
return Available;
}
public MicrophoneDeviceSnapshot GetMicrophoneSnapshot(MeetingAssistantOptions options)
=> new(Available, Available.FirstOrDefault());
public async Task WaitForPollsAsync()
{
while (polls.Reader.TryRead(out _)) { }
for (var i = 0; i < 3; i++)
await polls.Reader.ReadAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(2));
}
}
private sealed class ImmediatelyAcceptedPrompt(Action beforeAcceptance) : IMicrophoneSwitchPromptService
{
public bool WasShown { get; private set; }
public Task<bool> ConfirmSwitchAsync(MicrophoneDevice microphone, CancellationToken cancellationToken)
{
WasShown = true;
beforeAcceptance();
return Task.FromResult(true);
}
}
private sealed class ControlledPrompts : IMicrophoneSwitchPromptService
{
private readonly Channel<Prompt> requests = Channel.CreateUnbounded<Prompt>();
public bool HasMore => requests.Reader.TryPeek(out _);
public Task<Prompt> NextAsync() => requests.Reader.ReadAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(2));
public async Task<bool> ConfirmSwitchAsync(MicrophoneDevice microphone, CancellationToken cancellationToken)
{
var prompt = new Prompt(microphone, cancellationToken);
requests.Writer.TryWrite(prompt);
return await prompt.Response.Task.WaitAsync(cancellationToken);
}
}
private sealed record Prompt(MicrophoneDevice Device, CancellationToken Cancellation)
{
public TaskCompletionSource<bool> Response { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
}
}
@@ -25,4 +25,16 @@ public sealed class NotificationActivationArgumentsTests
Assert.Equal("abc123", arguments["promptId"]);
Assert.Equal("continue", arguments["response"]);
}
[Fact]
public void InactivityPromptActionsIncludePauseAndParseItsResponse()
{
Assert.Contains(
MeetingInactivityPromptActions.All,
action => action.Content == "Pause transcription" &&
action.ResponseArgument == "pause");
Assert.Equal(
MeetingInactivityPromptResponse.Pause,
MeetingInactivityPromptActions.ParseResponse("pause"));
}
}
@@ -1,10 +1,90 @@
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Summary;
using System.Text.Json;
namespace MeetingAssistant.Tests;
public sealed class ProjectKnowledgeToolTests
{
[Theory]
[InlineData("name: Incomplete project")]
[InlineData("description: Missing name")]
[InlineData("name: ' '\ndescription: Missing name")]
public async Task CatalogUsesLegacyFallbackForIncompleteMetadata(string frontmatter)
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var projectRoot = Path.Combine(root, "Projects", "Alpha");
Directory.CreateDirectory(projectRoot);
await File.WriteAllTextAsync(Path.Combine(projectRoot, "PROJECT.md"), $"---\n{frontmatter}\n---\nNotes");
var tools = new MeetingSummaryTools(CreateArtifacts(root),
new MeetingAssistantOptions { Vault = { ProjectsFolder = Path.Combine(root, "Projects") } });
using var catalog = JsonDocument.Parse(await tools.ListProjects());
var project = Assert.Single(catalog.RootElement.EnumerateArray());
Assert.Equal("Alpha", project.GetProperty("displayname").GetString());
Assert.Equal("", project.GetProperty("description").GetString());
}
[Fact]
public async Task ProjectMetadataWriteEnforcesDescriptionLimitAndPreservesOtherContent()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var projectRoot = Path.Combine(root, "Projects", "Alpha");
Directory.CreateDirectory(projectRoot);
var artifacts = CreateArtifacts(root);
Directory.CreateDirectory(Path.GetDirectoryName(artifacts.MeetingNotePath)!);
await File.WriteAllTextAsync(artifacts.MeetingNotePath, "---\nprojects: Alpha\n---");
var tools = new MeetingSummaryTools(artifacts,
new MeetingAssistantOptions { Vault = { ProjectsFolder = Path.Combine(root, "Projects") } });
var original = $"---\nname: Alpha Platform\ndescription: {new string('a', 256)}\nowner: Ada\n---\nUser-authored project notes.";
Assert.Equal("Alpha/PROJECT.md", await tools.WriteProjectFile("Alpha", "PROJECT.md", original));
var rejected = await tools.WriteProjectFile("Alpha", "PROJECT.md",
$"description: {new string('b', 257)}", from: 3, to: 3);
Assert.StartsWith("Refused:", rejected);
Assert.Equal(original, await File.ReadAllTextAsync(Path.Combine(projectRoot, "PROJECT.md")));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("Alpha", "PROJECT.md", "No metadata", replace_file: true));
Assert.Equal("Alpha/PROJECT.md", await tools.WriteProjectFile("Alpha", "PROJECT.md",
"description: Updated project purpose", from: 3, to: 3));
var updated = await tools.ReadProjectFile("Alpha", "PROJECT.md");
Assert.Contains("owner: Ada", updated);
Assert.Contains("User-authored project notes.", updated);
Assert.Contains("Updated project purpose", await tools.ListProjects());
}
[Fact]
public async Task CatalogListsAllProjectsWithMetadataAndLegacyFallbackWithoutGrantingReadAccess()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var projectsRoot = Path.Combine(root, "Projects");
Directory.CreateDirectory(Path.Combine(projectsRoot, "Alpha"));
Directory.CreateDirectory(Path.Combine(projectsRoot, "Legacy"));
Directory.CreateDirectory(Path.Combine(projectsRoot, "Malformed"));
await File.WriteAllTextAsync(Path.Combine(projectsRoot, "Alpha", "PROJECT.md"),
"---\nname: Alpha Platform\ndescription: Meeting automation\n---\nPrivate project notes");
await File.WriteAllTextAsync(Path.Combine(projectsRoot, "Alpha", "AGENTS.md"), "Private instructions");
await File.WriteAllTextAsync(Path.Combine(projectsRoot, "Malformed", "PROJECT.md"), "---\nname: [broken\n---");
var tools = new MeetingSummaryTools(CreateArtifacts(root),
new MeetingAssistantOptions { Vault = { ProjectsFolder = projectsRoot } });
var catalog = await tools.ListProjects();
using var json = JsonDocument.Parse(catalog);
var projects = json.RootElement.EnumerateArray().ToArray();
Assert.Equal(3, projects.Length);
Assert.Equal("Alpha", projects[0].GetProperty("id").GetString());
Assert.Equal("Alpha Platform", projects[0].GetProperty("displayname").GetString());
Assert.Equal("Meeting automation", projects[0].GetProperty("description").GetString());
Assert.Equal("Legacy", projects[1].GetProperty("displayname").GetString());
Assert.Equal("", projects[1].GetProperty("description").GetString());
Assert.Equal("Malformed", projects[2].GetProperty("displayname").GetString());
Assert.Equal("", projects[2].GetProperty("description").GetString());
Assert.DoesNotContain("Private", catalog);
Assert.Equal("", await tools.ReadProjectFile("Alpha", "AGENTS.md"));
}
[Fact]
public async Task ToolsOperateOnProjectsBoundInMeetingFrontmatter()
{
@@ -48,7 +128,7 @@ public sealed class ProjectKnowledgeToolTests
}
});
Assert.Equal("MeetingAssistant", await tools.ListProjects());
Assert.Contains("MeetingAssistant", await tools.ListProjects());
Assert.Equal("README.md\nnotes/context.md", await tools.ListProjectFiles("MeetingAssistant"));
Assert.Equal("Second alpha line\nThird beta line", await tools.ReadProjectFile("MeetingAssistant", "README.md", 2, 99));
@@ -56,15 +136,15 @@ public sealed class ProjectKnowledgeToolTests
Assert.Equal("MeetingAssistant/notes/summary.md", writeResult);
Assert.Equal("# Project Update", await File.ReadAllTextAsync(Path.Combine(meetingAssistantRoot, "notes", "summary.md")));
Assert.Equal("IgnoredProject/ignored.md", await tools.WriteProjectFile("IgnoredProject", "ignored.md", "changed"));
Assert.Equal("alpha should not be searched\nchanged", await File.ReadAllTextAsync(Path.Combine(ignoredRoot, "ignored.md")));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("IgnoredProject", "ignored.md", "changed"));
Assert.Equal("alpha should not be searched", await File.ReadAllTextAsync(Path.Combine(ignoredRoot, "ignored.md")));
Assert.Equal(
"README.md:2 Second alpha line",
await tools.Search("alpha"));
}
[Fact]
public async Task ListProjectsAcceptsScalarProjectFrontmatter()
public async Task ProjectFileAccessAcceptsScalarProjectFrontmatter()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var projectsRoot = Path.Combine(root, "Projects");
@@ -88,7 +168,8 @@ public sealed class ProjectKnowledgeToolTests
}
});
Assert.Equal("MeetingAssistant", await tools.ListProjects());
Assert.Equal("MeetingAssistant/notes.md", await tools.WriteProjectFile("MeetingAssistant", "notes.md", "Bound project"));
Assert.Equal("Bound project", await tools.ReadProjectFile("MeetingAssistant", "notes.md"));
}
[Fact]
@@ -100,8 +181,11 @@ public sealed class ProjectKnowledgeToolTests
Directory.CreateDirectory(projectRoot);
var projectFile = Path.Combine(projectRoot, "notes.md");
await File.WriteAllTextAsync(projectFile, "one\ntwo\nthree\nfour");
var artifacts = CreateArtifacts(root);
Directory.CreateDirectory(Path.GetDirectoryName(artifacts.MeetingNotePath)!);
await File.WriteAllTextAsync(artifacts.MeetingNotePath, "---\nprojects: MeetingAssistant\n---\n");
var tools = new MeetingSummaryTools(
CreateArtifacts(root),
artifacts,
new MeetingAssistantOptions
{
Vault =
@@ -132,8 +216,13 @@ public sealed class ProjectKnowledgeToolTests
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var projectsRoot = Path.Combine(root, "Projects");
Directory.CreateDirectory(Path.Combine(projectsRoot, "MeetingAssistant"));
var artifacts = CreateArtifacts(root);
Directory.CreateDirectory(Path.GetDirectoryName(artifacts.MeetingNotePath)!);
await File.WriteAllTextAsync(artifacts.MeetingNotePath, "---\nprojects: MeetingAssistant\n---\n");
var outsidePath = Path.Combine(projectsRoot, "outside.md");
await File.WriteAllTextAsync(outsidePath, "Preserve outside content");
var tools = new MeetingSummaryTools(
CreateArtifacts(root),
artifacts,
new MeetingAssistantOptions
{
Vault =
@@ -144,6 +233,7 @@ public sealed class ProjectKnowledgeToolTests
Assert.StartsWith("Refused:", await tools.WriteProjectFile("MissingProject", "notes.md", "content"));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("MeetingAssistant", "../outside.md", "content"));
Assert.Equal("Preserve outside content", await File.ReadAllTextAsync(outsidePath));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("MeetingAssistant", "notes.md", "content", from: 1));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("MeetingAssistant", "notes.md", "content", from: 1, to: 1, insert: 1));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("MeetingAssistant", "notes.md", "content", from: 1, to: 1, replace_file: true));
@@ -3,11 +3,92 @@ using MeetingAssistant.LaunchProfiles;
using MeetingAssistant.Transcription;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using System.Collections.Concurrent;
using System.Runtime.ExceptionServices;
namespace MeetingAssistant.Tests;
public sealed class PyannoteDiarizationWarmupHostedServiceTests
{
[Fact]
public async Task HostedServiceStopsSafelyWhenShutdownRequestsOverlap()
{
const int iterations = 64;
const int shutdownCallers = 8;
using var applicationLifetime = new CancellationTokenSource();
var finalizer = new PyannoteTranscriptFinalizer(
new BlockingCommandRunner(),
Options.Create(new MeetingAssistantOptions()),
NullLogger<PyannoteTranscriptFinalizer>.Instance);
using var callersReady = new Barrier(shutdownCallers + 1);
var shutdownErrors = new ConcurrentQueue<ExceptionDispatchInfo>();
PyannoteDiarizationWarmupHostedService? service = null;
var shutdownTasks = Enumerable.Range(0, shutdownCallers)
.Select(_ => Task.Factory.StartNew(() =>
{
try
{
for (var iteration = 0; iteration < iterations; iteration++)
{
WaitForShutdownCallers();
try
{
service!.StopAsync(CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(5)).GetAwaiter().GetResult();
}
catch (Exception exception)
{
shutdownErrors.Enqueue(ExceptionDispatchInfo.Capture(exception));
}
WaitForShutdownCallers();
if (!shutdownErrors.IsEmpty)
{
break;
}
}
}
catch (Exception exception)
{
shutdownErrors.Enqueue(ExceptionDispatchInfo.Capture(exception));
}
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default))
.ToArray();
try
{
for (var iteration = 0; iteration < iterations; iteration++)
{
service = new PyannoteDiarizationWarmupHostedService(
finalizer,
new FakeLaunchProfileOptionsProvider(new MeetingAssistantOptions()),
NullLogger<PyannoteDiarizationWarmupHostedService>.Instance);
await service.StartAsync(applicationLifetime.Token);
WaitForShutdownCallers();
WaitForShutdownCallers();
if (!shutdownErrors.IsEmpty)
{
break;
}
}
}
finally
{
await Task.WhenAll(shutdownTasks).WaitAsync(TimeSpan.FromSeconds(10));
}
if (shutdownErrors.TryPeek(out var shutdownError))
{
shutdownError.Throw();
}
void WaitForShutdownCallers()
{
Assert.True(
callersReady.SignalAndWait(TimeSpan.FromSeconds(5)),
"Timed out coordinating overlapping shutdown requests.");
}
}
[Fact]
public async Task HostedServiceWarmsEnabledValidationRuntimeWithoutBlockingStartup()
{
@@ -18,31 +99,8 @@ public sealed class PyannoteDiarizationWarmupHostedServiceTests
NullLogger<PyannoteTranscriptFinalizer>.Instance);
var service = new PyannoteDiarizationWarmupHostedService(
finalizer,
new FakeLaunchProfileOptionsProvider(new MeetingAssistantOptions
{
Recording = { TranscriptionProvider = "azure-speech" },
SpeakerIdentification =
{
PyannoteValidation =
{
Enabled = true,
Diarization =
{
Enabled = true,
DockerCommand = "docker",
Image = "meeting-assistant-pyannote-validation:local",
ModelsFolder = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"),
"models"),
Token = "hf_test",
TokenEnv = "",
CommandTimeout = TimeSpan.FromMinutes(1)
}
}
}
}),
new FakeLaunchProfileOptionsProvider(
CreateValidationOptions("meeting-assistant-pyannote-validation:local")),
NullLogger<PyannoteDiarizationWarmupHostedService>.Instance);
await service.StartAsync(CancellationToken.None).WaitAsync(TimeSpan.FromSeconds(1));
@@ -54,23 +112,109 @@ public sealed class PyannoteDiarizationWarmupHostedServiceTests
Assert.True(commandRunner.RunCancellationWasObserved);
}
[Fact]
public async Task HostedServiceUsesApplicationValidationRuntimeInsteadOfNamedProfileOverride()
{
var commandRunner = new BlockingCommandRunner();
var finalizer = new PyannoteTranscriptFinalizer(
commandRunner,
Options.Create(new MeetingAssistantOptions()),
NullLogger<PyannoteTranscriptFinalizer>.Instance);
var defaultOptions = CreateValidationOptions("meeting-assistant-pyannote-default:local");
var namedOptions = CreateValidationOptions("meeting-assistant-pyannote-named:local");
var service = new PyannoteDiarizationWarmupHostedService(
finalizer,
new FakeLaunchProfileOptionsProvider(
[
new LaunchProfile("english", namedOptions),
new LaunchProfile(ConfigurationLaunchProfileOptionsProvider.DefaultProfileName, defaultOptions)
]),
NullLogger<PyannoteDiarizationWarmupHostedService>.Instance);
await service.StartAsync(CancellationToken.None).WaitAsync(TimeSpan.FromSeconds(1));
await commandRunner.WaitForRunAsync();
await service.StopAsync(CancellationToken.None);
Assert.Contains(commandRunner.Commands, command => command.Arguments.Contains("meeting-assistant-pyannote-default:local"));
Assert.DoesNotContain(commandRunner.Commands, command => command.Arguments.Contains("meeting-assistant-pyannote-named:local"));
}
[Fact]
public async Task HostedServiceDoesNotWarmIdentityValidationWhenResemblyzerIsSelected()
{
var commandRunner = new BlockingCommandRunner();
var finalizer = new PyannoteTranscriptFinalizer(
commandRunner,
Options.Create(new MeetingAssistantOptions()),
NullLogger<PyannoteTranscriptFinalizer>.Instance);
var options = CreateValidationOptions("meeting-assistant-pyannote-validation:local");
options.SpeakerIdentification.Resemblyzer.Enabled = true;
var service = new PyannoteDiarizationWarmupHostedService(
finalizer,
new FakeLaunchProfileOptionsProvider(options),
NullLogger<PyannoteDiarizationWarmupHostedService>.Instance);
await service.StartAsync(CancellationToken.None);
await Task.Delay(TimeSpan.FromMilliseconds(50));
await service.StopAsync(CancellationToken.None);
Assert.Empty(commandRunner.Commands);
}
private static MeetingAssistantOptions CreateValidationOptions(string image)
{
return new MeetingAssistantOptions
{
Recording = { TranscriptionProvider = "azure-speech" },
SpeakerIdentification =
{
PyannoteValidation =
{
Enabled = true,
Diarization =
{
DockerCommand = "docker",
Image = image,
ModelsFolder = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"),
"models"),
Token = "hf_test",
TokenEnv = "",
CommandTimeout = TimeSpan.FromMinutes(1)
}
}
}
};
}
private sealed class FakeLaunchProfileOptionsProvider : ILaunchProfileOptionsProvider
{
private readonly MeetingAssistantOptions options;
private readonly IReadOnlyList<LaunchProfile> profiles;
public FakeLaunchProfileOptionsProvider(MeetingAssistantOptions options)
: this([new LaunchProfile(ConfigurationLaunchProfileOptionsProvider.DefaultProfileName, options)])
{
this.options = options;
}
public FakeLaunchProfileOptionsProvider(IReadOnlyList<LaunchProfile> profiles)
{
this.profiles = profiles;
}
public LaunchProfile GetRequiredProfile(string? name)
{
return new LaunchProfile(ConfigurationLaunchProfileOptionsProvider.DefaultProfileName, options);
var profileName = string.IsNullOrWhiteSpace(name)
? ConfigurationLaunchProfileOptionsProvider.DefaultProfileName
: name;
return profiles.Single(profile => profile.Name.Equals(profileName, StringComparison.OrdinalIgnoreCase));
}
public IReadOnlyList<LaunchProfile> GetProfiles()
{
return [GetRequiredProfile(null)];
return profiles;
}
public IReadOnlyList<LaunchProfileHotkey> GetHotkeys()
@@ -2,6 +2,7 @@ using MeetingAssistant;
using MeetingAssistant.Recording;
using MeetingAssistant.Speakers;
using MeetingAssistant.Transcription;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NAudio.Wave;
@@ -57,17 +58,49 @@ public sealed class PyannoteSpeakerIdentityMatchValidatorTests
Assert.Empty(commandRunner.Commands);
}
[Fact]
public async Task OuterToggleControlsValidationWhenLegacyNestedToggleIsFalse()
{
var commandRunner = new CapturingCommandRunner(
"""
__MEETING_ASSISTANT_PYANNOTE_JSON_START__
[{"start":0.0,"end":20.0,"speaker":"SPEAKER_00"}]
__MEETING_ASSISTANT_PYANNOTE_JSON_END__
""");
var modelsFolder = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"),
"models");
var configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["SpeakerIdentification:PyannoteValidation:Enabled"] = "true",
["SpeakerIdentification:PyannoteValidation:Diarization:Enabled"] = "false",
["SpeakerIdentification:PyannoteValidation:Diarization:BuildImage"] = "false",
["SpeakerIdentification:PyannoteValidation:Diarization:ModelsFolder"] = modelsFolder,
["SpeakerIdentification:PyannoteValidation:Diarization:Token"] = "hf_test",
["SpeakerIdentification:PyannoteValidation:Diarization:TokenEnv"] = ""
})
.Build();
var configuredOptions = configuration.Get<MeetingAssistantOptions>()!;
var validator = CreateValidator(commandRunner, configuredOptions);
var valid = await validator.ValidateSampleAsync(
CreateWav(TimeSpan.FromSeconds(20)),
CancellationToken.None);
Assert.True(valid);
Assert.Contains(commandRunner.Commands, command => command.Arguments.Contains("run"));
}
private static PyannoteSpeakerIdentityMatchValidator CreateValidator(
CapturingCommandRunner commandRunner,
bool enabled = true)
{
var finalizer = new PyannoteTranscriptFinalizer(
return CreateValidator(
commandRunner,
Options.Create(new MeetingAssistantOptions()),
NullLogger<PyannoteTranscriptFinalizer>.Instance);
return new PyannoteSpeakerIdentityMatchValidator(
finalizer,
Options.Create(new MeetingAssistantOptions
new MeetingAssistantOptions
{
SpeakerIdentification = new SpeakerIdentificationOptions
{
@@ -76,9 +109,8 @@ public sealed class PyannoteSpeakerIdentityMatchValidatorTests
Enabled = enabled,
MinimumSingleSpeakerCoverage = 0.90,
MinimumMatchingKnownSnippetRatio = 1,
Diarization = new PyannoteDiarizationOptions
Diarization = new PyannoteRuntimeOptions
{
Enabled = true,
BuildImage = false,
DockerCommand = "docker",
Image = "meeting-assistant-pyannote:local",
@@ -94,7 +126,20 @@ public sealed class PyannoteSpeakerIdentityMatchValidatorTests
}
}
}
}),
});
}
private static PyannoteSpeakerIdentityMatchValidator CreateValidator(
CapturingCommandRunner commandRunner,
MeetingAssistantOptions configuredOptions)
{
var finalizer = new PyannoteTranscriptFinalizer(
commandRunner,
Options.Create(new MeetingAssistantOptions()),
NullLogger<PyannoteTranscriptFinalizer>.Instance);
return new PyannoteSpeakerIdentityMatchValidator(
finalizer,
Options.Create(configuredOptions),
NullLogger<PyannoteSpeakerIdentityMatchValidator>.Instance);
}
@@ -156,9 +156,8 @@ public sealed class PyannoteTranscriptFinalizerTests
commandRunner,
Options.Create(new MeetingAssistantOptions()),
NullLogger<PyannoteTranscriptFinalizer>.Instance);
var explicitDiarization = new PyannoteDiarizationOptions
var explicitDiarization = new PyannoteRuntimeOptions
{
Enabled = true,
DockerCommand = "docker",
BaseImage = "python:3.11-slim",
Image = "meeting-assistant-pyannote-azure:local",
@@ -169,7 +168,7 @@ public sealed class PyannoteTranscriptFinalizerTests
CommandTimeout = TimeSpan.FromMinutes(1)
};
await finalizer.FinalizeAsync(
await finalizer.FinalizeEnabledAsync(
audioPath,
[new TranscriptionSegment(TimeSpan.Zero, TimeSpan.FromSeconds(1), "Unknown", "hello")],
explicitDiarization,
@@ -186,9 +185,8 @@ public sealed class PyannoteTranscriptFinalizerTests
{
var commandRunner = new CapturingCommandRunner("");
var finalizer = CreateFinalizer(commandRunner, token: "hf_test");
var diarization = new PyannoteDiarizationOptions
var diarization = new PyannoteRuntimeOptions
{
Enabled = true,
DockerCommand = "docker",
BaseImage = "python:3.11-slim",
Image = "meeting-assistant-pyannote-warmup:local",
@@ -218,9 +216,8 @@ public sealed class PyannoteTranscriptFinalizerTests
var commandRunner = new CapturingCommandRunner("");
var finalizer = CreateFinalizer(commandRunner, token: null);
await finalizer.WarmUpAsync(new PyannoteDiarizationOptions
await finalizer.WarmUpAsync(new PyannoteRuntimeOptions
{
Enabled = true,
Token = "",
TokenEnv = "",
ModelsFolder = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"), "models")
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,596 @@
using MeetingAssistant;
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Speakers;
using MeetingAssistant.Transcription;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Tests;
public sealed class ResemblyzerSpeakerIdentificationServiceTests
{
[Fact]
public async Task LiveMatchRelabelsSpeakerAndStoresFiveVectorsWithoutWavSnippets()
{
await using var fixture = await Fixture.CreateAsync();
var identity = await fixture.AddIdentityAsync("Chris", [UnitVector(0), UnitVector(0, 20, 0.02f)]);
fixture.Encoder.Vectors = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
var result = await service.IdentifyKnownSpeakersAsync(
fixture.CreateRequest("Guest03", sampleCount: 5),
CancellationToken.None);
Assert.Equal("Chris", result.Segments.Single().Speaker);
Assert.Equal("Chris", result.SpeakerMappings["Guest03"]);
var saved = await fixture.LoadIdentityAsync(identity.Id);
Assert.Equal(7, saved.VoiceVectors.Count);
Assert.Empty(saved.Snippets);
Assert.Single(saved.References);
Assert.Single(fixture.Encoder.Requests);
Assert.Equal(5, fixture.Encoder.Requests[0].Count);
}
[Fact]
public async Task SummaryOverrideCreatesNamedIdentityFromThreeAvailableVectors()
{
await using var fixture = await Fixture.CreateAsync();
fixture.Encoder.Vectors = Enumerable.Range(0, 3)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
var request = fixture.CreateRequest("Guest-01", sampleCount: 3);
await service.ApplySpeakerOverrideAsync(
request,
"Guest-01",
"Sabrina",
CancellationToken.None);
var saved = await fixture.LoadOnlyIdentityAsync();
Assert.Equal("Sabrina", saved.CanonicalName);
Assert.Equal(3, saved.VoiceVectors.Count);
Assert.Empty(saved.Snippets);
Assert.Single(saved.References);
Assert.Equal(3, fixture.Encoder.Requests.Single().Count);
}
[Fact]
public async Task FinalProcessingLearnsUnmatchedSpeakerFromFiveVectorsAndAttendees()
{
await using var fixture = await Fixture.CreateAsync();
fixture.Encoder.Vectors = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
await service.ProcessFinishedTranscriptAsync(
fixture.CreateRequest("Guest-01", sampleCount: 5, attendees: ["John", "Mike"]),
CancellationToken.None);
var saved = await fixture.LoadOnlyIdentityAsync();
Assert.Null(saved.CanonicalName);
Assert.Equal(["John", "Mike"], saved.CandidateNames.Select(candidate => candidate.Name).Order());
Assert.Equal(5, saved.VoiceVectors.Count);
Assert.Empty(saved.Snippets);
Assert.Single(saved.References);
}
[Fact]
public async Task FiveVectorThresholdDoesNotCapQualifyingCurrentRunEvidence()
{
await using var fixture = await Fixture.CreateAsync();
fixture.Encoder.Vectors = Enumerable.Range(0, 8)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
await service.ProcessFinishedTranscriptAsync(
fixture.CreateRequest("Guest-01", sampleCount: 8, attendees: ["John", "Mike"]),
CancellationToken.None);
var saved = await fixture.LoadOnlyIdentityAsync();
Assert.Equal(8, saved.VoiceVectors.Count);
Assert.Equal(8, fixture.Encoder.Requests.Single().Count);
}
[Fact]
public async Task FinalProcessingPrunesForeignVectorsFromNewIdentityAtMinimum()
{
await using var fixture = await Fixture.CreateAsync(options =>
{
options.OutlierPruningMinimumVectors = 20;
options.OutlierPruningNeighborSimilarity = 0.90;
options.OutlierPruningMinimumNeighbors = 3;
options.OutlierPruningMinimumClusterRatio = 0.60;
});
fixture.Encoder.Vectors = Enumerable.Range(0, 16)
.Select(index => UnitVector(0, index + 2, 0.04f))
.Concat(Enumerable.Range(0, 4)
.Select(index => UnitVector(1, index + 30, 0.04f)))
.ToList();
var service = fixture.CreateService();
await service.ProcessFinishedTranscriptAsync(
fixture.CreateRequest("Guest-01", sampleCount: 20, attendees: ["John", "Mike"]),
CancellationToken.None);
var saved = await fixture.LoadOnlyIdentityAsync();
Assert.Equal(16, saved.VoiceVectors.Count);
Assert.All(
saved.VoiceVectors,
stored => Assert.True(SpeakerVoiceVectors.Decode(stored)[0] > 0.9f));
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public async Task FinalProcessingStoresVectorsCollectedAfterLiveMatch(bool transcriptAlreadyRelabeled)
{
await using var fixture = await Fixture.CreateAsync();
var identity = await fixture.AddIdentityAsync(
"Chris",
[UnitVector(0), UnitVector(0, 20, 0.02f)]);
var currentRunVectors = Enumerable.Range(0, 8)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
fixture.Encoder.Vectors = currentRunVectors.Take(5).ToList();
var service = fixture.CreateService();
var liveResult = await service.IdentifyKnownSpeakersAsync(
fixture.CreateRequest("Guest03", sampleCount: 5),
CancellationToken.None);
fixture.Encoder.Vectors = currentRunVectors;
var finalRequest = fixture.CreateRequest("Guest03", sampleCount: 8) with
{
KnownSpeakerMappings = liveResult.SpeakerMappings
};
if (transcriptAlreadyRelabeled)
{
finalRequest = finalRequest with { Segments = liveResult.Segments };
}
await service.ProcessFinishedTranscriptAsync(finalRequest, CancellationToken.None);
var saved = await fixture.LoadIdentityAsync(identity.Id);
Assert.Equal(10, saved.VoiceVectors.Count);
Assert.Equal([5, 8], fixture.Encoder.Requests.Select(request => request.Count));
}
[Fact]
public async Task RepeatedMatchDeduplicatesVectorsAndHonorsConfiguredLimit()
{
await using var fixture = await Fixture.CreateAsync(options => options.MaxVectorsPerIdentity = 6);
var identity = await fixture.AddIdentityAsync("Chris", [UnitVector(0), UnitVector(0, 20, 0.02f)]);
fixture.Encoder.Vectors =
[
UnitVector(0),
UnitVector(0, 2, 0.01f),
UnitVector(0, 3, 0.02f),
UnitVector(0, 4, 0.03f),
UnitVector(0, 5, 0.04f)
];
var service = fixture.CreateService();
var request = fixture.CreateRequest("Guest03", sampleCount: 5);
await service.IdentifyKnownSpeakersAsync(request, CancellationToken.None);
await service.IdentifyKnownSpeakersAsync(request, CancellationToken.None);
var saved = await fixture.LoadIdentityAsync(identity.Id);
Assert.Equal(6, saved.VoiceVectors.Count);
Assert.Single(saved.References);
}
[Fact]
public async Task FinishedMatchingExtractsFiveNonOverlappingSamplesWhenLiveSamplesAreMissing()
{
await using var fixture = await Fixture.CreateAsync();
await fixture.AddIdentityAsync("Chris", [UnitVector(0), UnitVector(0, 20, 0.02f)]);
fixture.Encoder.Vectors = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
var result = await service.IdentifyFinishedSpeakersAsync(
fixture.CreateRequest("Guest03", sampleCount: 0, segmentCount: 5),
CancellationToken.None);
Assert.Equal("Chris", result.Segments[0].Speaker);
Assert.Equal(5, fixture.SnippetExtractor.Requests.Count);
Assert.All(
fixture.SnippetExtractor.Requests.Zip(fixture.SnippetExtractor.Requests.Skip(1)),
pair => Assert.True(pair.First[^1].End <= pair.Second[0].Start));
Assert.Equal(5, fixture.Encoder.Requests.Single().Count);
}
[Fact]
public async Task FinalMatchPromotesTheRemainingCandidateAndAuditsTheTranscript()
{
await using var fixture = await Fixture.CreateAsync();
var identity = await fixture.AddIdentityAsync(
null,
[UnitVector(0), UnitVector(0, 20, 0.02f)],
candidates: ["John", "Mike"]);
fixture.Encoder.Vectors = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
var result = await service.ProcessFinishedTranscriptAsync(
fixture.CreateRequest("Guest03", sampleCount: 5, attendees: ["Jane", "John", "Chris"]),
CancellationToken.None);
var saved = await fixture.LoadIdentityAsync(identity.Id);
Assert.Equal("John", saved.CanonicalName);
Assert.Equal(["John"], saved.CandidateNames.Select(candidate => candidate.Name));
Assert.Equal("John", result.Segments.Single().Speaker);
Assert.All(saved.References, reference =>
Assert.Contains("Guest03 was identified as John", File.ReadAllText(reference.TranscriptPath)));
}
[Fact]
public async Task FinalMatchResetsCandidatesWhenAttendeesDoNotIntersect()
{
await using var fixture = await Fixture.CreateAsync();
var identity = await fixture.AddIdentityAsync(
null,
[UnitVector(0), UnitVector(0, 20, 0.02f)],
candidates: ["John", "Mike"]);
fixture.Encoder.Vectors = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
await service.ProcessFinishedTranscriptAsync(
fixture.CreateRequest("Guest03", sampleCount: 5, attendees: ["Jane", "Chris"]),
CancellationToken.None);
var saved = await fixture.LoadIdentityAsync(identity.Id);
Assert.Null(saved.CanonicalName);
Assert.Equal(["Chris", "Jane"], saved.CandidateNames.Select(candidate => candidate.Name).Order());
}
[Fact]
public async Task FinalUnmatchedSpeakerWithOneCandidateIsAuditedWhenLearned()
{
await using var fixture = await Fixture.CreateAsync();
fixture.Encoder.Vectors = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, index + 2, 0.03f))
.ToList();
var service = fixture.CreateService();
await service.ProcessFinishedTranscriptAsync(
fixture.CreateRequest("Guest-01", sampleCount: 5, attendees: ["Manuel"]),
CancellationToken.None);
var saved = await fixture.LoadOnlyIdentityAsync();
Assert.Equal("Manuel", saved.CanonicalName);
Assert.Contains("Guest-01 was identified as Manuel", File.ReadAllText(saved.References.Single().TranscriptPath));
}
[Fact]
public async Task CandidateLimitCountsOnlyIdentitiesWithCompatibleVectors()
{
await using var fixture = await Fixture.CreateAsync(
configureSpeaker: options => options.MaxMatchCandidates = 1);
await fixture.AddIdentityAsync("Legacy WAV identity", []);
await fixture.AddIdentityAsync("Chris", [UnitVector(0)]);
fixture.Encoder.Vectors = Enumerable.Repeat(UnitVector(0), 5).ToList();
var service = fixture.CreateService();
var result = await service.IdentifyKnownSpeakersAsync(
fixture.CreateRequest("Guest03", sampleCount: 5, attendees: []),
CancellationToken.None);
Assert.Equal("Chris", result.Segments.Single().Speaker);
}
[Fact]
public async Task SummaryOverrideMergePreservesAllEvidenceFromCurrentRunCandidate()
{
await using var fixture = await Fixture.CreateAsync();
var target = await fixture.AddIdentityAsync("Sabrina", [UnitVector(0)]);
var request = fixture.CreateRequest("Guest-01", sampleCount: 3, attendees: ["Sabrina"]);
await fixture.AddCurrentRunCandidateAsync(request, "Sabrina");
fixture.Encoder.Vectors = Enumerable.Range(0, 3)
.Select(index => UnitVector(0, index + 2, 0.02f))
.ToList();
var service = fixture.CreateService();
await service.ApplySpeakerOverrideAsync(
request,
"Guest-01",
"Sabrina",
CancellationToken.None);
var saved = await fixture.LoadIdentityAsync(target.Id);
Assert.Single(saved.Snippets);
Assert.Equal(5, saved.VoiceVectors.Count);
Assert.Contains(saved.VoiceVectors, vector => vector.ModelId == "older-model");
await using var context = new TestDbContextFactory(fixture.DatabasePath).CreateDbContext();
Assert.Single(await context.SpeakerIdentities.ToListAsync());
}
private static float[] UnitVector(
int primaryDimension,
int? secondaryDimension = null,
float secondaryValue = 0)
{
var vector = new float[256];
vector[primaryDimension] = 1;
if (secondaryDimension is { } dimension)
{
vector[dimension] = secondaryValue;
}
return vector;
}
private static byte[] ToBytes(float[] vector)
{
var bytes = new byte[vector.Length * sizeof(float)];
Buffer.BlockCopy(vector, 0, bytes, 0, bytes.Length);
return bytes;
}
private sealed class Fixture : IAsyncDisposable
{
private readonly string directory;
private readonly string databasePath;
private readonly SpeakerIdentificationOptions speakerOptions;
private Fixture(string directory, string databasePath, SpeakerIdentificationOptions speakerOptions)
{
this.directory = directory;
this.databasePath = databasePath;
this.speakerOptions = speakerOptions;
}
public FakeEncoder Encoder { get; } = new();
public FakeSnippetExtractor SnippetExtractor { get; } = new();
public string DatabasePath => databasePath;
public static async Task<Fixture> CreateAsync(
Action<ResemblyzerSpeakerRecognitionOptions>? configure = null,
Action<SpeakerIdentificationOptions>? configureSpeaker = null)
{
var directory = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(directory);
var databasePath = Path.Combine(directory, "speaker-identities.db");
var speakerOptions = new SpeakerIdentificationOptions
{
DatabasePath = databasePath,
MatchBatchSize = 6,
MaxMatchCandidates = 100,
MatchIdentityActiveAge = TimeSpan.FromDays(365),
MinimumSampleSpeechDuration = TimeSpan.Zero,
Resemblyzer = new ResemblyzerSpeakerRecognitionOptions
{
Enabled = true,
RequiredVectorsPerSpeaker = 5,
MaxVectorsPerIdentity = 1000,
MinimumClusterCohesion = 0.75,
MinimumIdentitySimilarity = 0.75,
MinimumSimilarityMargin = 0.05,
ModelId = "resemblyzer-0.1.4-pretrained"
}
};
configure?.Invoke(speakerOptions.Resemblyzer);
configureSpeaker?.Invoke(speakerOptions);
await using var context = new SpeakerIdentityDbContext(
new DbContextOptionsBuilder<SpeakerIdentityDbContext>()
.UseSqlite($"Data Source={databasePath};Pooling=False")
.Options);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, CancellationToken.None);
return new Fixture(directory, databasePath, speakerOptions);
}
public ResemblyzerSpeakerIdentificationService CreateService()
{
var appOptions = new MeetingAssistantOptions { SpeakerIdentification = speakerOptions };
return new ResemblyzerSpeakerIdentificationService(
new TestDbContextFactory(databasePath),
SnippetExtractor,
Encoder,
new ResemblyzerVoiceClusterMatcher(
speakerOptions.Resemblyzer,
NullLogger<ResemblyzerVoiceClusterMatcher>.Instance),
new ResemblyzerVoiceVectorOutlierPruner(
speakerOptions.Resemblyzer,
NullLogger<ResemblyzerVoiceVectorOutlierPruner>.Instance),
Options.Create(appOptions),
NullLogger<ResemblyzerSpeakerIdentificationService>.Instance);
}
public SpeakerIdentificationRequest CreateRequest(
string speaker,
int sampleCount,
IReadOnlyList<string>? attendees = null,
int segmentCount = 1)
{
var transcriptPath = Path.Combine(directory, "transcript.md");
File.WriteAllText(transcriptPath, "Transcript");
var segments = Enumerable.Range(0, segmentCount)
.Select(index => new TranscriptionSegment(
TimeSpan.FromSeconds(index * 30),
TimeSpan.FromSeconds((index + 1) * 30),
speaker,
"enough useful words for a speaker sample"))
.ToList();
var segment = segments[0];
return new SpeakerIdentificationRequest(
Path.Combine(directory, "meeting.wav"),
new MeetingNote(
Path.Combine(directory, "meeting.md"),
new MeetingNoteFrontmatter
{
Title = "Test",
Attendees = attendees?.ToList() ?? ["Chris"],
Transcript = transcriptPath,
AssistantContext = Path.Combine(directory, "context.md"),
Summary = Path.Combine(directory, "summary.md")
},
""),
segments,
Enumerable.Range(0, sampleCount)
.Select(index => new SpeakerAudioSample(speaker, segment, [(byte)(index + 1)], 100 - index))
.ToList());
}
public async Task<SpeakerIdentity> AddIdentityAsync(
string? name,
IReadOnlyList<float[]> vectors,
IReadOnlyList<string>? candidates = null,
IReadOnlyList<string>? aliases = null)
{
await using var context = new TestDbContextFactory(databasePath).CreateDbContext();
var now = DateTimeOffset.UtcNow;
var identity = new SpeakerIdentity
{
CanonicalName = name,
CreatedAt = now,
UpdatedAt = now,
CandidateNames = candidates?.Select(candidate => new SpeakerCandidateName { Name = candidate }).ToList() ?? [],
Aliases = aliases?.Select(alias => new SpeakerAlias { Name = alias }).ToList() ?? [],
VoiceVectors = vectors.Select((vector, index) => new SpeakerVoiceVector
{
ModelId = speakerOptions.Resemblyzer.ModelId,
Dimensions = 256,
VectorBytes = ToBytes(vector),
Fingerprint = $"known-{index}",
CreatedAt = now.AddMinutes(index)
}).ToList()
};
context.SpeakerIdentities.Add(identity);
await context.SaveChangesAsync();
return identity;
}
public async Task AddCurrentRunCandidateAsync(
SpeakerIdentificationRequest request,
string candidateName)
{
await using var context = new TestDbContextFactory(databasePath).CreateDbContext();
var now = DateTimeOffset.UtcNow;
context.SpeakerIdentities.Add(new SpeakerIdentity
{
CreatedAt = now,
UpdatedAt = now,
CandidateNames = [new SpeakerCandidateName { Name = candidateName }],
Snippets = [new SpeakerSnippet { WavBytes = [9, 8, 7], CreatedAt = now }],
VoiceVectors =
[
new SpeakerVoiceVector
{
ModelId = "older-model",
Dimensions = 256,
VectorBytes = ToBytes(UnitVector(1)),
Fingerprint = "older-model-vector",
CreatedAt = now
}
],
References =
[
SpeakerIdentityReferences.Create(
request.MeetingNote.Path,
request.MeetingNote.Frontmatter.Transcript,
now)
]
});
await context.SaveChangesAsync();
}
public async Task<SpeakerIdentity> LoadIdentityAsync(int id)
{
await using var context = new TestDbContextFactory(databasePath).CreateDbContext();
return await context.SpeakerIdentities
.Include(identity => identity.Snippets)
.Include(identity => identity.VoiceVectors)
.Include(identity => identity.CandidateNames)
.Include(identity => identity.Aliases)
.Include(identity => identity.References)
.SingleAsync(identity => identity.Id == id);
}
public async Task<SpeakerIdentity> LoadOnlyIdentityAsync()
{
await using var context = new TestDbContextFactory(databasePath).CreateDbContext();
return await context.SpeakerIdentities
.Include(identity => identity.Snippets)
.Include(identity => identity.VoiceVectors)
.Include(identity => identity.CandidateNames)
.Include(identity => identity.References)
.SingleAsync();
}
public ValueTask DisposeAsync()
{
if (Directory.Exists(directory))
{
Directory.Delete(directory, recursive: true);
}
return ValueTask.CompletedTask;
}
}
private sealed class FakeEncoder : IResemblyzerVoiceEncoder
{
public IReadOnlyList<float[]> Vectors { get; set; } = [];
public List<IReadOnlyList<byte[]>> Requests { get; } = [];
public Task<IReadOnlyList<float[]>> EncodeAsync(
IReadOnlyList<byte[]> wavSamples,
CancellationToken cancellationToken)
{
Requests.Add(wavSamples.Select(sample => sample.ToArray()).ToList());
return Task.FromResult<IReadOnlyList<float[]>>(Vectors.Take(wavSamples.Count).ToList());
}
public Task WarmUpAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
private sealed class FakeSnippetExtractor : ISpeakerSnippetExtractor
{
public List<IReadOnlyList<TranscriptionSegment>> Requests { get; } = [];
public Task<byte[]> ExtractSnippetAsync(
string audioPath,
IReadOnlyList<TranscriptionSegment> speakerSegments,
CancellationToken cancellationToken)
{
Requests.Add(speakerSegments.ToList());
return Task.FromResult<byte[]>([checked((byte)Requests.Count)]);
}
}
private sealed class TestDbContextFactory : IDbContextFactory<SpeakerIdentityDbContext>
{
private readonly string databasePath;
public TestDbContextFactory(string databasePath)
{
this.databasePath = databasePath;
}
public SpeakerIdentityDbContext CreateDbContext()
{
return new SpeakerIdentityDbContext(
new DbContextOptionsBuilder<SpeakerIdentityDbContext>()
.UseSqlite($"Data Source={databasePath};Pooling=False")
.Options);
}
}
}
@@ -0,0 +1,174 @@
using MeetingAssistant;
using MeetingAssistant.Speakers;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Tests;
public sealed class ResemblyzerSpeakerIdentityMergeServiceTests
{
[Fact]
public async Task MergeRecentIdentitiesRequiresTwoDisjointMatchingVectorClusters()
{
var directory = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(directory);
try
{
var databasePath = Path.Combine(directory, "identities.db");
var factory = new TestDbContextFactory(databasePath);
await using (var context = factory.CreateDbContext())
{
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, CancellationToken.None);
context.SpeakerIdentities.Add(CreateIdentity(
"Chris",
DateTimeOffset.UtcNow.AddMonths(-2),
Enumerable.Range(0, 10)
.Select(index => UnitVector(0, index + 20, 0.02f))
.ToList(),
directory));
context.SpeakerIdentities.Add(CreateIdentity(
"Chris duplicate",
DateTimeOffset.UtcNow.AddDays(-1),
Enumerable.Range(0, 10)
.Select(index => UnitVector(0, index + 2, 0.02f))
.Concat(Enumerable.Range(0, 4)
.Select(index => UnitVector(1, index + 40, 0.02f)))
.ToList(),
directory));
context.SpeakerIdentities.Add(CreateIdentity(
"Decoy",
DateTimeOffset.UtcNow.AddMonths(-1),
[UnitVector(0)],
directory));
await context.SaveChangesAsync();
}
var resemblyzerOptions = new ResemblyzerSpeakerRecognitionOptions
{
Enabled = true,
RequiredVectorsPerSpeaker = 5,
MaxVectorsPerIdentity = 1000,
MinimumClusterCohesion = 0.75,
MinimumIdentitySimilarity = 0.75,
MinimumSimilarityMargin = 0.05,
ModelId = "resemblyzer-0.1.4-pretrained"
};
var service = new ResemblyzerSpeakerIdentityMergeService(
factory,
new ResemblyzerVoiceClusterMatcher(
resemblyzerOptions,
NullLogger<ResemblyzerVoiceClusterMatcher>.Instance),
new ResemblyzerVoiceVectorOutlierPruner(
resemblyzerOptions,
NullLogger<ResemblyzerVoiceVectorOutlierPruner>.Instance),
Options.Create(new MeetingAssistantOptions
{
SpeakerIdentification = new SpeakerIdentificationOptions
{
MergeRecentIdentityAge = TimeSpan.FromDays(14),
MaxMatchCandidates = 1,
MaxSnippetsPerSpeaker = 3,
Resemblyzer = resemblyzerOptions
}
}),
NullLogger<ResemblyzerSpeakerIdentityMergeService>.Instance);
var result = await service.MergeRecentIdentitiesAsync(TimeSpan.FromDays(14), CancellationToken.None);
Assert.Equal(2, result.MatchAttempts);
Assert.Equal(1, result.MergedPairs);
await using var verification = factory.CreateDbContext();
var saved = await verification.SpeakerIdentities
.Include(identity => identity.Aliases)
.Include(identity => identity.VoiceVectors)
.SingleAsync(identity => identity.CanonicalName == "Chris");
Assert.Equal(2, await verification.SpeakerIdentities.CountAsync());
Assert.Equal("Chris", saved.CanonicalName);
Assert.Contains(saved.Aliases, alias => alias.Name == "Chris duplicate");
Assert.Equal(20, saved.VoiceVectors.Count);
Assert.All(saved.VoiceVectors, vector => Assert.True(ToVector(vector.VectorBytes)[0] > 0.9f));
}
finally
{
Directory.Delete(directory, recursive: true);
}
}
private static SpeakerIdentity CreateIdentity(
string name,
DateTimeOffset createdAt,
IReadOnlyList<float[]> vectors,
string directory)
{
var transcriptPath = Path.Combine(directory, $"{Guid.NewGuid():N}.md");
File.WriteAllText(transcriptPath, "Transcript");
return new SpeakerIdentity
{
CanonicalName = name,
CreatedAt = createdAt,
UpdatedAt = createdAt,
VoiceVectors = vectors.Select((vector, index) => new SpeakerVoiceVector
{
ModelId = "resemblyzer-0.1.4-pretrained",
Dimensions = 256,
VectorBytes = ToBytes(vector),
Fingerprint = $"{name}-{index}",
CreatedAt = createdAt.AddMinutes(index)
}).ToList(),
References =
[
new SpeakerIdentityReference
{
MeetingNotePath = Path.Combine(directory, $"{Guid.NewGuid():N}.md"),
TranscriptPath = transcriptPath,
CreatedAt = createdAt
}
]
};
}
private static float[] UnitVector(int primary, int? secondary = null, float secondaryValue = 0)
{
var vector = new float[256];
vector[primary] = 1;
if (secondary is { } index)
{
vector[index] = secondaryValue;
}
return vector;
}
private static byte[] ToBytes(float[] vector)
{
var bytes = new byte[vector.Length * sizeof(float)];
Buffer.BlockCopy(vector, 0, bytes, 0, bytes.Length);
return bytes;
}
private static float[] ToVector(byte[] bytes)
{
var vector = new float[bytes.Length / sizeof(float)];
Buffer.BlockCopy(bytes, 0, vector, 0, bytes.Length);
return vector;
}
private sealed class TestDbContextFactory : IDbContextFactory<SpeakerIdentityDbContext>
{
private readonly string databasePath;
public TestDbContextFactory(string databasePath)
{
this.databasePath = databasePath;
}
public SpeakerIdentityDbContext CreateDbContext()
{
return new SpeakerIdentityDbContext(
new DbContextOptionsBuilder<SpeakerIdentityDbContext>()
.UseSqlite($"Data Source={databasePath};Pooling=False")
.Options);
}
}
}
@@ -0,0 +1,122 @@
using MeetingAssistant;
using MeetingAssistant.Speakers;
using Microsoft.Extensions.Logging.Abstractions;
namespace MeetingAssistant.Tests;
public sealed class ResemblyzerVoiceClusterMatcherTests
{
[Fact]
public void CoherentSimilarClusterSelectsUnambiguousIdentity()
{
var matcher = CreateMatcher();
var query = Enumerable.Range(0, 5)
.Select(index => UnitVector(0, secondaryDimension: index + 2, secondaryValue: 0.05f))
.ToList();
var candidates = new[]
{
new ResemblyzerVoiceVectorCandidate(42, [UnitVector(0), UnitVector(0, 10, 0.02f)]),
new ResemblyzerVoiceVectorCandidate(77, [UnitVector(1), UnitVector(1, 11, 0.02f)])
};
var result = matcher.Match(query, candidates);
Assert.True(result.Accepted);
Assert.Equal(42, result.IdentityId);
Assert.True(result.Cohesion >= 0.99);
Assert.True(result.BestSimilarity >= 0.99);
Assert.True(result.RunnerUpSimilarity < 0.1);
}
[Fact]
public void FourVectorsDoNotTriggerAutomaticMatching()
{
var result = CreateMatcher().Match(
Enumerable.Repeat(UnitVector(0), 4).ToList(),
[new ResemblyzerVoiceVectorCandidate(42, [UnitVector(0)])]);
Assert.False(result.Accepted);
Assert.Contains("4/5", result.Reason);
}
[Fact]
public void IncoherentClusterIsRejectedBeforeIdentityScoring()
{
var result = CreateMatcher().Match(
Enumerable.Range(0, 5).Select(index => UnitVector(index)).ToList(),
[new ResemblyzerVoiceVectorCandidate(42, [UnitVector(0)])]);
Assert.False(result.Accepted);
Assert.Contains("cohesion", result.Reason);
Assert.Null(result.BestSimilarity);
}
[Fact]
public void SimilarityBelowThresholdIsRejected()
{
var result = CreateMatcher().Match(
Enumerable.Repeat(UnitVector(0), 5).ToList(),
[new ResemblyzerVoiceVectorCandidate(42, [UnitVector(1)])]);
Assert.False(result.Accepted);
Assert.Contains("best similarity", result.Reason);
Assert.Equal(0, result.BestSimilarity);
}
[Fact]
public void SimilarCandidatesWithinMarginAreRejected()
{
var result = CreateMatcher().Match(
Enumerable.Repeat(UnitVector(0), 5).ToList(),
[
new ResemblyzerVoiceVectorCandidate(42, [UnitVector(0)]),
new ResemblyzerVoiceVectorCandidate(77, [UnitVector(0, 1, 0.01f)])
]);
Assert.False(result.Accepted);
Assert.Contains("margin", result.Reason);
Assert.NotNull(result.RunnerUpSimilarity);
}
[Fact]
public void InvalidStoredCandidateDoesNotAbortScoringOtherIdentities()
{
var result = CreateMatcher().Match(
Enumerable.Repeat(UnitVector(0), 5).ToList(),
[
new ResemblyzerVoiceVectorCandidate(13, [new float[256]]),
new ResemblyzerVoiceVectorCandidate(42, [UnitVector(0)])
]);
Assert.True(result.Accepted);
Assert.Equal(42, result.IdentityId);
}
private static ResemblyzerVoiceClusterMatcher CreateMatcher()
{
return new ResemblyzerVoiceClusterMatcher(
new ResemblyzerSpeakerRecognitionOptions
{
RequiredVectorsPerSpeaker = 5,
MinimumClusterCohesion = 0.75,
MinimumIdentitySimilarity = 0.75,
MinimumSimilarityMargin = 0.05
},
NullLogger<ResemblyzerVoiceClusterMatcher>.Instance);
}
private static float[] UnitVector(
int primaryDimension,
int? secondaryDimension = null,
float secondaryValue = 0)
{
var vector = new float[256];
vector[primaryDimension] = 1;
if (secondaryDimension is { } dimension)
{
vector[dimension] = secondaryValue;
}
return vector;
}
}
@@ -0,0 +1,141 @@
using MeetingAssistant;
using MeetingAssistant.Speakers;
using Microsoft.Extensions.Logging.Abstractions;
namespace MeetingAssistant.Tests;
public sealed class ResemblyzerVoiceVectorOutlierPrunerTests
{
[Fact]
public void DominantDensityClusterRemovesForeignSpeakerVectorsAtThreshold()
{
var identity = IdentityWithVectors(
Enumerable.Range(0, 16)
.Select(index => UnitVector(0, index + 2, 0.04f))
.Concat(Enumerable.Range(0, 4)
.Select(index => UnitVector(1, index + 30, 0.04f)))
.ToList());
var pruner = CreatePruner();
var result = pruner.Prune(identity);
Assert.Equal(4, result);
Assert.Equal(16, identity.VoiceVectors.Count);
Assert.All(
identity.VoiceVectors,
stored => Assert.True(SpeakerVoiceVectors.Decode(stored)[0] > 0.9f));
}
[Fact]
public void AmbiguousDenseClustersPreserveAllVectors()
{
var identity = IdentityWithVectors(
Enumerable.Range(0, 10)
.Select(index => UnitVector(0, index + 2, 0.04f))
.Concat(Enumerable.Range(0, 10)
.Select(index => UnitVector(1, index + 30, 0.04f)))
.ToList());
var pruner = CreatePruner();
var result = pruner.Prune(identity);
Assert.Equal(0, result);
Assert.Equal(20, identity.VoiceVectors.Count);
}
[Fact]
public void BelowMinimumVectorCountPreservesAllVectorsWithoutEvaluation()
{
var identity = IdentityWithVectors(
Enumerable.Range(0, 15)
.Select(index => UnitVector(0, index + 2, 0.04f))
.Concat(Enumerable.Range(0, 4)
.Select(index => UnitVector(1, index + 30, 0.04f)))
.ToList());
var pruner = CreatePruner();
var result = pruner.Prune(identity);
Assert.Equal(0, result);
Assert.Equal(19, identity.VoiceVectors.Count);
}
[Fact]
public void PruningPreservesOtherModelsAndMalformedStoredRows()
{
var identity = IdentityWithVectors(
Enumerable.Range(0, 16)
.Select(index => UnitVector(0, index + 2, 0.04f))
.Concat(Enumerable.Range(0, 4)
.Select(index => UnitVector(1, index + 30, 0.04f)))
.ToList());
SpeakerVoiceVectors.AddDistinct(
identity,
[UnitVector(2, 60, 0.04f)],
"older-model",
1000,
DateTimeOffset.UtcNow);
identity.VoiceVectors.Add(new SpeakerVoiceVector
{
ModelId = "resemblyzer-0.1.4-pretrained",
Dimensions = 256,
VectorBytes = [1],
Fingerprint = "malformed",
CreatedAt = DateTimeOffset.UtcNow
});
identity.VoiceVectors.Add(new SpeakerVoiceVector
{
ModelId = "resemblyzer-0.1.4-pretrained",
Dimensions = 256,
VectorBytes = new byte[256 * sizeof(float)],
Fingerprint = "zero-magnitude",
CreatedAt = DateTimeOffset.UtcNow
});
var pruner = CreatePruner();
var result = pruner.Prune(identity);
Assert.Equal(4, result);
Assert.Equal(19, identity.VoiceVectors.Count);
Assert.Contains(identity.VoiceVectors, vector => vector.ModelId == "older-model");
Assert.Contains(identity.VoiceVectors, vector => vector.Fingerprint == "malformed");
Assert.Contains(identity.VoiceVectors, vector => vector.Fingerprint == "zero-magnitude");
}
private static ResemblyzerVoiceVectorOutlierPruner CreatePruner()
{
return new ResemblyzerVoiceVectorOutlierPruner(
new ResemblyzerSpeakerRecognitionOptions
{
ModelId = "resemblyzer-0.1.4-pretrained",
OutlierPruningMinimumVectors = 20,
OutlierPruningNeighborSimilarity = 0.90,
OutlierPruningMinimumNeighbors = 3,
OutlierPruningMinimumClusterRatio = 0.60
},
NullLogger<ResemblyzerVoiceVectorOutlierPruner>.Instance);
}
private static SpeakerIdentity IdentityWithVectors(IReadOnlyList<float[]> vectors)
{
var identity = new SpeakerIdentity();
SpeakerVoiceVectors.AddDistinct(
identity,
vectors,
"resemblyzer-0.1.4-pretrained",
1000,
DateTimeOffset.UtcNow);
return identity;
}
private static float[] UnitVector(
int primaryDimension,
int secondaryDimension,
float secondaryValue)
{
var vector = new float[256];
vector[primaryDimension] = 1;
vector[secondaryDimension] = secondaryValue;
return vector;
}
}
@@ -0,0 +1,78 @@
using MeetingAssistant;
using MeetingAssistant.Speakers;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Tests;
public sealed class ResemblyzerWarmupHostedServiceTests
{
[Fact]
public async Task EnabledWarmupStartsWithoutBlockingApplicationStartup()
{
var encoder = new BlockingEncoder();
var options = new MeetingAssistantOptions();
options.SpeakerIdentification.Resemblyzer.Enabled = true;
var service = new ResemblyzerWarmupHostedService(
encoder,
Options.Create(options),
NullLogger<ResemblyzerWarmupHostedService>.Instance);
await service.StartAsync(CancellationToken.None).WaitAsync(TimeSpan.FromSeconds(1));
await encoder.WaitForWarmupAsync();
await service.StopAsync(CancellationToken.None);
Assert.True(encoder.CancellationObserved);
}
[Fact]
public async Task DisabledWarmupDoesNotInvokeEncoder()
{
var encoder = new BlockingEncoder();
var service = new ResemblyzerWarmupHostedService(
encoder,
Options.Create(new MeetingAssistantOptions()),
NullLogger<ResemblyzerWarmupHostedService>.Instance);
await service.StartAsync(CancellationToken.None);
await service.StopAsync(CancellationToken.None);
Assert.False(encoder.WarmupStarted);
}
private sealed class BlockingEncoder : IResemblyzerVoiceEncoder
{
private readonly TaskCompletionSource started = new(TaskCreationOptions.RunContinuationsAsynchronously);
public bool WarmupStarted { get; private set; }
public bool CancellationObserved { get; private set; }
public Task<IReadOnlyList<float[]>> EncodeAsync(
IReadOnlyList<byte[]> wavSamples,
CancellationToken cancellationToken)
{
return Task.FromResult<IReadOnlyList<float[]>>([]);
}
public async Task WarmUpAsync(CancellationToken cancellationToken)
{
WarmupStarted = true;
started.TrySetResult();
try
{
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
CancellationObserved = true;
}
}
public Task WaitForWarmupAsync()
{
return started.Task.WaitAsync(TimeSpan.FromSeconds(1));
}
}
}
@@ -8,13 +8,22 @@ namespace MeetingAssistant.Tests;
public sealed class SpeakerAudioSampleCollectorTests
{
[Fact]
public void CollectorWaitsForConfiguredUninterruptedSpeechDuration()
public void DefaultSampleDurationsAreTenAndSixtySeconds()
{
var options = new SpeakerIdentificationOptions();
Assert.Equal(TimeSpan.FromSeconds(10), options.MinimumSampleSpeechDuration);
Assert.Equal(TimeSpan.FromSeconds(60), options.MaximumSampleDuration);
}
[Fact]
public void CollectorWaitsForConfiguredSpeakerAudioDuration()
{
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 3,
minimumUninterruptedSpeechDuration: TimeSpan.FromSeconds(30),
maximumSegmentGap: TimeSpan.FromSeconds(1));
minimumSampleSpeechDuration: TimeSpan.FromSeconds(30),
maximumSampleDuration: TimeSpan.FromSeconds(60));
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(35)));
collector.TryAdd(Segment(0, 12, "Guest01", "one two three four five."));
@@ -37,8 +46,8 @@ public sealed class SpeakerAudioSampleCollectorTests
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 3,
minimumUninterruptedSpeechDuration: TimeSpan.FromSeconds(30),
maximumSegmentGap: TimeSpan.FromSeconds(1));
minimumSampleSpeechDuration: TimeSpan.FromSeconds(30),
maximumSampleDuration: TimeSpan.FromSeconds(60));
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(50)));
collector.TryAdd(Segment(0, 20, "Guest01", "one two three four five."));
@@ -48,6 +57,61 @@ public sealed class SpeakerAudioSampleCollectorTests
Assert.DoesNotContain(collector.Snapshot(), sample => sample.Speaker == "Guest01");
}
[Fact]
public void CollectorCombinesConsecutiveSameSpeakerSegmentsAcrossProviderPauses()
{
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 3,
minimumSampleSpeechDuration: TimeSpan.FromSeconds(10),
maximumSampleDuration: TimeSpan.FromSeconds(60));
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(20)));
collector.TryAdd(Segment(0, 4, "Guest01", "one two three four."));
collector.TryAdd(Segment(7, 13, "Guest01", "five six seven eight."));
var sample = Assert.Single(collector.Snapshot());
Assert.Equal(TimeSpan.Zero, sample.Segment.Start);
Assert.Equal(TimeSpan.FromSeconds(13), sample.Segment.End);
}
[Fact]
public void CollectorDoesNotCountProviderPausesAsSpeakerAudio()
{
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 3,
minimumSampleSpeechDuration: TimeSpan.FromSeconds(10),
maximumSampleDuration: TimeSpan.FromSeconds(60));
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(30)));
collector.TryAdd(Segment(0, 4, "Guest01", "one two three four."));
collector.TryAdd(Segment(20, 25, "Guest01", "five six seven eight."));
Assert.Empty(collector.Snapshot());
}
[Fact]
public void CollectorCapsRecognitionWavAtConfiguredMaximumDuration()
{
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 3,
minimumSampleSpeechDuration: TimeSpan.FromSeconds(10),
maximumSampleDuration: TimeSpan.FromSeconds(60));
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(75)));
var sample = collector.TryAdd(Segment(
0,
70,
"Guest01",
"one two three four five six seven eight nine ten."));
Assert.NotNull(sample);
Assert.Equal(TimeSpan.FromSeconds(60), sample.Segment.End);
Assert.True(ReadDuration(sample.WavBytes) <= TimeSpan.FromSeconds(60));
}
[Fact]
public void CollectorLogsWhenSampleIsDiscardedBecauseSpeechIsTooShort()
{
@@ -55,8 +119,8 @@ public sealed class SpeakerAudioSampleCollectorTests
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 3,
minimumUninterruptedSpeechDuration: TimeSpan.FromSeconds(30),
maximumSegmentGap: TimeSpan.FromSeconds(1),
minimumSampleSpeechDuration: TimeSpan.FromSeconds(30),
maximumSampleDuration: TimeSpan.FromSeconds(60),
logger: logger);
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(35)));
@@ -69,6 +133,47 @@ public sealed class SpeakerAudioSampleCollectorTests
Assert.Contains("minimum duration", message);
}
[Fact]
public void CollectorCanResetThePendingSpanAfterEachAcceptedSample()
{
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 5,
minimumSampleSpeechDuration: TimeSpan.FromSeconds(2),
maximumSampleDuration: TimeSpan.FromSeconds(60),
requireNonOverlappingSamples: true);
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(6)));
collector.TryAdd(Segment(0, 2, "Guest01", "one two three."));
collector.TryAdd(Segment(2, 4, "Guest01", "four five six."));
collector.TryAdd(Segment(4, 6, "Guest01", "seven eight nine."));
var samples = collector.Snapshot().OrderBy(sample => sample.Segment.Start).ToList();
Assert.Collection(
samples,
sample => Assert.Equal((TimeSpan.Zero, TimeSpan.FromSeconds(2)), (sample.Segment.Start, sample.Segment.End)),
sample => Assert.Equal((TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(4)), (sample.Segment.Start, sample.Segment.End)),
sample => Assert.Equal((TimeSpan.FromSeconds(4), TimeSpan.FromSeconds(6)), (sample.Segment.Start, sample.Segment.End)));
}
[Fact]
public void ResettingCollectorRejectsASegmentThatOverlapsAnAcceptedSample()
{
var collector = new SpeakerAudioSampleCollector(
TimeSpan.FromMinutes(2),
maxSamplesPerSpeaker: 5,
minimumSampleSpeechDuration: TimeSpan.FromSeconds(2),
maximumSampleDuration: TimeSpan.FromSeconds(60),
requireNonOverlappingSamples: true);
collector.AppendAudio(CreateAudio(TimeSpan.FromSeconds(4)));
collector.TryAdd(Segment(0, 2, "Guest01", "one two three."));
collector.TryAdd(Segment(1, 3, "Guest01", "four five six."));
Assert.Single(collector.Snapshot());
}
private static TranscriptionSegment Segment(
double start,
double end,
@@ -107,6 +107,9 @@ public sealed class SpeakerIdentityMergeServiceTests
return new SpeakerIdentityMergeService(
new TestSpeakerIdentityDbContextFactory(dbPath),
Matcher,
new ResemblyzerVoiceVectorOutlierPruner(
new ResemblyzerSpeakerRecognitionOptions(),
NullLogger<ResemblyzerVoiceVectorOutlierPruner>.Instance),
Options.Create(new MeetingAssistantOptions
{
SpeakerIdentification = new SpeakerIdentificationOptions
@@ -0,0 +1,47 @@
using MeetingAssistant.Speakers;
namespace MeetingAssistant.Tests;
public sealed class SpeakerIdentityMergerTests
{
[Fact]
public void MergeIntoRetainsNewestDistinctVoiceVectorsUpToConfiguredLimit()
{
var now = DateTimeOffset.UtcNow;
var target = new SpeakerIdentity
{
CanonicalName = "Chris",
VoiceVectors =
[
Vector("oldest", 1, now.AddMinutes(-4)),
Vector("duplicate", 2, now.AddMinutes(-3))
]
};
var source = new SpeakerIdentity
{
VoiceVectors =
[
Vector("duplicate", 2, now.AddMinutes(-2)),
Vector("newer", 3, now.AddMinutes(-1)),
Vector("newest", 4, now)
]
};
SpeakerIdentityMerger.MergeInto(target, source, maxSnippets: 3, maxVoiceVectors: 3);
Assert.Equal(["duplicate", "newer", "newest"], target.VoiceVectors.Select(vector => vector.Fingerprint).Order());
Assert.All(target.VoiceVectors, vector => Assert.Same(target, vector.SpeakerIdentity));
}
private static SpeakerVoiceVector Vector(string fingerprint, byte value, DateTimeOffset createdAt)
{
return new SpeakerVoiceVector
{
ModelId = "model",
Dimensions = 256,
VectorBytes = Enumerable.Repeat(value, 256 * sizeof(float)).ToArray(),
Fingerprint = fingerprint,
CreatedAt = createdAt
};
}
}
@@ -0,0 +1,80 @@
using MeetingAssistant.Speakers;
using Microsoft.Data.Sqlite;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
namespace MeetingAssistant.Tests;
public sealed class SpeakerRecognitionFeatureSelectionTests
{
[Theory]
[InlineData(false, typeof(SpeakerIdentityService), typeof(SpeakerIdentityMergeService), 1, false)]
[InlineData(true, typeof(ResemblyzerSpeakerIdentificationService), typeof(ResemblyzerSpeakerIdentityMergeService), 1000, true)]
public async Task FeatureFlagSelectsExactlyOneIdentificationBackend(
bool enabled,
Type expectedIdentificationType,
Type expectedMergeType,
int expectedMinimumSamples,
bool expectedNonOverlappingSamples)
{
var directory = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(directory);
try
{
await using var factory = new WebApplicationFactory<Program>()
.WithWebHostBuilder(builder =>
{
builder.ConfigureAppConfiguration((_, configuration) =>
{
configuration.AddInMemoryCollection(new Dictionary<string, string?>
{
["MeetingAssistant:FunAsr:Backend:Enabled"] = "false",
["MeetingAssistant:SpeakerIdentification:DatabasePath"] = Path.Combine(directory, "identities.db"),
["MeetingAssistant:SpeakerIdentification:PyannoteValidation:Enabled"] = "false",
["MeetingAssistant:SpeakerIdentification:Resemblyzer:Enabled"] = enabled.ToString()
});
});
builder.ConfigureTestServices(services =>
{
services.RemoveAll<IResemblyzerVoiceEncoder>();
services.AddSingleton<IResemblyzerVoiceEncoder, NoopEncoder>();
});
});
var service = factory.Services.GetRequiredService<ISpeakerIdentificationService>();
Assert.IsType(expectedIdentificationType, service);
Assert.Single(factory.Services.GetServices<ISpeakerIdentificationService>());
var mergeService = factory.Services.GetRequiredService<ISpeakerIdentityMergeService>();
Assert.IsType(expectedMergeType, mergeService);
Assert.Single(factory.Services.GetServices<ISpeakerIdentityMergeService>());
var policy = factory.Services.GetRequiredService<SpeakerSampleCollectionPolicy>();
Assert.Equal(expectedMinimumSamples, policy.MinimumRetainedSamples);
Assert.Equal(expectedNonOverlappingSamples, policy.RequireNonOverlappingSamples);
}
finally
{
SqliteConnection.ClearAllPools();
Directory.Delete(directory, recursive: true);
}
}
private sealed class NoopEncoder : IResemblyzerVoiceEncoder
{
public Task<IReadOnlyList<float[]>> EncodeAsync(
IReadOnlyList<byte[]> wavSamples,
CancellationToken cancellationToken)
{
return Task.FromResult<IReadOnlyList<float[]>>([]);
}
public Task WarmUpAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
}
@@ -0,0 +1,142 @@
using MeetingAssistant.Speakers;
using MeetingAssistant.Transcription;
namespace MeetingAssistant.Tests;
public sealed class SpeakerSampleSpanSelectorTests
{
[Fact]
public void SelectBestSameSpeakerSpanCombinesLinesAndCapsTheClip()
{
var span = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpan(
[
Segment(0, 4),
Segment(7, 13),
Segment(20, 70)
],
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60));
Assert.Equal(3, span.Count);
Assert.Equal(TimeSpan.Zero, span[0].Start);
Assert.Equal(TimeSpan.FromSeconds(60), span[^1].End);
Assert.Equal(TimeSpan.FromSeconds(60), SpeakerSampleSpanSelector.SpanDuration(span));
}
[Fact]
public void SelectBestSameSpeakerSpanDoesNotCountProviderPausesAsSpeakerAudio()
{
var span = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpan(
[
Segment(0, 4),
Segment(20, 25)
],
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60));
Assert.Empty(span);
}
[Fact]
public void SelectBestSameSpeakerSpansReturnsDistinctMinimumLengthSamples()
{
var segments = Enumerable.Range(0, 6)
.Select(index => new TranscriptionSegment(
TimeSpan.FromSeconds(index * 10),
TimeSpan.FromSeconds((index + 1) * 10),
"Guest01",
$"segment {index} has enough words"))
.ToList();
var spans = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpans(
segments,
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60),
maxSpans: 5);
Assert.Equal(5, spans.Count);
Assert.All(spans, span => Assert.True(SpeakerSampleSpanSelector.SpanDuration(span) >= TimeSpan.FromSeconds(10)));
Assert.All(
spans.Zip(spans.Skip(1)),
pair => Assert.True(pair.First[^1].End <= pair.Second[0].Start));
}
[Fact]
public void SelectBestSameSpeakerSpansSkipsSegmentsThatOverlapACompletedSample()
{
var spans = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpans(
[
Segment(0, 10),
Segment(9, 19),
Segment(20, 30)
],
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60),
maxSpans: 5);
Assert.Equal(2, spans.Count);
Assert.Equal(TimeSpan.Zero, spans[0][0].Start);
Assert.Equal(TimeSpan.FromSeconds(20), spans[1][0].Start);
}
[Fact]
public void SelectBestSameSpeakerSpansCombinesConsecutiveLinesAcrossProviderPauses()
{
var spans = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpans(
[
Segment(0, 4),
Segment(7, 13)
],
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60),
maxSpans: 5);
var span = Assert.Single(spans);
Assert.Equal(TimeSpan.Zero, span[0].Start);
Assert.Equal(TimeSpan.FromSeconds(13), span[^1].End);
}
[Fact]
public void SelectBestSameSpeakerSpansDoesNotCountProviderPausesAsSpeakerAudio()
{
var spans = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpans(
[
Segment(0, 4),
Segment(20, 25)
],
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60),
maxSpans: 5);
Assert.Empty(spans);
}
[Fact]
public void SelectBestSameSpeakerSpansCapsFinalizedSamplesAtConfiguredMaximumDuration()
{
var spans = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpans(
[Segment(0, 70)],
"Guest01",
TimeSpan.FromSeconds(10),
TimeSpan.FromSeconds(60),
maxSpans: 5);
var span = Assert.Single(spans);
Assert.Equal(TimeSpan.FromSeconds(60), SpeakerSampleSpanSelector.SpanDuration(span));
}
private static TranscriptionSegment Segment(double start, double end)
{
return new TranscriptionSegment(
TimeSpan.FromSeconds(start),
TimeSpan.FromSeconds(end),
"Guest01",
"enough words for this segment");
}
}
@@ -0,0 +1,106 @@
using MeetingAssistant.Speakers;
using Microsoft.EntityFrameworkCore;
namespace MeetingAssistant.Tests;
public sealed class SpeakerVoiceVectorPersistenceTests
{
[Fact]
public async Task IdentityStoresVersionedVoiceVectorAndDeletesItWithIdentity()
{
var databasePath = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"),
"speaker-identities.db");
Directory.CreateDirectory(Path.GetDirectoryName(databasePath)!);
var options = new DbContextOptionsBuilder<SpeakerIdentityDbContext>()
.UseSqlite($"Data Source={databasePath}")
.Options;
await using (var context = new SpeakerIdentityDbContext(options))
{
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, CancellationToken.None);
context.SpeakerIdentities.Add(new SpeakerIdentity
{
CanonicalName = "Chris",
CreatedAt = DateTimeOffset.UtcNow,
UpdatedAt = DateTimeOffset.UtcNow,
VoiceVectors =
[
new SpeakerVoiceVector
{
ModelId = "resemblyzer-0.1.4-pretrained",
Dimensions = 256,
VectorBytes = new byte[256 * sizeof(float)],
Fingerprint = "vector-a",
CreatedAt = DateTimeOffset.UtcNow
}
]
});
await context.SaveChangesAsync();
}
await using (var context = new SpeakerIdentityDbContext(options))
{
var identity = await context.SpeakerIdentities
.Include(candidate => candidate.VoiceVectors)
.SingleAsync();
var vector = Assert.Single(identity.VoiceVectors);
Assert.Equal("resemblyzer-0.1.4-pretrained", vector.ModelId);
Assert.Equal(256, vector.Dimensions);
Assert.Equal(256 * sizeof(float), vector.VectorBytes.Length);
context.SpeakerIdentities.Remove(identity);
await context.SaveChangesAsync();
}
await using (var context = new SpeakerIdentityDbContext(options))
{
Assert.Empty(await context.SpeakerVoiceVectors.ToListAsync());
}
}
[Fact]
public async Task IdentityRejectsDuplicateVoiceVectorFingerprint()
{
var databasePath = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"),
"speaker-identities.db");
Directory.CreateDirectory(Path.GetDirectoryName(databasePath)!);
var options = new DbContextOptionsBuilder<SpeakerIdentityDbContext>()
.UseSqlite($"Data Source={databasePath}")
.Options;
await using var context = new SpeakerIdentityDbContext(options);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, CancellationToken.None);
var identity = new SpeakerIdentity
{
CanonicalName = "Chris",
CreatedAt = DateTimeOffset.UtcNow,
UpdatedAt = DateTimeOffset.UtcNow
};
context.SpeakerIdentities.Add(identity);
await context.SaveChangesAsync();
context.SpeakerVoiceVectors.AddRange(
CreateVector(identity.Id, "same-vector"),
CreateVector(identity.Id, "same-vector"));
await Assert.ThrowsAsync<DbUpdateException>(() => context.SaveChangesAsync());
}
private static SpeakerVoiceVector CreateVector(int identityId, string fingerprint)
{
return new SpeakerVoiceVector
{
SpeakerIdentityId = identityId,
ModelId = "resemblyzer-0.1.4-pretrained",
Dimensions = 256,
VectorBytes = new byte[256 * sizeof(float)],
Fingerprint = fingerprint,
CreatedAt = DateTimeOffset.UtcNow
};
}
}
@@ -0,0 +1,209 @@
using System.Threading.Channels;
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Notifications;
using MeetingAssistant.Summary;
using Microsoft.Extensions.Logging.Abstractions;
using MeetingAssistant.Workflow;
namespace MeetingAssistant.Tests;
public sealed class SummaryAgentRequestTests
{
[Theory]
[InlineData("denied")]
[InlineData("expired")]
[InlineData("shutdown")]
public async Task WorkflowDenialExpirationAndShutdownRemovePromptWithoutStartingAgent(string outcome)
{
var prompts = new ControlledPrompts();
var editor = new RecordingEditor();
var approvals = new AgentApprovalService(prompts, NullLogger<AgentApprovalService>.Instance,
outcome == "expired" ? TimeSpan.FromMilliseconds(100) : TimeSpan.FromMinutes(10));
using var requests = new WorkflowChangeRequestService(approvals, editor, NullLogger<WorkflowChangeRequestService>.Instance);
await requests.StartAsync(CancellationToken.None);
try
{
Assert.StartsWith("Requested", requests.Request("Add association rule", "Given Alpha, then associate Alpha.", "meeting.md"));
var prompt = await prompts.NextAsync();
if (outcome == "denied") prompt.Response.SetResult(false);
if (outcome == "shutdown") await requests.StopAsync(CancellationToken.None);
await prompt.Removed.Task.WaitAsync(TimeSpan.FromSeconds(2));
prompt.Response.TrySetResult(true);
}
finally { await requests.StopAsync(CancellationToken.None); }
Assert.Empty(editor.Prompts);
Assert.StartsWith("Refused", requests.Request("Too late", "Do something", "meeting.md"));
}
[Fact]
public async Task InvalidRequestsAreRefusedBeforeNotification()
{
var fixture = await CreateAsync();
var original = await File.ReadAllTextAsync(fixture.Artifacts.MeetingNotePath);
var prompts = new ControlledPrompts();
var approvals = new AgentApprovalService(prompts, NullLogger<AgentApprovalService>.Instance);
using var requests = new WorkflowChangeRequestService(approvals, new RecordingEditor(), NullLogger<WorkflowChangeRequestService>.Instance);
var tools = new MeetingSummaryTools(fixture.Artifacts, fixture.Options, approvals: approvals, workflowRequests: requests);
Assert.StartsWith("Refused", await tools.RequestProjectAssociation("../Alpha", "Relevant"));
Assert.StartsWith("Refused", await tools.RequestProjectAssociation("Alpha", new string('a', 101)));
Assert.StartsWith("Refused", await tools.RequestProjectAssociation("Alpha", "two\nlines"));
Assert.StartsWith("Refused", await tools.RequestProjectAssociation("Alpha", ""));
Assert.StartsWith("Refused", tools.RequestWorkflowChange(new string('a', 101), "Detailed scenario"));
Assert.StartsWith("Refused", tools.RequestWorkflowChange("Intent", " "));
Assert.StartsWith("Refused", tools.RequestWorkflowChange("two\nlines", "Detailed scenario"));
Assert.False(prompts.HasMore);
Assert.Equal(original, await File.ReadAllTextAsync(fixture.Artifacts.MeetingNotePath));
}
[Fact]
public async Task WorkflowRequestReturnsImmediatelyAndApprovalStartsItsDetailedPromptOnce()
{
var fixture = await CreateAsync();
var prompts = new ControlledPrompts();
var editor = new RecordingEditor();
var approvals = new AgentApprovalService(prompts, NullLogger<AgentApprovalService>.Instance);
using var requests = new WorkflowChangeRequestService(approvals, editor, NullLogger<WorkflowChangeRequestService>.Instance);
await requests.StartAsync(CancellationToken.None);
try
{
const string detail = "Given a meeting with Alpha in its title, when metadata arrives, then add project Alpha.";
var tools = new MeetingSummaryTools(fixture.Artifacts, fixture.Options, workflowRequests: requests);
var result = tools.RequestWorkflowChange("Associate Alpha meetings automatically", detail);
Assert.StartsWith("Requested", result);
var prompt = await prompts.NextAsync();
Assert.Contains("Workflow change requested: Associate Alpha meetings automatically", prompt.Prompt.Title);
Assert.Empty(editor.Prompts);
await tools.WriteSummary("The summary can finish while approval is pending.", "Summary finished");
Assert.True(tools.SummaryWasWritten);
prompt.Response.SetResult(true);
Assert.Equal(detail, await editor.Started.Task.WaitAsync(TimeSpan.FromSeconds(2)));
Assert.False(prompt.Response.TrySetResult(true));
Assert.Single(editor.Prompts);
await prompt.Removed.Task.WaitAsync(TimeSpan.FromSeconds(2));
}
finally { await requests.StopAsync(CancellationToken.None); }
}
[Theory]
[InlineData("denied")]
[InlineData("expired")]
[InlineData("canceled")]
[InlineData("removed")]
public async Task UnapprovedOrStaleAssociationNeverChangesNoteOrProjectAccess(string outcome)
{
var fixture = await CreateAsync();
var original = await File.ReadAllTextAsync(fixture.Artifacts.MeetingNotePath);
var prompts = new ControlledPrompts();
var approvals = new AgentApprovalService(prompts, NullLogger<AgentApprovalService>.Instance,
outcome == "expired" ? TimeSpan.FromMilliseconds(100) : TimeSpan.FromMinutes(10));
var tools = new MeetingSummaryTools(fixture.Artifacts, fixture.Options, approvals: approvals);
using var cancellation = new CancellationTokenSource();
var request = tools.RequestProjectAssociation("Alpha", "Relevant project", cancellation.Token);
var prompt = await prompts.NextAsync();
if (outcome == "denied") prompt.Response.SetResult(false);
if (outcome == "canceled") await cancellation.CancelAsync();
if (outcome == "removed")
{
Directory.Delete(Path.Combine(fixture.Options.Vault.ProjectsFolder, "Alpha"), recursive: true);
prompt.Response.SetResult(true);
}
if (outcome == "canceled")
await Assert.ThrowsAnyAsync<OperationCanceledException>(() => request);
else
Assert.DoesNotContain("Approved", await request.WaitAsync(TimeSpan.FromSeconds(2)));
await prompt.Removed.Task.WaitAsync(TimeSpan.FromSeconds(2));
prompt.Response.TrySetResult(true);
Assert.Equal(original, await File.ReadAllTextAsync(fixture.Artifacts.MeetingNotePath));
Assert.Equal("", await tools.ReadProjectFile("Alpha", "AGENTS.md"));
}
[Fact]
public async Task AssociationWaitsForApprovalThenPreservesLatestNoteAndUnlocksProjectWithInstructions()
{
var fixture = await CreateAsync();
var prompts = new ControlledPrompts();
var approvals = new AgentApprovalService(prompts, NullLogger<AgentApprovalService>.Instance);
var tools = new MeetingSummaryTools(fixture.Artifacts, fixture.Options, approvals: approvals);
var request = tools.RequestProjectAssociation("alpha", "Transcript discusses the Alpha launch");
var prompt = await prompts.NextAsync();
Assert.Contains("Alpha Platform", prompt.Prompt.Message);
Assert.Contains("Transcript discusses", prompt.Prompt.Message);
Assert.Equal(TimeSpan.FromMinutes(10), prompt.Prompt.Timeout);
Assert.False(request.IsCompleted);
Assert.Equal("", await tools.ReadProjectFile("Alpha", "AGENTS.md"));
Assert.StartsWith("Refused:", await tools.WriteProjectFile("Alpha", "notes.md", "unauthorized"));
var latestNote = "---\ntitle: Edited by user\nprojects: Existing\ncustom: keep-me\n---\n\nUser's latest notes.";
await File.WriteAllTextAsync(fixture.Artifacts.MeetingNotePath, latestNote);
prompt.Response.SetResult(true);
var result = await request.WaitAsync(TimeSpan.FromSeconds(2));
Assert.StartsWith("Approved", result);
Assert.Contains("Alpha instructions", result);
Assert.Contains("Alpha Platform", result);
var note = await File.ReadAllTextAsync(fixture.Artifacts.MeetingNotePath);
Assert.Contains("Edited by user", note);
Assert.Contains("custom: keep-me", note);
Assert.Contains("User's latest notes.", note);
Assert.Contains("Existing", note);
Assert.Contains("Alpha", note);
Assert.Empty(Directory.EnumerateFiles(Path.GetDirectoryName(fixture.Artifacts.MeetingNotePath)!, "*.tmp"));
Assert.Equal("Alpha instructions", await tools.ReadProjectFile("Alpha", "AGENTS.md"));
Assert.Equal("Alpha/notes.md", await tools.WriteProjectFile("Alpha", "notes.md", "approved"));
Assert.Contains("notes.md:1 approved", await tools.Search("approved", ["Alpha"]));
Assert.StartsWith("Approved", await tools.RequestProjectAssociation("Alpha", "Already associated"));
Assert.False(prompts.HasMore);
}
private static async Task<(MeetingSessionArtifacts Artifacts, MeetingAssistantOptions Options)> CreateAsync()
{
var root = Path.Combine(Path.GetTempPath(), "meeting-assistant-tests", Guid.NewGuid().ToString("N"));
var project = Path.Combine(root, "Projects", "Alpha");
Directory.CreateDirectory(project);
var artifacts = new MeetingSessionArtifacts(Path.Combine(root, "meeting.md"), Path.Combine(root, "transcript.md"),
Path.Combine(root, "context.md"), Path.Combine(root, "summary.md"));
await File.WriteAllTextAsync(artifacts.MeetingNotePath, "---\ntitle: Planning\nprojects: []\n---\nUser notes.");
await File.WriteAllTextAsync(Path.Combine(project, "PROJECT.md"),
"---\nname: Alpha Platform\ndescription: Meeting automation\n---");
await File.WriteAllTextAsync(Path.Combine(project, "AGENTS.md"), "Alpha instructions");
return (artifacts, new MeetingAssistantOptions { Vault = { ProjectsFolder = Path.Combine(root, "Projects") } });
}
private sealed class RecordingEditor : IWorkflowRulesEditorWindowService
{
public List<string> Prompts { get; } = [];
public TaskCompletionSource<string> Started { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public void Show() { }
public void ShowWithPrompt(string initialPrompt)
{
Prompts.Add(initialPrompt);
Started.TrySetResult(initialPrompt);
}
}
private sealed class ControlledPrompts : IAgentApprovalPromptService
{
private readonly Channel<PendingPrompt> requests = Channel.CreateUnbounded<PendingPrompt>();
public bool HasMore => requests.Reader.TryPeek(out _);
public Task<PendingPrompt> NextAsync() => requests.Reader.ReadAsync().AsTask().WaitAsync(TimeSpan.FromSeconds(2));
public async Task<bool> ConfirmAsync(AgentApprovalPrompt prompt, CancellationToken cancellationToken)
{
var request = new PendingPrompt(prompt, cancellationToken);
requests.Writer.TryWrite(request);
try { return await request.Response.Task.WaitAsync(cancellationToken); }
finally { request.Removed.TrySetResult(); }
}
}
private sealed record PendingPrompt(AgentApprovalPrompt Prompt, CancellationToken Cancellation)
{
public TaskCompletionSource<bool> Response { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
public TaskCompletionSource Removed { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
}
}
+34 -4
View File
@@ -74,7 +74,8 @@ public sealed class TaskbarIconTests
menu,
("Open agent", MeetingTaskbarAction.EditRules, false),
("Finish meeting", MeetingTaskbarAction.StopRecording, true),
("Microphone", MeetingTaskbarAction.OpenSubmenu, true),
("Pause transcription", MeetingTaskbarAction.PauseTranscription, true),
("Microphone", MeetingTaskbarAction.OpenSubmenu, false),
("Cancel meeting recording and discard", MeetingTaskbarAction.AbortRecording, false),
("Switch to english\tCtrl+Alt+L", MeetingTaskbarAction.SwitchProfile, false),
("Exit", MeetingTaskbarAction.Exit, true));
@@ -94,10 +95,32 @@ public sealed class TaskbarIconTests
menu,
("Open agent", MeetingTaskbarAction.EditRules, false),
("Finish meeting", MeetingTaskbarAction.StopRecording, true),
("Cancel meeting recording and discard", MeetingTaskbarAction.AbortRecording, true),
("Pause transcription", MeetingTaskbarAction.PauseTranscription, true),
("Cancel meeting recording and discard", MeetingTaskbarAction.AbortRecording, false),
("Exit", MeetingTaskbarAction.Exit, true));
}
[Fact]
public void TranscriptionPauseActionTracksActivePauseStateWithoutHidingFinish()
{
var runningMenu = MeetingTaskbarMenuBuilder.Build(
Status(isRecording: true, state: RecordingProcessState.Recording, profile: "default"),
[Profile("default")]);
var pausedMenu = MeetingTaskbarMenuBuilder.Build(
Status(isRecording: true, state: RecordingProcessState.Recording, profile: "default", isPaused: true),
[Profile("default")]);
Assert.Contains(runningMenu.Items, item =>
item.Action == MeetingTaskbarAction.PauseTranscription &&
item.Text == "Pause transcription");
Assert.Contains(pausedMenu.Items, item =>
item.Action == MeetingTaskbarAction.UnpauseTranscription &&
item.Text == "Unpause transcription");
Assert.Contains(pausedMenu.Items, item =>
item.Action == MeetingTaskbarAction.StopRecording &&
item.Text == "Finish meeting");
}
[Fact]
public void ProcessingMenuShowsSummarizingButAllowsStartingNewRecordings()
{
@@ -158,6 +181,11 @@ public sealed class TaskbarIconTests
[SupportedOSPlatform("windows")]
public void TaskbarIconGlyphsAreVisuallyCentered()
{
if (!OperatingSystem.IsWindows())
{
return;
}
foreach (var state in new[]
{
RecordingProcessState.Idle,
@@ -200,7 +228,8 @@ public sealed class TaskbarIconTests
private static RecordingStatus Status(
bool isRecording = false,
RecordingProcessState state = RecordingProcessState.Idle,
string? profile = null)
string? profile = null,
bool isPaused = false)
{
return new RecordingStatus(
isRecording,
@@ -209,7 +238,8 @@ public sealed class TaskbarIconTests
state == RecordingProcessState.Idle ? null : "context.md",
state == RecordingProcessState.Idle ? null : "summary.md",
state,
profile);
profile,
isPaused);
}
[SupportedOSPlatform("windows")]
+9 -8
View File
@@ -7,16 +7,16 @@ public sealed class VaultPathTests
[Fact]
public void ResolveUsesVaultBaseFolderForRelativePaths()
{
var baseFolder = Path.Combine(Path.GetTempPath(), "meeting-assistant-vault");
var vault = new VaultOptions
{
BaseFolder = @"C:\Users\masc3\OpenCloud\Persönlich\Vault\Exxeta"
BaseFolder = baseFolder
};
var resolved = VaultPath.Resolve(vault, @"Meetings\Transcripts");
var relativePath = Path.Combine("Meetings", "Transcripts");
var resolved = VaultPath.Resolve(vault, relativePath);
Assert.Equal(
@"C:\Users\masc3\OpenCloud\Persönlich\Vault\Exxeta\Meetings\Transcripts",
resolved);
Assert.Equal(Path.Combine(baseFolder, relativePath), resolved);
}
[Fact]
@@ -24,11 +24,12 @@ public sealed class VaultPathTests
{
var vault = new VaultOptions
{
BaseFolder = @"C:\Users\masc3\OpenCloud\Persönlich\Vault\Exxeta"
BaseFolder = Path.Combine(Path.GetTempPath(), "meeting-assistant-vault")
};
var absolutePath = Path.Combine(Path.GetTempPath(), "meeting-assistant-other");
var resolved = VaultPath.Resolve(vault, @"C:\Other\Path");
var resolved = VaultPath.Resolve(vault, absolutePath);
Assert.Equal(@"C:\Other\Path", resolved);
Assert.Equal(absolutePath, resolved);
}
}
@@ -0,0 +1,251 @@
using MeetingAssistant;
using MeetingAssistant.Speakers;
using MeetingAssistant.Transcription;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using System.Text.Json;
namespace MeetingAssistant.Tests;
public sealed class VenvResemblyzerVoiceEncoderTests
{
[Fact]
public async Task WarmUpAsyncProvisionsVersionedCpuOnlyEnvironmentWithoutDocker()
{
using var fixture = new EncoderFixture();
await fixture.Encoder.WarmUpAsync(CancellationToken.None);
Assert.DoesNotContain(
fixture.Runner.Commands,
command => command.FileName.Contains("docker", StringComparison.OrdinalIgnoreCase));
Assert.Contains(
fixture.Runner.Commands,
command => command.FileName == "python-test"
&& command.Arguments.Take(2).SequenceEqual(["-m", "venv"]));
Assert.Contains(
fixture.Runner.Commands,
command => command.Arguments.Contains("torch==2.14.0+cpu")
&& command.Arguments.Contains("https://download.pytorch.org/whl/cpu"));
Assert.Contains(
fixture.Runner.Commands,
command => command.Arguments.Contains("webrtcvad-wheels==2.0.14"));
Assert.Contains(
fixture.Runner.Commands,
command => command.Arguments.Contains("Resemblyzer==0.1.4")
&& command.Arguments.Contains("--no-deps"));
}
[Fact]
public async Task WarmUpAsyncDoesNotMarkFailedEnvironmentReady()
{
using var fixture = new EncoderFixture(
warmupExitCode: 21,
warmupError: "incompatible environment");
await Assert.ThrowsAsync<InvalidOperationException>(
() => fixture.Encoder.WarmUpAsync(CancellationToken.None));
Assert.Empty(Directory.EnumerateFiles(
fixture.RuntimeFolder,
".ready",
SearchOption.AllDirectories));
}
[Fact]
public async Task EncodeAsyncRunsBatchWithManagedEnvironmentPython()
{
var first = new float[256];
first[0] = 2;
var second = new float[256];
second[1] = 3;
using var fixture = new EncoderFixture(VectorOutput([first, second]));
var vectors = await fixture.Encoder.EncodeAsync([[1, 2, 3], [4, 5, 6]], CancellationToken.None);
Assert.Equal(2, vectors.Count);
Assert.Equal(1f, vectors[0][0], 5);
Assert.Equal(1f, vectors[1][1], 5);
var encodingCommand = Assert.Single(
fixture.Runner.Commands,
command => command.Arguments.Any(
argument => argument.EndsWith("encode.py", StringComparison.Ordinal)));
Assert.EndsWith(
OperatingSystem.IsWindows()
? Path.Combine("Scripts", "python.exe")
: Path.Combine("bin", "python"),
encodingCommand.FileName,
StringComparison.OrdinalIgnoreCase);
Assert.Equal(2, encodingCommand.Arguments.Count);
Assert.True(Path.IsPathFullyQualified(encodingCommand.Arguments[1]));
}
[Fact]
public async Task EncodeAsyncRejectsWrongVectorDimension()
{
var vector = new float[255];
vector[0] = 1;
using var fixture = new EncoderFixture(VectorOutput([vector]));
var error = await Assert.ThrowsAsync<InvalidDataException>(
() => fixture.Encoder.EncodeAsync([[1]], CancellationToken.None));
Assert.Contains("255 dimensions", error.Message);
}
[Fact]
public async Task EncodeAsyncRejectsMalformedVectorJson()
{
using var fixture = new EncoderFixture(
"__MEETING_ASSISTANT_RESEMBLYZER_JSON_START__\n[not-json]\n__MEETING_ASSISTANT_RESEMBLYZER_JSON_END__");
await Assert.ThrowsAsync<InvalidDataException>(
() => fixture.Encoder.EncodeAsync([[1]], CancellationToken.None));
}
[Fact]
public async Task EncodeAsyncRejectsNonFiniteVector()
{
var values = string.Join(',', new[] { "NaN" }.Concat(Enumerable.Repeat("0", 255)));
using var fixture = new EncoderFixture(
$"__MEETING_ASSISTANT_RESEMBLYZER_JSON_START__\n[[{values}]]\n__MEETING_ASSISTANT_RESEMBLYZER_JSON_END__");
await Assert.ThrowsAsync<InvalidDataException>(
() => fixture.Encoder.EncodeAsync([[1]], CancellationToken.None));
}
[Fact]
public async Task EncodeAsyncRejectsWrongResultCount()
{
var vector = new float[256];
vector[0] = 1;
using var fixture = new EncoderFixture(VectorOutput([vector, vector]));
var error = await Assert.ThrowsAsync<InvalidDataException>(
() => fixture.Encoder.EncodeAsync([[1]], CancellationToken.None));
Assert.Contains("2 vectors for 1 WAV samples", error.Message);
}
[Fact]
public async Task EncodeAsyncRejectsZeroMagnitudeVector()
{
using var fixture = new EncoderFixture(VectorOutput([new float[256]]));
var error = await Assert.ThrowsAsync<InvalidDataException>(
() => fixture.Encoder.EncodeAsync([[1]], CancellationToken.None));
Assert.Contains("zero magnitude", error.Message);
}
[Fact]
public async Task EncodeAsyncReportsFailedLocalCommand()
{
using var fixture = new EncoderFixture("", 17, "runtime failed");
var error = await Assert.ThrowsAsync<InvalidOperationException>(
() => fixture.Encoder.EncodeAsync([[1]], CancellationToken.None));
Assert.Contains("exit code 17", error.Message);
Assert.Contains("runtime failed", error.Message);
}
private static string VectorOutput(IReadOnlyList<float[]> vectors)
{
return $"""
__MEETING_ASSISTANT_RESEMBLYZER_JSON_START__
{JsonSerializer.Serialize(vectors)}
__MEETING_ASSISTANT_RESEMBLYZER_JSON_END__
""";
}
private sealed class EncoderFixture : IDisposable
{
public EncoderFixture(
string encodingOutput = "",
int encodingExitCode = 0,
string encodingError = "",
int warmupExitCode = 0,
string warmupError = "")
{
RuntimeFolder = Path.Combine(
Path.GetTempPath(),
"meeting-assistant-tests",
Guid.NewGuid().ToString("N"),
"resemblyzer");
Runner = new CapturingCommandRunner(
encodingOutput,
encodingExitCode,
encodingError,
warmupExitCode,
warmupError);
var options = new MeetingAssistantOptions();
options.SpeakerIdentification.Resemblyzer.PythonCommand = "python-test";
options.SpeakerIdentification.Resemblyzer.RuntimeFolder = RuntimeFolder;
Encoder = new VenvResemblyzerVoiceEncoder(
Runner,
Options.Create(options),
NullLogger<VenvResemblyzerVoiceEncoder>.Instance);
}
public CapturingCommandRunner Runner { get; }
public VenvResemblyzerVoiceEncoder Encoder { get; }
public string RuntimeFolder { get; }
public void Dispose()
{
if (Directory.Exists(RuntimeFolder))
{
Directory.Delete(RuntimeFolder, recursive: true);
}
}
}
private sealed class CapturingCommandRunner : ICommandRunner
{
private readonly string encodingOutput;
private readonly int encodingExitCode;
private readonly string encodingError;
private readonly int warmupExitCode;
private readonly string warmupError;
public CapturingCommandRunner(
string encodingOutput = "",
int encodingExitCode = 0,
string encodingError = "",
int warmupExitCode = 0,
string warmupError = "")
{
this.encodingOutput = encodingOutput;
this.encodingExitCode = encodingExitCode;
this.encodingError = encodingError;
this.warmupExitCode = warmupExitCode;
this.warmupError = warmupError;
}
public List<CapturedCommand> Commands { get; } = [];
public Task<CommandResult> RunAsync(
string fileName,
IReadOnlyList<string> arguments,
CancellationToken cancellationToken,
IReadOnlyDictionary<string, string>? environment = null)
{
Commands.Add(new CapturedCommand(fileName, arguments.ToList()));
var isEncoding = arguments.Any(argument => argument.EndsWith("encode.py", StringComparison.Ordinal));
if (isEncoding)
{
return Task.FromResult(new CommandResult(encodingExitCode, encodingOutput, encodingError));
}
var isWarmup = arguments.Any(argument => argument.Contains("VoiceEncoder", StringComparison.Ordinal));
return Task.FromResult(isWarmup
? new CommandResult(warmupExitCode, string.Empty, warmupError)
: new CommandResult(0, string.Empty, string.Empty));
}
}
private sealed record CapturedCommand(string FileName, IReadOnlyList<string> Arguments);
}
@@ -864,6 +864,32 @@ public sealed class WorkflowRulesEditorTests
Assert.Empty(await fixture.Context.SpeakerIdentities.ToListAsync());
}
[Fact]
public async Task RulesEditorToolsExposeVoiceVectorCountsSeparatelyFromWavSamples()
{
await using var fixture = await WorkflowRulesEditorIdentityFixture.CreateAsync();
var identity = await fixture.AddIdentityAsync("Sabrina", sample: [1, 2, 3]);
fixture.Context.SpeakerVoiceVectors.Add(new SpeakerVoiceVector
{
SpeakerIdentityId = identity.Id,
ModelId = "resemblyzer-0.1.4-pretrained",
Dimensions = 256,
VectorBytes = new byte[256 * sizeof(float)],
Fingerprint = "vector-1",
CreatedAt = DateTimeOffset.UtcNow
});
await fixture.Context.SaveChangesAsync();
var tools = fixture.CreateTools();
var searchResult = await tools.SearchIdentities("Sabrina");
var readResult = await tools.ReadIdentity(identity.Id);
Assert.Contains("\"sampleCount\": 1", searchResult);
Assert.Contains("\"voiceVectorCount\": 1", searchResult);
Assert.Contains("\"voiceVectorCount\": 1", readResult);
Assert.DoesNotContain("vector-1", readResult);
}
[Fact]
public async Task RulesEditorToolsRefusesSamplelessSpeakerIdentityCreation()
{
@@ -872,7 +898,7 @@ public sealed class WorkflowRulesEditorTests
var result = await tools.CreateIdentity("Sabrina", ["Sabi"], ["Guest-01"]);
Assert.StartsWith("Refused: speaker identities require at least one audio sample.", result);
Assert.StartsWith("Refused: speaker identities require audio evidence", result);
Assert.Empty(await fixture.Context.SpeakerIdentities.ToListAsync());
}
@@ -969,6 +995,30 @@ public sealed class WorkflowRulesEditorTests
AssertCompletedActivity(viewModel, ["Thinking..."], "Updated rules.");
}
[Fact]
public async Task ApprovedWorkflowPromptStartsImmediatelyInItsOwnConversation()
{
var existing = new WorkflowRulesEditorChatViewModel(new BlockingRulesEditorPipeline("Other work"))
{
Draft = "My unfinished draft"
};
var pipeline = new BlockingRulesEditorPipeline("Workflow updated.");
var requested = new WorkflowRulesEditorChatViewModel(pipeline);
const string prompt = "Given Alpha in the meeting title, when metadata arrives, then associate project Alpha.";
var task = requested.SendPromptAsync(prompt);
await pipeline.Started.Task.WaitAsync(TimeSpan.FromSeconds(2));
Assert.True(requested.IsThinking);
Assert.Empty(pipeline.LastConversation);
Assert.Equal(new WorkflowRulesEditorChatMessage(WorkflowRulesEditorChatRole.User, prompt), requested.Messages[0].Message);
Assert.Equal("My unfinished draft", existing.Draft);
Assert.Empty(existing.Messages);
pipeline.Release.SetResult();
await task.WaitAsync(TimeSpan.FromSeconds(2));
Assert.False(requested.IsThinking);
}
[Fact]
public async Task ViewModelShowsAgentErrorWhenRequestTimesOut()
{
@@ -32,7 +32,14 @@ Calendar-Feature is refined, Golem feature needs more work.
PROJECT.md should use this structure:
Maintain its YAML `name` as the project's display name and `description` as a grounded description of its purpose in at most 256 characters. These fields help summarizers identify missing project associations. Preserve other frontmatter and body content when updating them.
```markdown
---
name: Project display name
description: Concise project purpose and distinguishing context.
---
# Executive Summary
<Elevator Pitch>
@@ -1,4 +1,5 @@
using Microsoft.Extensions.Configuration;
using MeetingAssistant.Speakers;
namespace MeetingAssistant.LaunchProfiles;
@@ -48,6 +49,7 @@ public sealed class ConfigurationLaunchProfileOptionsProvider : ILaunchProfileOp
var options = BindDefaultOptions();
if (profileName.Equals(DefaultProfileName, StringComparison.OrdinalIgnoreCase))
{
ValidateSpeakerSampleDurations(options, DefaultProfileName);
return new LaunchProfile(DefaultProfileName, options);
}
@@ -59,6 +61,7 @@ public sealed class ConfigurationLaunchProfileOptionsProvider : ILaunchProfileOp
profileSection.Bind(options);
ApplyArrayOverrides(profileSection, options);
ValidateSpeakerSampleDurations(options, profileName);
return new LaunchProfile(profileName, options);
}
@@ -149,6 +152,18 @@ public sealed class ConfigurationLaunchProfileOptionsProvider : ILaunchProfileOp
: name.Trim();
}
private static void ValidateSpeakerSampleDurations(
MeetingAssistantOptions options,
string profileName)
{
SpeakerSampleDurationConfiguration.ValidateOrThrow(
options.SpeakerIdentification,
$"Launch profile '{profileName}' has invalid speaker sample durations");
ResemblyzerSpeakerRecognitionConfiguration.ValidateOrThrow(
options.SpeakerIdentification.Resemblyzer,
$"Launch profile '{profileName}' has invalid Resemblyzer speaker-recognition settings");
}
private static void ApplyArrayOverrides(
IConfigurationSection profileSection,
MeetingAssistantOptions options)
@@ -28,6 +28,12 @@ public sealed class MacOsWorkflowRulesEditorWindowService : IWorkflowRulesEditor
UseShellExecute = false
});
}
public void ShowWithPrompt(string initialPrompt)
{
throw new PlatformNotSupportedException(
"Automatic workflow-change conversations require the Windows approval UI. Use the interactive macOS agent instead.");
}
}
public sealed class MacOsWorkflowRulesEditorSession
+42 -10
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<PreserveCompilationContext>true</PreserveCompilationContext>
<MicrosoftSpeechVersion>1.51.1</MicrosoftSpeechVersion>
<MicrosoftSpeechVersion>1.51.2</MicrosoftSpeechVersion>
</PropertyGroup>
<PropertyGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'windows'">
@@ -18,7 +18,9 @@
<PropertyGroup Condition="$([MSBuild]::IsOSPlatform('OSX')) and $([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) != 'windows'">
<MacOsNativeHelpersEnabled>true</MacOsNativeHelpersEnabled>
<MacOsAudioCaptureSource>$(MSBuildProjectDirectory)/Native/MacOsMeetingAudioCapture/main.swift</MacOsAudioCaptureSource>
<MacOsAudioCaptureOutputPath>Native/macos-meeting-audio-capture</MacOsAudioCaptureOutputPath>
<MacOsAudioCaptureInfoPlist>$(MSBuildProjectDirectory)/Native/MacOsMeetingAudioCapture/Info.plist</MacOsAudioCaptureInfoPlist>
<MacOsAudioCaptureBundlePath>Native/MeetingAssistantAudioCapture.app</MacOsAudioCaptureBundlePath>
<MacOsAudioCaptureOutputPath>$(MacOsAudioCaptureBundlePath)/Contents/MacOS/macos-meeting-audio-capture</MacOsAudioCaptureOutputPath>
<MacOsAudioCaptureRid Condition="'$(RuntimeIdentifier)' != ''">$(RuntimeIdentifier)</MacOsAudioCaptureRid>
<MacOsAudioCaptureRid Condition="'$(MacOsAudioCaptureRid)' == ''">$(NETCoreSdkRuntimeIdentifier)</MacOsAudioCaptureRid>
<MacOsAudioCaptureArchitecture Condition="'$(MacOsAudioCaptureRid)' == 'osx-x64'">x86_64</MacOsAudioCaptureArchitecture>
@@ -28,6 +30,10 @@
<MacOsMeetingIntegrationsSource>$(MSBuildProjectDirectory)/Native/MacOsMeetingIntegrations/main.swift</MacOsMeetingIntegrationsSource>
<MacOsMeetingIntegrationsInfoPlist>$(MSBuildProjectDirectory)/Native/MacOsMeetingIntegrations/Info.plist</MacOsMeetingIntegrationsInfoPlist>
<MacOsMeetingIntegrationsOutputPath>Native/macos-meeting-integrations</MacOsMeetingIntegrationsOutputPath>
<MacOsLauncherSource>$(MSBuildProjectDirectory)/Native/MacOsLauncher/main.swift</MacOsLauncherSource>
<MacOsLauncherInfoPlist>$(MSBuildProjectDirectory)/Native/MacOsLauncher/Info.plist</MacOsLauncherInfoPlist>
<MacOsLauncherOutputPath>Native/macos-meeting-assistant-launcher</MacOsLauncherOutputPath>
<MacOsApplicationBundlePath>MeetingAssistant.app</MacOsApplicationBundlePath>
</PropertyGroup>
<ItemGroup>
@@ -35,16 +41,16 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.17.0" />
<PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.21.0" />
<PackageReference Include="DiffPlex" Version="1.9.0" />
<PackageReference Include="Microsoft.CognitiveServices.Speech" Version="$(MicrosoftSpeechVersion)" />
<PackageReference Include="Microsoft.CognitiveServices.Speech.Extension.MAS" Version="$(MicrosoftSpeechVersion)" ExcludeAssets="build" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.10" />
<PackageReference Include="NAudio" Version="2.3.0" />
<PackageReference Include="NCalcSync" Version="7.0.2" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.12" />
<PackageReference Include="NAudio" Version="3.1.0" />
<PackageReference Include="NCalcSync" Version="7.2.0" />
<PackageReference Include="RazorLight" Version="2.3.1" />
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3" Version="3.0.5" />
<PackageReference Include="System.Drawing.Common" Version="10.0.10" />
<PackageReference Include="System.Drawing.Common" Version="10.0.12" />
<PackageReference Include="Whisper.net" Version="1.9.1" />
<PackageReference Include="Whisper.net.Runtime" Version="1.9.1" />
<PackageReference Include="YamlDotNet" Version="18.1.0" />
@@ -52,7 +58,7 @@
<ItemGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'windows'">
<PackageReference Include="CommunityToolkit.WinUI.Notifications" Version="7.1.2" />
<PackageReference Include="H.NotifyIcon.Uno.WinUI" Version="2.4.1" />
<PackageReference Include="H.NotifyIcon" Version="2.4.1" />
</ItemGroup>
<ItemGroup>
@@ -82,9 +88,32 @@
AfterTargets="Build"
Condition="'$(MacOsNativeHelpersEnabled)' == 'true'">
<MakeDir Directories="$(TargetDir)Native" />
<Exec Command="/usr/bin/xcrun swiftc -parse-as-library -O -target $(MacOsAudioCaptureArchitecture)-apple-macos13.0 -framework AVFoundation -framework CoreMedia -framework ScreenCaptureKit &quot;$(MacOsAudioCaptureSource)&quot; -o &quot;$(TargetDir)$(MacOsAudioCaptureOutputPath)&quot;" />
<MakeDir Directories="$(TargetDir)$(MacOsAudioCaptureBundlePath)/Contents/MacOS" />
<Copy SourceFiles="$(MacOsAudioCaptureInfoPlist)" DestinationFiles="$(TargetDir)$(MacOsAudioCaptureBundlePath)/Contents/Info.plist" />
<Exec Command="/usr/bin/xcrun swiftc -parse-as-library -O -target $(MacOsAudioCaptureArchitecture)-apple-macos13.0 -framework AVFoundation -framework CoreMedia -framework ScreenCaptureKit -Xlinker -sectcreate -Xlinker __TEXT -Xlinker __info_plist -Xlinker &quot;$(MacOsAudioCaptureInfoPlist)&quot; &quot;$(MacOsAudioCaptureSource)&quot; -o &quot;$(TargetDir)$(MacOsAudioCaptureOutputPath)&quot;" />
<Exec Command="/usr/bin/codesign --force --deep --sign - &quot;$(TargetDir)$(MacOsAudioCaptureBundlePath)&quot;" />
<Exec Command="/usr/bin/xcrun swiftc -parse-as-library -O -target $(MacOsAudioCaptureArchitecture)-apple-macos13.0 -framework AppKit -framework Carbon -framework WebKit &quot;$(MacOsDesktopControlsSource)&quot; -o &quot;$(TargetDir)$(MacOsDesktopControlsOutputPath)&quot;" />
<Exec Command="/usr/bin/xcrun swiftc -parse-as-library -O -target $(MacOsAudioCaptureArchitecture)-apple-macos13.0 -framework AppKit -framework CoreGraphics -framework EventKit -framework ImageIO -framework UniformTypeIdentifiers -Xlinker -sectcreate -Xlinker __TEXT -Xlinker __info_plist -Xlinker &quot;$(MacOsMeetingIntegrationsInfoPlist)&quot; &quot;$(MacOsMeetingIntegrationsSource)&quot; -o &quot;$(TargetDir)$(MacOsMeetingIntegrationsOutputPath)&quot;" />
<Exec Command="/usr/bin/xcrun swiftc -parse-as-library -O -target $(MacOsAudioCaptureArchitecture)-apple-macos13.0 &quot;$(MacOsLauncherSource)&quot; -o &quot;$(TargetDir)$(MacOsLauncherOutputPath)&quot;" />
</Target>
<Target
Name="PackageMacOsApplication"
AfterTargets="Publish"
DependsOnTargets="CopyMeetingAssistantConfigToPublishRoot;CopyMacOsNativeHelpersToPublish"
Condition="'$(MacOsNativeHelpersEnabled)' == 'true'">
<RemoveDir Directories="$(PublishDir)$(MacOsApplicationBundlePath)" />
<MakeDir Directories="$(PublishDir)$(MacOsApplicationBundlePath)/Contents/MacOS" />
<MakeDir Directories="$(PublishDir)$(MacOsApplicationBundlePath)/Contents/Resources/app" />
<ItemGroup>
<MacOsPublishedRuntimeFiles Include="$(PublishDir)**/*" Exclude="$(PublishDir)$(MacOsApplicationBundlePath)/**/*" />
</ItemGroup>
<Copy
SourceFiles="@(MacOsPublishedRuntimeFiles)"
DestinationFiles="@(MacOsPublishedRuntimeFiles->'$(PublishDir)$(MacOsApplicationBundlePath)/Contents/Resources/app/%(RecursiveDir)%(Filename)%(Extension)')" />
<Copy SourceFiles="$(TargetDir)$(MacOsLauncherOutputPath)" DestinationFiles="$(PublishDir)$(MacOsApplicationBundlePath)/Contents/MacOS/MeetingAssistant" />
<Copy SourceFiles="$(MacOsLauncherInfoPlist)" DestinationFiles="$(PublishDir)$(MacOsApplicationBundlePath)/Contents/Info.plist" />
<Exec Command="/usr/bin/codesign --force --deep --sign - --requirements '=designated =&gt; identifier &quot;cloud.schweigert.meeting-assistant&quot;' &quot;$(PublishDir)$(MacOsApplicationBundlePath)&quot;" />
</Target>
<Target
@@ -92,7 +121,10 @@
AfterTargets="Publish"
Condition="'$(MacOsNativeHelpersEnabled)' == 'true'">
<MakeDir Directories="$(PublishDir)Native" />
<Copy SourceFiles="$(TargetDir)$(MacOsAudioCaptureOutputPath)" DestinationFolder="$(PublishDir)Native" />
<MakeDir Directories="$(PublishDir)$(MacOsAudioCaptureBundlePath)/Contents/MacOS" />
<Copy SourceFiles="$(TargetDir)$(MacOsAudioCaptureOutputPath)" DestinationFiles="$(PublishDir)$(MacOsAudioCaptureOutputPath)" />
<Copy SourceFiles="$(MacOsAudioCaptureInfoPlist)" DestinationFiles="$(PublishDir)$(MacOsAudioCaptureBundlePath)/Contents/Info.plist" />
<Exec Command="/usr/bin/codesign --force --deep --sign - &quot;$(PublishDir)$(MacOsAudioCaptureBundlePath)&quot;" />
<Copy SourceFiles="$(TargetDir)$(MacOsDesktopControlsOutputPath)" DestinationFolder="$(PublishDir)Native" />
<Copy SourceFiles="$(TargetDir)$(MacOsMeetingIntegrationsOutputPath)" DestinationFolder="$(PublishDir)Native" />
</Target>
+52 -7
View File
@@ -165,6 +165,8 @@ public sealed class RecordingInactivitySafeguardOptions
public TimeSpan AutoStopAfter { get; set; } = TimeSpan.FromMinutes(30);
public TimeSpan MaximumPauseDuration { get; set; } = TimeSpan.FromHours(4);
public TimeSpan InferredEndPadding { get; set; } = TimeSpan.FromMinutes(1);
public TimeSpan CheckInterval { get; set; } = TimeSpan.FromSeconds(15);
@@ -181,10 +183,8 @@ public sealed class WhisperLocalOptions
public PyannoteDiarizationOptions Diarization { get; set; } = new();
}
public sealed class PyannoteDiarizationOptions
public class PyannoteRuntimeOptions
{
public bool Enabled { get; set; } = true;
public string DockerCommand { get; set; } = "docker";
public string BaseImage { get; set; } = "python:3.11-slim";
@@ -208,6 +208,11 @@ public sealed class PyannoteDiarizationOptions
public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromMinutes(30);
}
public sealed class PyannoteDiarizationOptions : PyannoteRuntimeOptions
{
public bool Enabled { get; set; } = true;
}
public sealed class AzureSpeechOptions
{
public string Endpoint { get; set; } = "";
@@ -340,9 +345,9 @@ public sealed class SpeakerIdentificationOptions
public int MaxSnippetsPerSpeaker { get; set; } = 3;
public TimeSpan MinimumSampleSpeechDuration { get; set; } = TimeSpan.FromSeconds(30);
public TimeSpan MinimumSampleSpeechDuration { get; set; } = TimeSpan.FromSeconds(10);
public TimeSpan MaximumSampleSegmentGap { get; set; } = TimeSpan.FromSeconds(1);
public TimeSpan MaximumSampleDuration { get; set; } = TimeSpan.FromSeconds(60);
public double SilenceBetweenSnippetsSeconds { get; set; } = 1;
@@ -355,6 +360,47 @@ public sealed class SpeakerIdentificationOptions
public AzureSpeechOptions AzureSpeech { get; set; } = new();
public SpeakerIdentityPyannoteValidationOptions PyannoteValidation { get; set; } = new();
public ResemblyzerSpeakerRecognitionOptions Resemblyzer { get; set; } = new();
}
public sealed class ResemblyzerSpeakerRecognitionOptions
{
public bool Enabled { get; set; }
public int RequiredVectorsPerSpeaker { get; set; } = 5;
public int MaxVectorsPerIdentity { get; set; } = 1000;
public int OutlierPruningMinimumVectors { get; set; } = 20;
public double OutlierPruningNeighborSimilarity { get; set; } = 0.75;
public int OutlierPruningMinimumNeighbors { get; set; } = 3;
public double OutlierPruningMinimumClusterRatio { get; set; } = 0.60;
public double MinimumClusterCohesion { get; set; } = 0.75;
public double MinimumIdentitySimilarity { get; set; } = 0.75;
public double MinimumSimilarityMargin { get; set; } = 0.05;
public string ModelId { get; set; } = "resemblyzer-0.1.4-pretrained";
public string PackageVersion { get; set; } = "0.1.4";
public string PythonCommand { get; set; } = "python";
public string TorchVersion { get; set; } = "2.14.0+cpu";
public string TorchIndexUrl { get; set; } = "https://download.pytorch.org/whl/cpu";
public string WebRtcVadVersion { get; set; } = "2.0.14";
public string RuntimeFolder { get; set; } = @"%LOCALAPPDATA%\MeetingAssistant\Resemblyzer";
public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromMinutes(15);
}
public sealed class SpeakerIdentityPyannoteValidationOptions
@@ -365,9 +411,8 @@ public sealed class SpeakerIdentityPyannoteValidationOptions
public double MinimumMatchingKnownSnippetRatio { get; set; } = 0.50;
public PyannoteDiarizationOptions Diarization { get; set; } = new()
public PyannoteRuntimeOptions Diarization { get; set; } = new()
{
Enabled = true,
CommandTimeout = TimeSpan.FromHours(1),
AlignmentMode = PyannoteAlignmentMode.PyannoteTurns
};
@@ -0,0 +1,24 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleExecutable</key>
<string>MeetingAssistant</string>
<key>CFBundleIdentifier</key>
<string>cloud.schweigert.meeting-assistant</string>
<key>CFBundleName</key>
<string>Meeting Assistant</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>1.0</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>LSUIElement</key>
<true/>
<key>NSCalendarsFullAccessUsageDescription</key>
<string>Meeting Assistant reads calendar metadata for meeting notes and recording prompts.</string>
<key>NSMicrophoneUsageDescription</key>
<string>Meeting Assistant records microphone audio for live meeting transcription.</string>
</dict>
</plist>
@@ -0,0 +1,70 @@
import Darwin
import Dispatch
import Foundation
@main
private struct MeetingAssistantLauncher {
private static let serviceDirectoryName = "MeetingAssistant"
static func main() {
let runtimeDirectory = Bundle.main.bundleURL
.appendingPathComponent("Contents/Resources/app", isDirectory: true)
let assemblyPath = runtimeDirectory.appendingPathComponent("MeetingAssistant.dll").path
let environmentFile = FileManager.default.homeDirectoryForCurrentUser
.appendingPathComponent("Library/Application Support")
.appendingPathComponent(serviceDirectoryName)
.appendingPathComponent("config/meeting-assistant.env")
guard FileManager.default.fileExists(atPath: assemblyPath) else {
fail("Meeting Assistant runtime not found at \(assemblyPath)")
}
guard FileManager.default.fileExists(atPath: environmentFile.path) else {
fail("Meeting Assistant environment file not found at \(environmentFile.path)")
}
let process = Process()
process.executableURL = URL(fileURLWithPath: "/bin/zsh")
process.arguments = [
"-c",
"set -a; source \"$1\"; set +a; exec /usr/local/share/dotnet/dotnet \"$2\" --contentRoot \"$3\"",
"meeting-assistant-launcher",
environmentFile.path,
assemblyPath,
runtimeDirectory.path
]
process.currentDirectoryURL = runtimeDirectory
signal(SIGTERM, SIG_IGN)
signal(SIGINT, SIG_IGN)
let terminationSignal = DispatchSource.makeSignalSource(signal: SIGTERM, queue: .global())
terminationSignal.setEventHandler {
if process.isRunning {
process.terminate()
}
}
terminationSignal.resume()
let interruptSignal = DispatchSource.makeSignalSource(signal: SIGINT, queue: .global())
interruptSignal.setEventHandler {
if process.isRunning {
kill(process.processIdentifier, SIGINT)
}
}
interruptSignal.resume()
do {
try process.run()
process.waitUntilExit()
Darwin.exit(process.terminationStatus)
} catch {
fail("Meeting Assistant launcher failed: \(error)")
}
}
private static func fail(_ message: String) -> Never {
FileHandle.standardError.write(Data("\(message)\n".utf8))
Darwin.exit(1)
}
}
@@ -0,0 +1,22 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleExecutable</key>
<string>macos-meeting-audio-capture</string>
<key>CFBundleIdentifier</key>
<string>cloud.schweigert.meeting-assistant.audio-capture</string>
<key>CFBundleName</key>
<string>Meeting Assistant Audio Capture</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>1.0</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>LSUIElement</key>
<true/>
<key>NSMicrophoneUsageDescription</key>
<string>Meeting Assistant records microphone audio for live meeting transcription.</string>
</dict>
</plist>
@@ -0,0 +1,56 @@
namespace MeetingAssistant.Notifications;
public sealed record AgentApprovalPrompt(string Title, string Message, TimeSpan Timeout);
public interface IAgentApprovalPromptService
{
Task<bool> ConfirmAsync(AgentApprovalPrompt prompt, CancellationToken cancellationToken);
}
public sealed class NoopAgentApprovalPromptService : IAgentApprovalPromptService
{
public Task<bool> ConfirmAsync(AgentApprovalPrompt prompt, CancellationToken cancellationToken) => Task.FromResult(false);
}
public sealed class AgentApprovalService
{
private readonly IAgentApprovalPromptService prompts;
private readonly ILogger<AgentApprovalService> logger;
private readonly TimeSpan timeout;
public AgentApprovalService(IAgentApprovalPromptService prompts, ILogger<AgentApprovalService> logger)
: this(prompts, logger, TimeSpan.FromMinutes(10)) { }
internal AgentApprovalService(IAgentApprovalPromptService prompts, ILogger<AgentApprovalService> logger, TimeSpan timeout)
{
this.prompts = prompts;
this.logger = logger;
this.timeout = timeout;
}
public async Task<bool> ConfirmAsync(string title, string message, CancellationToken cancellationToken)
{
using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
deadline.CancelAfter(timeout);
try
{
var accepted = await prompts.ConfirmAsync(new AgentApprovalPrompt(title, message, timeout), deadline.Token)
.WaitAsync(deadline.Token);
cancellationToken.ThrowIfCancellationRequested();
return accepted && !deadline.IsCancellationRequested;
}
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
return false;
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
logger.LogWarning(exception, "Could not obtain user approval for {Title}", title);
return false;
}
finally
{
await deadline.CancelAsync();
}
}
}
@@ -0,0 +1,85 @@
#if WINDOWS
using System.Collections.Concurrent;
using CommunityToolkit.WinUI.Notifications;
using MeetingAssistant.Recording;
namespace MeetingAssistant.Notifications;
public sealed class WindowsAgentApprovalPromptService(ILogger<WindowsAgentApprovalPromptService> logger)
: IAgentApprovalPromptService, IDisposable
{
private const string Source = "summary-agent-approval";
private readonly ConcurrentDictionary<string, TaskCompletionSource<bool>> pending = new();
private readonly object gate = new();
private bool registered;
private bool disposed;
public async Task<bool> ConfirmAsync(AgentApprovalPrompt prompt, CancellationToken cancellationToken)
{
if (!OperatingSystem.IsWindowsVersionAtLeast(10, 0, 17763)) return false;
cancellationToken.ThrowIfCancellationRequested();
var id = Guid.NewGuid().ToString("N");
var response = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
try
{
lock (gate)
{
if (disposed) return false;
if (!registered)
{
ToastNotificationManagerCompat.OnActivated += OnActivated;
registered = true;
}
pending[id] = response;
new ToastContentBuilder()
.AddArgument("source", Source).AddArgument("requestId", id)
.SetToastScenario(ToastScenario.Reminder)
.SetToastDuration(ToastDuration.Long)
.AddText(prompt.Title)
.AddText(prompt.Message)
.AddButton(new ToastButton().SetContent("Yes")
.AddArgument("source", Source).AddArgument("requestId", id)
.AddArgument("response", "yes").SetBackgroundActivation())
.AddButton(new ToastButton().SetContent("No")
.AddArgument("source", Source).AddArgument("requestId", id)
.AddArgument("response", "no").SetBackgroundActivation())
.Show(toast =>
{
toast.Tag = id;
toast.Group = Source;
toast.ExpirationTime = DateTimeOffset.Now.Add(prompt.Timeout);
});
}
return await response.Task.WaitAsync(cancellationToken);
}
finally
{
pending.TryRemove(id, out _);
try { ToastNotificationManagerCompat.History.Remove(id, Source); }
catch (Exception exception) { logger.LogDebug(exception, "Could not remove approval notification {RequestId}", id); }
}
}
private void OnActivated(ToastNotificationActivatedEventArgsCompat args)
{
var arguments = NotificationActivationArguments.Parse(args.Argument);
if (arguments.GetValueOrDefault("source") != Source ||
!arguments.TryGetValue("requestId", out var id) ||
!pending.TryRemove(id, out var response)) return;
response.TrySetResult(arguments.GetValueOrDefault("response") == "yes");
}
public void Dispose()
{
lock (gate)
{
disposed = true;
if (registered) ToastNotificationManagerCompat.OnActivated -= OnActivated;
registered = false;
foreach (var response in pending.Values) response.TrySetResult(false);
pending.Clear();
}
}
}
#endif
+47 -3
View File
@@ -5,6 +5,7 @@ using MeetingAssistant.LaunchProfiles;
using MeetingAssistant.Logging;
using MeetingAssistant.MacOs;
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Notifications;
using MeetingAssistant.Recording;
using MeetingAssistant.Screenshots;
using MeetingAssistant.Speakers;
@@ -17,7 +18,10 @@ using Microsoft.Extensions.Options;
var builder = WebApplication.CreateBuilder(args);
builder.Logging.AddProvider(new MeetingAssistantFileLoggerProvider());
builder.Services.Configure<MeetingAssistantOptions>(builder.Configuration.GetSection("MeetingAssistant"));
builder.Services.AddSingleton<IValidateOptions<MeetingAssistantOptions>, MeetingAssistantSpeakerSampleOptionsValidator>();
builder.Services.AddOptions<MeetingAssistantOptions>()
.Bind(builder.Configuration.GetSection("MeetingAssistant"))
.ValidateOnStart();
builder.Services.AddSingleton(HostOperatingSystem.DetectCurrent());
builder.Services.AddSingleton<ILaunchProfileOptionsProvider, ConfigurationLaunchProfileOptionsProvider>();
#if WINDOWS
@@ -27,6 +31,9 @@ builder.Services.AddSingleton<IMicrophoneDeviceProvider>(services =>
services.GetRequiredService<WindowsMicrophoneDeviceProvider>());
builder.Services.AddSingleton<IMicrophoneCaptureSourceFactory>(services =>
services.GetRequiredService<WindowsMicrophoneDeviceProvider>());
builder.Services.AddSingleton<IMicrophoneSwitchPromptService, WindowsMicrophoneSwitchPromptService>();
builder.Services.AddSingleton<IAgentApprovalPromptService, WindowsAgentApprovalPromptService>();
builder.Services.AddSingleton<MicrophoneSwitchMonitor>();
builder.Services.AddSingleton<MicrophoneAudioSource>();
builder.Services.AddSingleton<SystemAudioSource>();
builder.Services.AddSingleton<IAcousticEchoCancellerFactory, AdaptiveFilterAcousticEchoCancellerFactory>();
@@ -40,6 +47,8 @@ builder.Services.AddSingleton<IMeetingMetadataProvider, OutlookClassicMeetingMet
PortableMeetingAudioRegistration.Add(builder.Services);
MacOsMeetingIntegrationRegistration.Add(builder.Services, OperatingSystem.IsMacOS());
builder.Services.AddHostedService<MacOsDesktopControlService>();
builder.Services.AddSingleton<IMicrophoneSwitchPromptService, NoopMicrophoneSwitchPromptService>();
builder.Services.AddSingleton<IAgentApprovalPromptService, NoopAgentApprovalPromptService>();
#endif
builder.Services.AddSingleton<ITranscriptStore, VaultTranscriptStore>();
builder.Services.AddSingleton<IMeetingNoteStore, MarkdownMeetingNoteStore>();
@@ -88,8 +97,35 @@ builder.Services.AddSingleton<ISpeakerSnippetExtractor, WavSpeakerSnippetExtract
builder.Services.AddSingleton<ISpeakerIdentityDiarizationClient, AzureSpeechSpeakerIdentityDiarizationClient>();
builder.Services.AddSingleton<ISpeakerIdentityMatchValidator, PyannoteSpeakerIdentityMatchValidator>();
builder.Services.AddSingleton<ISpeakerIdentityMatcher, AzureSpeechSpeakerIdentityMatcher>();
builder.Services.AddSingleton<ISpeakerIdentificationService, SpeakerIdentityService>();
builder.Services.AddSingleton<ISpeakerIdentityMergeService, SpeakerIdentityMergeService>();
builder.Services.AddSingleton<SpeakerIdentityService>();
builder.Services.AddSingleton<IResemblyzerVoiceEncoder, VenvResemblyzerVoiceEncoder>();
builder.Services.AddSingleton(services => new ResemblyzerVoiceClusterMatcher(
services.GetRequiredService<IOptions<MeetingAssistantOptions>>().Value.SpeakerIdentification.Resemblyzer,
services.GetRequiredService<ILogger<ResemblyzerVoiceClusterMatcher>>()));
builder.Services.AddSingleton(services => new ResemblyzerVoiceVectorOutlierPruner(
services.GetRequiredService<IOptions<MeetingAssistantOptions>>().Value.SpeakerIdentification.Resemblyzer,
services.GetRequiredService<ILogger<ResemblyzerVoiceVectorOutlierPruner>>()));
builder.Services.AddSingleton<ResemblyzerSpeakerIdentificationService>();
builder.Services.AddSingleton(services =>
{
var resemblyzer = services.GetRequiredService<IOptions<MeetingAssistantOptions>>()
.Value.SpeakerIdentification.Resemblyzer;
return resemblyzer.Enabled
? SpeakerSampleCollectionPolicy.IndependentVectors(resemblyzer.MaxVectorsPerIdentity)
: SpeakerSampleCollectionPolicy.ExistingBackend;
});
builder.Services.AddSingleton<ISpeakerIdentificationService>(services =>
services.GetRequiredService<IOptions<MeetingAssistantOptions>>()
.Value.SpeakerIdentification.Resemblyzer.Enabled
? services.GetRequiredService<ResemblyzerSpeakerIdentificationService>()
: services.GetRequiredService<SpeakerIdentityService>());
builder.Services.AddSingleton<SpeakerIdentityMergeService>();
builder.Services.AddSingleton<ResemblyzerSpeakerIdentityMergeService>();
builder.Services.AddSingleton<ISpeakerIdentityMergeService>(services =>
services.GetRequiredService<IOptions<MeetingAssistantOptions>>()
.Value.SpeakerIdentification.Resemblyzer.Enabled
? services.GetRequiredService<ResemblyzerSpeakerIdentityMergeService>()
: services.GetRequiredService<SpeakerIdentityMergeService>());
builder.Services.AddSingleton<ISpeakerIdentityAttendeeCanonicalizer, SpeakerIdentityAttendeeCanonicalizer>();
builder.Services.AddSingleton<IMeetingSummaryArtifactResolver, MeetingSummaryArtifactResolver>();
builder.Services.AddSingleton<IMeetingSummaryFailureWriter, MeetingSummaryFailureWriter>();
@@ -99,7 +135,11 @@ builder.Services.AddSingleton<IMeetingSummaryRetryRunner, MeetingSummaryRetryRun
builder.Services.AddSingleton<IMeetingWorkflowRulesProvider, FileMeetingWorkflowRulesProvider>();
builder.Services.AddSingleton<IMeetingWorkflowEngine, MeetingWorkflowEngine>();
builder.Services.AddSingleton<IWorkflowRulesEditorInstructionBuilder, WorkflowRulesEditorInstructionBuilder>();
#if WINDOWS
builder.Services.AddSingleton<IWorkflowRulesEditorSamplePlaybackQueue, WorkflowRulesEditorSamplePlaybackQueue>();
#else
builder.Services.AddSingleton<IWorkflowRulesEditorSamplePlaybackQueue, UnavailableWorkflowRulesEditorSamplePlaybackQueue>();
#endif
builder.Services.AddSingleton<IWorkflowRulesEditorChatPipeline, WorkflowRulesEditorChatPipeline>();
builder.Services.AddTransient<WorkflowRulesEditorChatViewModel>();
#if WINDOWS
@@ -110,6 +150,9 @@ builder.Services.AddSingleton<IWorkflowRulesEditorWindowService, WpfWorkflowRule
builder.Services.AddSingleton<MacOsWorkflowRulesEditorSession>();
builder.Services.AddSingleton<IWorkflowRulesEditorWindowService, MacOsWorkflowRulesEditorWindowService>();
#endif
builder.Services.AddSingleton<AgentApprovalService>();
builder.Services.AddSingleton<WorkflowChangeRequestService>();
builder.Services.AddHostedService(services => services.GetRequiredService<WorkflowChangeRequestService>());
builder.Services.AddSingleton<AsrDiagnosticService>();
builder.Services.AddSingleton<ICommandRunner, ProcessCommandRunner>();
builder.Services.AddSingleton<IFunAsrBackendReadinessProbe, FunAsrWebSocketBackendReadinessProbe>();
@@ -132,6 +175,7 @@ builder.Services.AddHostedService<OfflineTranscriptionBacklogHostedService>();
builder.Services.AddHostedService<SpeakerIdentityDatabaseInitializer>();
builder.Services.AddHostedService<SpeechRecognitionPipelineHostedService>();
builder.Services.AddHostedService<PyannoteDiarizationWarmupHostedService>();
builder.Services.AddHostedService<ResemblyzerWarmupHostedService>();
#if WINDOWS
builder.Services.AddHostedService<GlobalHotkeyService>();
builder.Services.AddHostedService<UnoTaskbarIconService>();
@@ -6,6 +6,8 @@ public interface IMeetingInactivityPromptService
MeetingInactivityPromptRequest request,
Func<MeetingInactivityPromptResponse, CancellationToken, Task> handleResponseAsync,
CancellationToken cancellationToken);
Task DismissAllAsync(CancellationToken cancellationToken);
}
public sealed record MeetingInactivityPromptRequest(
@@ -18,9 +20,32 @@ public enum MeetingInactivityPromptResponse
{
Dismissed,
Continue,
Pause,
Stop
}
internal sealed record MeetingInactivityPromptAction(
string Content,
string ResponseArgument,
MeetingInactivityPromptResponse Response);
internal static class MeetingInactivityPromptActions
{
public static IReadOnlyList<MeetingInactivityPromptAction> All { get; } =
[
new("Yes", "stop", MeetingInactivityPromptResponse.Stop),
new("No", "continue", MeetingInactivityPromptResponse.Continue),
new("Pause transcription", "pause", MeetingInactivityPromptResponse.Pause)
];
public static MeetingInactivityPromptResponse ParseResponse(string? value)
{
return All.FirstOrDefault(action =>
action.ResponseArgument.Equals(value, StringComparison.OrdinalIgnoreCase))
?.Response ?? MeetingInactivityPromptResponse.Continue;
}
}
public sealed class NoopMeetingInactivityPromptService : IMeetingInactivityPromptService
{
public Task ShowStopPromptAsync(
@@ -30,4 +55,9 @@ public sealed class NoopMeetingInactivityPromptService : IMeetingInactivityPromp
{
return Task.CompletedTask;
}
public Task DismissAllAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
@@ -2,5 +2,7 @@ namespace MeetingAssistant.Recording;
public interface IMicrophoneCaptureSourceFactory
{
// The caller owns the returned source and disposes it when it implements IDisposable,
// including when capture is canceled before enumeration starts.
IMeetingAudioSource CreateCapture(MeetingAssistantOptions options);
}
@@ -0,0 +1,12 @@
namespace MeetingAssistant.Recording;
public interface IMicrophoneSwitchPromptService
{
Task<bool> ConfirmSwitchAsync(MicrophoneDevice microphone, CancellationToken cancellationToken);
}
public sealed class NoopMicrophoneSwitchPromptService : IMicrophoneSwitchPromptService
{
public Task<bool> ConfirmSwitchAsync(MicrophoneDevice microphone, CancellationToken cancellationToken)
=> Task.FromResult(false);
}
@@ -33,6 +33,17 @@ public interface ITranscriptStore
string line,
CancellationToken cancellationToken);
async Task AppendLinesAsync(
TranscriptSession session,
IReadOnlyList<string> lines,
CancellationToken cancellationToken)
{
foreach (var line in lines)
{
await AppendLineAsync(session, line, cancellationToken);
}
}
Task ReplaceLineAsync(
TranscriptSession session,
TranscriptLineReference lineReference,
@@ -199,7 +199,13 @@ internal sealed class MacOsAudioCaptureProcessFactory : IMacOsAudioCaptureProces
throw new PlatformNotSupportedException("The macOS audio capture helper can only run on macOS.");
}
var helperPath = Path.Combine(AppContext.BaseDirectory, "Native", "macos-meeting-audio-capture");
var helperPath = Path.Combine(
AppContext.BaseDirectory,
"Native",
"MeetingAssistantAudioCapture.app",
"Contents",
"MacOS",
"macos-meeting-audio-capture");
if (!File.Exists(helperPath))
{
throw new FileNotFoundException(
@@ -34,6 +34,7 @@ public sealed class MeetingRecordingCoordinator
private readonly IMeetingInactivityClock inactivityClock;
private readonly IOfflineTranscriptionBacklog offlineTranscriptionBacklog;
private readonly MeetingAssistantOptions options;
private readonly SpeakerSampleCollectionPolicy speakerSampleCollectionPolicy;
private readonly ILogger<MeetingRecordingCoordinator> logger;
private readonly SemaphoreSlim gate = new(1, 1);
private RecordingRun? currentRun;
@@ -63,7 +64,8 @@ public sealed class MeetingRecordingCoordinator
IMeetingRunArtifactCleaner? artifactCleaner = null,
IMeetingInactivityPromptService? inactivityPromptService = null,
IMeetingInactivityClock? inactivityClock = null,
IOfflineTranscriptionBacklog? offlineTranscriptionBacklog = null)
IOfflineTranscriptionBacklog? offlineTranscriptionBacklog = null,
SpeakerSampleCollectionPolicy? speakerSampleCollectionPolicy = null)
{
this.audioSource = audioSource;
this.speechRecognitionPipelineFactory = speechRecognitionPipelineFactory;
@@ -86,6 +88,8 @@ public sealed class MeetingRecordingCoordinator
this.inactivityClock = inactivityClock ?? new SystemMeetingInactivityClock();
this.offlineTranscriptionBacklog = offlineTranscriptionBacklog ?? NoopOfflineTranscriptionBacklog.Instance;
this.options = options.Value;
this.speakerSampleCollectionPolicy = speakerSampleCollectionPolicy
?? SpeakerSampleCollectionPolicy.ExistingBackend;
this.logger = logger;
}
@@ -96,7 +100,8 @@ public sealed class MeetingRecordingCoordinator
currentArtifacts?.AssistantContextPath,
currentArtifacts?.SummaryPath,
GetProcessState(currentRun),
currentRun?.LaunchProfileName);
currentRun?.LaunchProfileName,
IsRecording && currentRun?.IsTranscriptionPaused == true);
public MeetingSessionArtifacts? CurrentArtifacts => currentArtifacts;
@@ -289,10 +294,12 @@ public sealed class MeetingRecordingCoordinator
startedAt,
launchProfile.Name,
runOptions.SpeakerIdentification.LiveSampleBufferDuration,
runOptions.SpeakerIdentification.MaxSnippetsPerSpeaker,
speakerSampleCollectionPolicy.ResolveRetainedSampleLimit(
runOptions.SpeakerIdentification.MaxSnippetsPerSpeaker),
speakerSampleCollectionPolicy.RequireNonOverlappingSamples,
logger);
run.Task = Task.Run(() => RecordAsync(run), CancellationToken.None);
if (runOptions.Recording.InactivitySafeguard.Enabled)
if (ShouldRunInactivitySafeguard(runOptions.Recording.InactivitySafeguard))
{
_ = Task.Run(
() => RunInactivitySafeguardAsync(run),
@@ -334,11 +341,14 @@ public sealed class MeetingRecordingCoordinator
public async Task<RecordingStatus> StopAsync(CancellationToken cancellationToken)
{
return await StopAsync(null, cancellationToken);
return await StopAsync(null, null, null, null, cancellationToken);
}
private async Task<RecordingStatus> StopAsync(
DateTimeOffset? inferredEndTime,
RecordingRun? expectedRun,
long? expectedActivityVersion,
DateTimeOffset? expectedPauseStartedAt,
CancellationToken cancellationToken)
{
RecordingRun run;
@@ -347,13 +357,30 @@ public sealed class MeetingRecordingCoordinator
await gate.WaitAsync(cancellationToken);
try
{
if (currentRun is null || currentRun.IsCaptureStopping)
if (currentRun is null ||
currentRun.IsCaptureStopping ||
expectedRun is not null && !ReferenceEquals(currentRun, expectedRun))
{
return CurrentStatus;
}
run = currentRun;
if (inferredEndTime is not null)
var inactivityStopClaimed = expectedActivityVersion is not null;
if (expectedActivityVersion is not null &&
(inferredEndTime is null ||
!run.TryClaimInactivityStop(expectedActivityVersion.Value, inferredEndTime.Value)))
{
return CurrentStatus;
}
var maximumPauseStopClaimed = expectedPauseStartedAt is not null;
if (expectedPauseStartedAt is not null &&
!run.TryClaimMaximumPauseStop(expectedPauseStartedAt.Value))
{
return CurrentStatus;
}
if (!inactivityStopClaimed && !maximumPauseStopClaimed && inferredEndTime is not null)
{
run.SetInferredEndTime(inferredEndTime.Value);
}
@@ -363,7 +390,7 @@ public sealed class MeetingRecordingCoordinator
{
run.Abort();
}
else
else if (!inactivityStopClaimed && !maximumPauseStopClaimed)
{
run.StopCapture();
}
@@ -389,6 +416,12 @@ public sealed class MeetingRecordingCoordinator
}
catch (TimeoutException)
{
if (run.ContextState is AssistantContextState.Summarizing or AssistantContextState.Finished or AssistantContextState.Error)
{
logger.LogInformation("Recording has reached summarization; continuing remaining processing in the background after the stop wait expired");
return CurrentStatus;
}
if (IsAzureSpeechRun(run))
{
logger.LogWarning("Timed out while draining Azure Speech transcription after recording stop; queueing offline transcription backlog item");
@@ -409,6 +442,65 @@ public sealed class MeetingRecordingCoordinator
return CurrentStatus;
}
public async Task<RecordingStatus> SetTranscriptionPausedAsync(
bool isPaused,
CancellationToken cancellationToken)
{
return await SetTranscriptionPausedAsync(isPaused, null, null, cancellationToken);
}
private async Task<RecordingStatus> SetTranscriptionPausedAsync(
bool isPaused,
RecordingRun? expectedRun,
long? expectedActivityVersion,
CancellationToken cancellationToken)
{
await gate.WaitAsync(cancellationToken);
try
{
var run = currentRun;
if (run is null ||
run.IsCaptureStopping ||
expectedRun is not null && !ReferenceEquals(run, expectedRun))
{
return CurrentStatus;
}
var changed = false;
await run.PipelineGate.WaitAsync(cancellationToken);
try
{
var changedAt = inactivityClock.Now;
changed = run.TrySetTranscriptionPaused(
isPaused,
changedAt,
expectedActivityVersion);
}
finally
{
run.PipelineGate.Release();
}
if (changed && isPaused)
{
await DismissInactivityPromptsAsync(cancellationToken);
}
if (changed)
{
logger.LogInformation(
"Meeting transcription {PauseState}",
isPaused ? "paused" : "unpaused");
}
return CurrentStatus;
}
finally
{
gate.Release();
}
}
public async Task<RecordingStatus> AbortAsync(CancellationToken cancellationToken)
{
RecordingRun run;
@@ -624,6 +716,24 @@ public sealed class MeetingRecordingCoordinator
catch (Exception exception)
{
logger.LogError(exception, "Meeting recording failed");
if (IsAzureSpeechRun(run) && !run.IsQueuedForOfflineTranscription)
{
logger.LogWarning(
"Azure Speech transcription failed; retaining recorded audio and queueing offline transcription backlog item");
run.MarkQueuedForOfflineTranscription();
try
{
await offlineTranscriptionBacklog.EnqueueAsync(
CreateOfflineBacklogItem(run),
CancellationToken.None);
}
catch (Exception backlogException)
{
logger.LogError(
backlogException,
"Could not queue offline transcription backlog item; retaining recorded audio for manual recovery");
}
}
}
finally
{
@@ -661,8 +771,7 @@ public sealed class MeetingRecordingCoordinator
chunk.Channels);
}
await run.RecordedAudio.AppendAsync(chunk, run.CaptureCancellation);
await run.WriteAudioAsync(chunk, run.CaptureCancellation);
await run.RouteCapturedAudioAsync(chunk, run.CaptureCancellation);
}
logger.LogInformation(
@@ -701,7 +810,6 @@ public sealed class MeetingRecordingCoordinator
run.ResetSpeakerIdentification();
}
run.RecordTranscriptActivity(segment, inactivityClock.Now);
var sample = run.TryAddSpeakerSample(segment);
if (sample is not null)
{
@@ -718,6 +826,56 @@ public sealed class MeetingRecordingCoordinator
var relabeledSegment = run.Relabel(segment);
run.AddLiveSegment(relabeledSegment);
await AppendTranscriptSegmentAsync(run, relabeledSegment, cancellationToken);
run.RecordTranscriptActivity(segment, inactivityClock.Now);
if (!string.IsNullOrWhiteSpace(segment.Text))
{
// Switching profiles holds the coordinator gate while draining this reader.
// Notification cleanup must not make transcript draining wait for that gate.
_ = DismissInactivityPromptsForRunAsync(run, run.CaptureCancellation);
}
}
}
private async Task DismissInactivityPromptsForRunAsync(
RecordingRun run,
CancellationToken cancellationToken)
{
try
{
await gate.WaitAsync(cancellationToken);
try
{
if (!ReferenceEquals(currentRun, run) || run.IsCaptureStopping)
{
return;
}
await DismissInactivityPromptsAsync(cancellationToken);
}
finally
{
gate.Release();
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
// Pending cleanup is no longer needed after this run stops capturing.
}
}
private async Task DismissInactivityPromptsAsync(CancellationToken cancellationToken)
{
try
{
await inactivityPromptService.DismissAllAsync(cancellationToken);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (Exception exception)
{
logger.LogWarning(exception, "Failed to dismiss transcript inactivity prompts");
}
}
@@ -845,7 +1003,10 @@ public sealed class MeetingRecordingCoordinator
private async Task RunInactivitySafeguardAsync(RecordingRun run)
{
var safeguardOptions = run.Options.Recording.InactivitySafeguard;
var promptThresholds = GetInactivityPromptThresholds(safeguardOptions);
var promptThresholds = safeguardOptions.Enabled
? GetInactivityPromptThresholds(safeguardOptions)
: [];
var checkInterval = GetInactivitySafeguardCheckInterval(safeguardOptions);
var promptedThresholds = new HashSet<TimeSpan>();
var lastActivityVersion = run.GetTranscriptActivitySnapshot().ActivityVersion;
@@ -853,7 +1014,7 @@ public sealed class MeetingRecordingCoordinator
"Recording inactivity safeguard started with prompt thresholds {PromptThresholds}, auto-stop {AutoStopAfter}, check interval {CheckInterval}",
string.Join(", ", promptThresholds),
safeguardOptions.AutoStopAfter,
safeguardOptions.CheckInterval);
checkInterval);
try
{
@@ -869,9 +1030,38 @@ public sealed class MeetingRecordingCoordinator
snapshot.LastTranscriptActivityAt);
}
var pauseStartedAt = snapshot.TranscriptionPausedAt;
if (pauseStartedAt is not null)
{
var maximumPauseDuration = safeguardOptions.MaximumPauseDuration;
if (maximumPauseDuration > TimeSpan.Zero &&
inactivityClock.Now - pauseStartedAt.Value >= maximumPauseDuration)
{
logger.LogWarning(
"Recording inactivity safeguard stopping meeting after transcription remained paused for {PauseDuration}",
maximumPauseDuration);
var stopStatus = await StopAsync(
null,
run,
null,
pauseStartedAt,
CancellationToken.None);
if (!stopStatus.IsRecording)
{
return;
}
}
await inactivityClock.DelayAsync(
checkInterval,
run.CaptureCancellation);
continue;
}
var now = inactivityClock.Now;
var inactivityDuration = now - snapshot.LastTranscriptActivityAt;
if (safeguardOptions.AutoStopAfter > TimeSpan.Zero &&
if (safeguardOptions.Enabled &&
safeguardOptions.AutoStopAfter > TimeSpan.Zero &&
inactivityDuration >= safeguardOptions.AutoStopAfter)
{
var inferredEndTime = run.GetInactivitySafeguardEndTime(safeguardOptions.InferredEndPadding);
@@ -879,8 +1069,16 @@ public sealed class MeetingRecordingCoordinator
"Recording inactivity safeguard auto-stopping meeting after {InactivityDuration} without transcript text; inferred end time {InferredEndTime}",
inactivityDuration,
inferredEndTime);
await StopAsync(inferredEndTime, CancellationToken.None);
return;
var stopStatus = await StopAsync(
inferredEndTime,
run,
snapshot.ActivityVersion,
null,
CancellationToken.None);
if (!stopStatus.IsRecording)
{
return;
}
}
foreach (var threshold in promptThresholds)
@@ -897,6 +1095,7 @@ public sealed class MeetingRecordingCoordinator
inactivityDuration,
threshold,
inferredEndTime);
var promptActivityVersion = snapshot.ActivityVersion;
await inactivityPromptService.ShowStopPromptAsync(
new MeetingInactivityPromptRequest(
inactivityDuration,
@@ -912,16 +1111,36 @@ public sealed class MeetingRecordingCoordinator
if (response == MeetingInactivityPromptResponse.Stop)
{
await StopAsync(inferredEndTime, CancellationToken.None);
await StopAsync(
inferredEndTime,
run,
promptActivityVersion,
null,
CancellationToken.None);
}
else if (response == MeetingInactivityPromptResponse.Pause)
{
await SetTranscriptionPausedAsync(
true,
run,
promptActivityVersion,
CancellationToken.None);
}
},
run.CaptureCancellation);
var currentActivity = run.GetTranscriptActivitySnapshot();
if (currentActivity.IsTranscriptionPaused ||
currentActivity.ActivityVersion != promptActivityVersion)
{
await DismissInactivityPromptsAsync(run.CaptureCancellation);
}
break;
}
await inactivityClock.DelayAsync(
GetInactivitySafeguardCheckInterval(safeguardOptions),
checkInterval,
run.CaptureCancellation);
}
}
@@ -989,11 +1208,11 @@ public sealed class MeetingRecordingCoordinator
try
{
var meetingNote = await meetingNoteStore.ReadAsync(run.MeetingNotePath, cancellationToken);
var checkpoint = run.CreateLiveIdentificationCheckpoint(meetingNote);
var checkpoint = run.CreateLiveIdentificationCheckpoint(meetingNote, samples);
if (checkpoint is null)
{
logger.LogInformation(
"Skipping live speaker identity matching because no new unmapped sample speakers or attendee changes were found");
"Skipping live speaker identity matching because no new eligible speaker evidence or attendee changes were found");
return;
}
@@ -1821,6 +2040,11 @@ public sealed class MeetingRecordingCoordinator
.ToList();
}
private static bool ShouldRunInactivitySafeguard(RecordingInactivitySafeguardOptions options)
{
return options.Enabled || options.MaximumPauseDuration > TimeSpan.Zero;
}
private static TimeSpan GetInactivitySafeguardCheckInterval(
RecordingInactivitySafeguardOptions options)
{
@@ -1877,6 +2101,8 @@ public sealed class MeetingRecordingCoordinator
private TimeSpan? lastTranscriptSegmentEnd;
private long transcriptActivityVersion;
private DateTimeOffset? inferredEndTime;
private bool isTranscriptionPaused;
private DateTimeOffset? transcriptionPausedAt;
public RecordingRun(
CancellationTokenSource captureCancellation,
@@ -1892,6 +2118,7 @@ public sealed class MeetingRecordingCoordinator
string launchProfileName,
TimeSpan liveSampleBufferDuration,
int maxSpeakerSamples,
bool requireNonOverlappingSpeakerSamples,
ILogger logger)
{
CaptureCancellationSource = captureCancellation;
@@ -1912,8 +2139,9 @@ public sealed class MeetingRecordingCoordinator
liveSampleBufferDuration,
maxSpeakerSamples,
options.SpeakerIdentification.MinimumSampleSpeechDuration,
options.SpeakerIdentification.MaximumSampleSegmentGap,
logger);
options.SpeakerIdentification.MaximumSampleDuration,
logger,
requireNonOverlappingSpeakerSamples);
}
public CancellationTokenSource CaptureCancellationSource { get; }
@@ -1960,6 +2188,17 @@ public sealed class MeetingRecordingCoordinator
public bool HasAttachedPromptMetadata { get; private set; }
public bool IsTranscriptionPaused
{
get
{
lock (transcriptActivityGate)
{
return isTranscriptionPaused;
}
}
}
public AssistantContextState ContextState { get; private set; } = AssistantContextState.CollectingMetadata;
public DateTimeOffset? InferredEndTime
@@ -1983,6 +2222,37 @@ public sealed class MeetingRecordingCoordinator
HasAttachedPromptMetadata = true;
}
public bool TrySetTranscriptionPaused(
bool isPaused,
DateTimeOffset changedAt,
long? expectedActivityVersion)
{
lock (transcriptActivityGate)
{
if (expectedActivityVersion is not null &&
transcriptActivityVersion != expectedActivityVersion.Value)
{
return false;
}
if (isTranscriptionPaused == isPaused)
{
return false;
}
isTranscriptionPaused = isPaused;
transcriptionPausedAt = isPaused ? changedAt : null;
if (!isPaused)
{
lastTranscriptActivityAt = changedAt;
}
transcriptActivityVersion++;
return true;
}
}
public void Abort()
{
IsAborted = true;
@@ -2011,13 +2281,17 @@ public sealed class MeetingRecordingCoordinator
return string.Equals(LaunchProfileName, launchProfileName, StringComparison.OrdinalIgnoreCase);
}
public async ValueTask WriteAudioAsync(AudioChunk chunk, CancellationToken cancellationToken)
public async ValueTask RouteCapturedAudioAsync(AudioChunk chunk, CancellationToken cancellationToken)
{
await PipelineGate.WaitAsync(cancellationToken);
try
{
AppendAudio(chunk);
await Pipeline.WriteAsync(chunk, cancellationToken);
var routedChunk = IsTranscriptionPaused
? chunk with { Pcm = new byte[chunk.Pcm.Length] }
: chunk;
await RecordedAudio.AppendAsync(routedChunk, cancellationToken);
AppendAudio(routedChunk);
await Pipeline.WriteAsync(routedChunk, cancellationToken);
}
finally
{
@@ -2201,7 +2475,9 @@ public sealed class MeetingRecordingCoordinator
return new TranscriptActivitySnapshot(
lastTranscriptActivityAt,
lastTranscriptSegmentEnd,
transcriptActivityVersion);
transcriptActivityVersion,
isTranscriptionPaused,
transcriptionPausedAt);
}
}
@@ -2221,6 +2497,35 @@ public sealed class MeetingRecordingCoordinator
}
}
public bool TryClaimInactivityStop(long expectedActivityVersion, DateTimeOffset endTime)
{
lock (transcriptActivityGate)
{
if (isTranscriptionPaused || transcriptActivityVersion != expectedActivityVersion)
{
return false;
}
inferredEndTime = endTime;
StopCapture();
return true;
}
}
public bool TryClaimMaximumPauseStop(DateTimeOffset expectedPauseStartedAt)
{
lock (transcriptActivityGate)
{
if (!isTranscriptionPaused || transcriptionPausedAt != expectedPauseStartedAt)
{
return false;
}
StopCapture();
return true;
}
}
public void AppendAudio(AudioChunk chunk)
{
speakerSampleCollector.AppendAudio(chunk);
@@ -2244,9 +2549,11 @@ public sealed class MeetingRecordingCoordinator
return speakerSampleCollector.Snapshot();
}
public LiveIdentificationCheckpoint? CreateLiveIdentificationCheckpoint(MeetingNote meetingNote)
public LiveIdentificationCheckpoint? CreateLiveIdentificationCheckpoint(
MeetingNote meetingNote,
IReadOnlyList<SpeakerAudioSample> samples)
{
var speakers = GetUnmappedSampleSpeakers();
var speakers = GetUnmappedSampleSpeakers(samples);
if (speakers.Count == 0)
{
return null;
@@ -2254,7 +2561,11 @@ public sealed class MeetingRecordingCoordinator
var checkpoint = new LiveIdentificationCheckpoint(
speakers,
BuildAttendeeSignature(meetingNote));
BuildAttendeeSignature(meetingNote),
Options.SpeakerIdentification.Resemblyzer.Enabled
? speakers.Select(speaker => samples.Count(sample =>
string.Equals(sample.Speaker, speaker, StringComparison.OrdinalIgnoreCase))).ToArray()
: []);
lock (liveIdentificationGate)
{
return lastLiveIdentificationCheckpoint?.Matches(checkpoint) == true
@@ -2312,9 +2623,9 @@ public sealed class MeetingRecordingCoordinator
StringComparer.OrdinalIgnoreCase);
}
private IReadOnlyList<string> GetUnmappedSampleSpeakers()
private IReadOnlyList<string> GetUnmappedSampleSpeakers(IReadOnlyList<SpeakerAudioSample> samples)
{
return GetSpeakerSamplesSnapshot()
return samples
.Select(sample => sample.Speaker)
.Where(speaker => !string.IsNullOrWhiteSpace(speaker))
.Where(speaker => !string.Equals(speaker, "Unknown", StringComparison.OrdinalIgnoreCase))
@@ -2350,7 +2661,9 @@ public sealed class MeetingRecordingCoordinator
public sealed record TranscriptActivitySnapshot(
DateTimeOffset LastTranscriptActivityAt,
TimeSpan? LastTranscriptSegmentEnd,
long ActivityVersion);
long ActivityVersion,
bool IsTranscriptionPaused,
DateTimeOffset? TranscriptionPausedAt);
private static int StateRank(AssistantContextState state)
{
@@ -2368,11 +2681,13 @@ public sealed class MeetingRecordingCoordinator
public sealed record LiveIdentificationCheckpoint(
IReadOnlyList<string> Speakers,
string AttendeeSignature)
string AttendeeSignature,
IReadOnlyList<int> SampleCounts)
{
public bool Matches(LiveIdentificationCheckpoint other)
{
return string.Equals(AttendeeSignature, other.AttendeeSignature, StringComparison.Ordinal) &&
SampleCounts.SequenceEqual(other.SampleCounts) &&
Speakers.Count == other.Speakers.Count &&
Speakers.SequenceEqual(other.Speakers, StringComparer.OrdinalIgnoreCase);
}
@@ -2405,7 +2720,8 @@ public sealed record RecordingStatus(
string? AssistantContextPath,
string? SummaryPath,
RecordingProcessState State = RecordingProcessState.Idle,
string? LaunchProfile = null);
string? LaunchProfile = null,
bool IsPaused = false);
public enum RecordingProcessState
{
@@ -8,22 +8,30 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
private readonly IMicrophoneCaptureSourceFactory captureSources;
private readonly ILogger<MicrophoneAudioSource> logger;
private readonly TimeSpan recoveryDelay;
private readonly MicrophoneSwitchMonitor? switchMonitor;
private readonly MicrophoneDeviceSelection? selection;
public MicrophoneAudioSource(
IMicrophoneCaptureSourceFactory captureSources,
ILogger<MicrophoneAudioSource> logger)
: this(captureSources, logger, DefaultRecoveryDelay)
ILogger<MicrophoneAudioSource> logger,
MicrophoneSwitchMonitor? switchMonitor = null,
MicrophoneDeviceSelection? selection = null)
: this(captureSources, logger, DefaultRecoveryDelay, switchMonitor, selection)
{
}
internal MicrophoneAudioSource(
IMicrophoneCaptureSourceFactory captureSources,
ILogger<MicrophoneAudioSource> logger,
TimeSpan recoveryDelay)
TimeSpan recoveryDelay,
MicrophoneSwitchMonitor? switchMonitor = null,
MicrophoneDeviceSelection? selection = null)
{
this.captureSources = captureSources;
this.logger = logger;
this.recoveryDelay = recoveryDelay;
this.switchMonitor = switchMonitor;
this.selection = selection;
}
public IAsyncEnumerable<AudioChunk> CaptureAsync(CancellationToken cancellationToken)
@@ -34,21 +42,44 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
public async IAsyncEnumerable<AudioChunk> CaptureAsync(
MeetingAssistantOptions options,
[EnumeratorCancellation] CancellationToken cancellationToken)
{
using var session = new CaptureSession(selection, cancellationToken);
var monitoring = switchMonitor?.RunAsync(session.SwitchTo, session.Token) ?? Task.CompletedTask;
try
{
await foreach (var chunk in CaptureWithRecoveryAsync(options, session, session.Token))
{
yield return chunk;
}
}
finally
{
session.Stop();
await monitoring;
}
}
private async IAsyncEnumerable<AudioChunk> CaptureWithRecoveryAsync(
MeetingAssistantOptions options,
CaptureSession session,
[EnumeratorCancellation] CancellationToken cancellationToken)
{
var failedAttempts = 0;
while (!cancellationToken.IsCancellationRequested)
{
var captureToken = session.BeginCapture();
IMeetingAudioSource? captureSource = null;
IAsyncEnumerator<AudioChunk>? capture = null;
Exception? failure = null;
try
{
capture = captureSources
.CreateCapture(options)
.CaptureAsync(options, cancellationToken)
.GetAsyncEnumerator(cancellationToken);
captureSource = captureSources.CreateCapture(options);
capture = captureSource
.CaptureAsync(options, captureToken)
.GetAsyncEnumerator(captureToken);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
catch (OperationCanceledException) when (captureToken.IsCancellationRequested)
{
}
catch (Exception exception)
@@ -56,6 +87,7 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
failure = exception;
}
using var captureOwner = captureSource as IDisposable;
if (cancellationToken.IsCancellationRequested)
{
yield break;
@@ -65,14 +97,14 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
{
try
{
while (!cancellationToken.IsCancellationRequested)
while (!captureToken.IsCancellationRequested)
{
var hasNext = false;
try
{
hasNext = await capture.MoveNextAsync();
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
catch (OperationCanceledException) when (captureToken.IsCancellationRequested)
{
}
catch (Exception exception)
@@ -80,7 +112,7 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
failure = exception;
}
if (cancellationToken.IsCancellationRequested || failure is not null || !hasNext)
if (captureToken.IsCancellationRequested || failure is not null || !hasNext)
{
break;
}
@@ -102,7 +134,7 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
{
await capture.DisposeAsync();
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
catch (OperationCanceledException) when (captureToken.IsCancellationRequested)
{
}
catch (Exception exception)
@@ -112,11 +144,19 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
}
}
var switchRequested = captureToken.IsCancellationRequested;
session.EndCapture();
if (cancellationToken.IsCancellationRequested)
{
yield break;
}
if (switchRequested)
{
logger.LogInformation("Replacing microphone capture after an accepted device-switch notification");
continue;
}
failedAttempts++;
logger.LogWarning(
failure,
@@ -142,4 +182,53 @@ public sealed class MicrophoneAudioSource : IMeetingAudioSource
return false;
}
}
private sealed class CaptureSession(MicrophoneDeviceSelection? selection, CancellationToken cancellationToken) : IDisposable
{
private readonly object gate = new();
private readonly CancellationTokenSource lifetime = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
private CancellationTokenSource? capture;
public CancellationToken Token => lifetime.Token;
public CancellationToken BeginCapture()
{
lock (gate)
{
capture = CancellationTokenSource.CreateLinkedTokenSource(lifetime.Token);
return capture.Token;
}
}
public void EndCapture()
{
lock (gate)
{
capture?.Dispose();
capture = null;
}
}
public void SwitchTo(MicrophoneDevice device, CancellationToken promptCancellation)
{
lock (gate)
{
if (lifetime.IsCancellationRequested || promptCancellation.IsCancellationRequested || selection is null) return;
selection.Select(device.Id);
capture?.Cancel();
}
}
public void Stop()
{
lock (gate) lifetime.Cancel();
}
public void Dispose()
{
Stop();
EndCapture();
lifetime.Dispose();
}
}
}
@@ -0,0 +1,114 @@
namespace MeetingAssistant.Recording;
public sealed class MicrophoneSwitchMonitor
{
private readonly IMicrophoneDeviceProvider devices;
private readonly IMicrophoneSwitchPromptService prompts;
private readonly ILogger<MicrophoneSwitchMonitor> logger;
private readonly TimeSpan pollInterval;
private readonly TimeSpan promptLifetime;
public MicrophoneSwitchMonitor(
IMicrophoneDeviceProvider devices,
IMicrophoneSwitchPromptService prompts,
ILogger<MicrophoneSwitchMonitor> logger)
: this(devices, prompts, logger, TimeSpan.FromSeconds(1), TimeSpan.FromMinutes(1))
{
}
internal MicrophoneSwitchMonitor(
IMicrophoneDeviceProvider devices,
IMicrophoneSwitchPromptService prompts,
ILogger<MicrophoneSwitchMonitor> logger,
TimeSpan pollInterval,
TimeSpan promptLifetime)
{
this.devices = devices;
this.prompts = prompts;
this.logger = logger;
this.pollInterval = pollInterval;
this.promptLifetime = promptLifetime;
}
public async Task RunAsync(Action<MicrophoneDevice, CancellationToken> switchMicrophone, CancellationToken cancellationToken)
{
var known = (ReadDevices() ?? []).Select(device => device.Id).ToHashSet(StringComparer.OrdinalIgnoreCase);
var pending = new Dictionary<string, PendingPrompt>(StringComparer.OrdinalIgnoreCase);
try
{
while (true)
{
await Task.Delay(pollInterval, cancellationToken);
var available = ReadDevices();
if (available is null) continue;
var availableIds = available.Select(device => device.Id).ToHashSet(StringComparer.OrdinalIgnoreCase);
foreach (var (id, prompt) in pending.ToArray())
{
if (!availableIds.Contains(id)) await prompt.Cancellation.CancelAsync();
if (prompt.Task.IsCompleted)
{
prompt.Cancellation.Dispose();
pending.Remove(id);
}
}
foreach (var device in available)
{
if (known.Add(device.Id))
{
var promptCancellation = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
promptCancellation.CancelAfter(promptLifetime);
pending.Add(device.Id, new PendingPrompt(promptCancellation,
OfferAsync(device, switchMicrophone, promptCancellation.Token)));
}
}
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
}
finally
{
foreach (var prompt in pending.Values) await prompt.Cancellation.CancelAsync();
await Task.WhenAll(pending.Values.Select(prompt => prompt.Task));
foreach (var prompt in pending.Values) prompt.Cancellation.Dispose();
}
}
private async Task OfferAsync(MicrophoneDevice device, Action<MicrophoneDevice, CancellationToken> switchMicrophone,
CancellationToken cancellationToken)
{
try
{
if (await prompts.ConfirmSwitchAsync(device, cancellationToken)
&& !cancellationToken.IsCancellationRequested
&& ReadDevices()?.Any(current => current.Id.Equals(device.Id, StringComparison.OrdinalIgnoreCase)) == true
&& !cancellationToken.IsCancellationRequested)
{
switchMicrophone(device, cancellationToken);
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
}
catch (Exception exception)
{
logger.LogWarning(exception, "Could not offer microphone {MicrophoneName}", device.Name);
}
}
private IReadOnlyList<MicrophoneDevice>? ReadDevices()
{
try
{
return devices.GetAvailableMicrophones();
}
catch (Exception exception)
{
logger.LogWarning(exception, "Could not check for newly available microphones");
return null;
}
}
private sealed record PendingPrompt(CancellationTokenSource Cancellation, Task Task);
}
@@ -4,11 +4,12 @@ using NAudio.Wave;
namespace MeetingAssistant.Recording;
internal sealed class NaudioCaptureAudioSource : IMeetingAudioSource
internal sealed class NaudioCaptureAudioSource : IMeetingAudioSource, IDisposable
{
private readonly IWaveIn capture;
private readonly string sourceName;
private readonly ILogger logger;
private int disposed;
public NaudioCaptureAudioSource(
IWaveIn capture,
@@ -22,13 +23,15 @@ internal sealed class NaudioCaptureAudioSource : IMeetingAudioSource
public IAsyncEnumerable<AudioChunk> CaptureAsync(CancellationToken cancellationToken)
{
return CaptureWith(capture, sourceName, logger, cancellationToken);
return CaptureWith(cancellationToken);
}
private static async IAsyncEnumerable<AudioChunk> CaptureWith(
IWaveIn capture,
string sourceName,
ILogger logger,
public void Dispose()
{
if (Interlocked.Exchange(ref disposed, 1) == 0) capture.Dispose();
}
private async IAsyncEnumerable<AudioChunk> CaptureWith(
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
{
var channel = Channel.CreateUnbounded<AudioChunk>();
@@ -83,7 +86,7 @@ internal sealed class NaudioCaptureAudioSource : IMeetingAudioSource
}
finally
{
capture.Dispose();
Dispose();
}
}
}
@@ -117,8 +117,25 @@ public sealed class OfflineTranscriptionBacklogProcessor
pipelineOptions,
cancellationToken);
if (!finishedSegments.Any(segment =>
segment.Kind == TranscriptionSegmentKind.Speech && !string.IsNullOrWhiteSpace(segment.Text)))
{
throw new InvalidOperationException(
"Offline transcription replay returned no speech text; preserving the existing transcript and recovery files.");
}
var lines = await TransformTranscriptLinesAsync(artifacts, runOptions, finishedSegments, cancellationToken);
await transcriptStore.ReplaceLinesAsync(session, lines, cancellationToken);
await transcriptStore.AppendLinesAsync(session,
[
"",
"## Offline transcription recovery",
"",
"> The following transcript was recovered by replaying the entire recording. "
+ "It may duplicate earlier transcript passages. "
+ "Treat repeated passages as the same discussion when summarizing.",
"",
.. lines
], cancellationToken);
meetingNote.Frontmatter.EndTime = item.InferredEndTime ?? DateTimeOffset.Now;
var completedMeetingNote = await meetingNoteStore.SaveAsync(meetingNote, runOptions, cancellationToken);
@@ -9,37 +9,31 @@ internal sealed class SpeakerAudioSampleCollector
private readonly object gate = new();
private readonly RollingAudioBuffer audioBuffer;
private readonly Dictionary<string, List<SpeakerAudioSample>> samplesBySpeaker = new(StringComparer.OrdinalIgnoreCase);
private readonly Dictionary<string, TimeSpan> lastAcceptedEndBySpeaker = new(StringComparer.OrdinalIgnoreCase);
private readonly int maxSamplesPerSpeaker;
private readonly TimeSpan minimumUninterruptedSpeechDuration;
private readonly TimeSpan maximumSegmentGap;
private readonly TimeSpan minimumSampleSpeechDuration;
private readonly TimeSpan maximumSampleDuration;
private readonly bool requireNonOverlappingSamples;
private readonly ILogger? logger;
private PendingSpeakerSpan? pendingSpan;
public SpeakerAudioSampleCollector(TimeSpan bufferDuration, int maxSamplesPerSpeaker)
: this(
bufferDuration,
maxSamplesPerSpeaker,
TimeSpan.FromSeconds(30),
TimeSpan.FromSeconds(1),
logger: null)
{
}
private SpeakerSampleSpan? pendingSpan;
public SpeakerAudioSampleCollector(
TimeSpan bufferDuration,
int maxSamplesPerSpeaker,
TimeSpan minimumUninterruptedSpeechDuration,
TimeSpan maximumSegmentGap,
ILogger? logger = null)
TimeSpan minimumSampleSpeechDuration,
TimeSpan maximumSampleDuration,
ILogger? logger = null,
bool requireNonOverlappingSamples = false)
{
SpeakerSampleDurationConfiguration.ValidateOrThrow(
minimumSampleSpeechDuration,
maximumSampleDuration,
"Speaker sample collector configuration is invalid");
audioBuffer = new RollingAudioBuffer(bufferDuration);
this.maxSamplesPerSpeaker = Math.Max(1, maxSamplesPerSpeaker);
this.minimumUninterruptedSpeechDuration = minimumUninterruptedSpeechDuration > TimeSpan.Zero
? minimumUninterruptedSpeechDuration
: TimeSpan.Zero;
this.maximumSegmentGap = maximumSegmentGap >= TimeSpan.Zero
? maximumSegmentGap
: TimeSpan.Zero;
this.minimumSampleSpeechDuration = minimumSampleSpeechDuration;
this.maximumSampleDuration = maximumSampleDuration;
this.requireNonOverlappingSamples = requireNonOverlappingSamples;
this.logger = logger;
}
@@ -53,6 +47,7 @@ internal sealed class SpeakerAudioSampleCollector
lock (gate)
{
samplesBySpeaker.Clear();
lastAcceptedEndBySpeaker.Clear();
pendingSpan = null;
audioBuffer.Reset();
}
@@ -68,34 +63,54 @@ internal sealed class SpeakerAudioSampleCollector
return null;
}
TranscriptionSegment sampleSegment;
PendingSpanReset? reset;
SpeakerSampleSpan? sampleSpan;
SpeakerSampleSpan? previousSpan;
lock (gate)
{
(sampleSegment, reset) = ExtendPendingSpan(segment);
if (requireNonOverlappingSamples &&
lastAcceptedEndBySpeaker.TryGetValue(segment.Speaker, out var lastAcceptedEnd) &&
segment.Start < lastAcceptedEnd)
{
logger?.LogInformation(
"Discarding speaker identity sample for {Speaker} because it overlaps an accepted sample ending at {AcceptedSampleEnd}",
segment.Speaker,
lastAcceptedEnd);
return null;
}
(sampleSpan, previousSpan) = ExtendPendingSpan(segment);
}
if (reset is not null)
if (sampleSpan is null)
{
logger?.LogInformation(
"Reset speaker identity sample span from {PreviousSpeaker} to {Speaker}: previous end {PreviousEnd}, next start {NextStart}, gap {Gap}, maximum gap {MaximumGap}",
reset.PreviousSpeaker,
segment.Speaker,
reset.PreviousEnd,
segment.Start,
reset.Gap,
maximumSegmentGap);
"Discarding speaker identity sample for {Speaker} because the segment duration is not positive",
segment.Speaker);
return null;
}
var score = Score(sampleSegment, minimumUninterruptedSpeechDuration);
var sampleSegment = sampleSpan.ToSegment();
if (previousSpan is not null)
{
logger?.LogInformation(
"Reset speaker identity sample span from {PreviousSpeaker} to {Speaker}: previous end {PreviousEnd}, next start {NextStart}",
previousSpan.Speaker,
segment.Speaker,
previousSpan.End,
segment.Start);
}
var score = Score(sampleSegment, sampleSpan.SpeechDuration, minimumSampleSpeechDuration);
if (!score.Accepted)
{
logger?.LogInformation(
"Discarding speaker identity sample for {Speaker} because {Reason}: duration {Duration}, minimum duration {MinimumDuration}, word count {WordCount}",
"Discarding speaker identity sample for {Speaker} because {Reason}: speaker audio {SpeakerAudioDuration}, clip duration {ClipDuration}, minimum duration {MinimumDuration}, word count {WordCount}",
sampleSegment.Speaker,
score.Reason,
sampleSpan.SpeechDuration,
sampleSegment.End - sampleSegment.Start,
minimumUninterruptedSpeechDuration,
minimumSampleSpeechDuration,
score.WordCount);
return null;
}
@@ -115,6 +130,13 @@ internal sealed class SpeakerAudioSampleCollector
var sample = new SpeakerAudioSample(sampleSegment.Speaker, sampleSegment, wavBytes, score.Value);
lock (gate)
{
if (requireNonOverlappingSamples &&
ReferenceEquals(pendingSpan, sampleSpan))
{
pendingSpan = null;
lastAcceptedEndBySpeaker[sampleSegment.Speaker] = sampleSegment.End;
}
if (!samplesBySpeaker.TryGetValue(sampleSegment.Speaker, out var samples))
{
samples = [];
@@ -163,35 +185,26 @@ internal sealed class SpeakerAudioSampleCollector
!string.Equals(speaker, "Unknown", StringComparison.OrdinalIgnoreCase);
}
private (TranscriptionSegment Segment, PendingSpanReset? Reset) ExtendPendingSpan(TranscriptionSegment segment)
private (SpeakerSampleSpan? Span, SpeakerSampleSpan? PreviousSpan) ExtendPendingSpan(TranscriptionSegment segment)
{
if (pendingSpan is null ||
!SpeakerSampleSpanSelector.CanExtend(pendingSpan.Speaker, pendingSpan.End, segment, maximumSegmentGap))
if (pendingSpan is not null && pendingSpan.TryExtend(segment, out var extended))
{
var reset = pendingSpan is null
? null
: new PendingSpanReset(
pendingSpan.Speaker,
pendingSpan.End,
segment.Start - pendingSpan.End);
pendingSpan = new PendingSpeakerSpan(
segment.Speaker,
segment.Start,
segment.End,
[segment.Text]);
return (pendingSpan.ToSegment(), reset);
pendingSpan = extended;
return (pendingSpan, null);
}
pendingSpan = pendingSpan.Extend(segment);
return (pendingSpan.ToSegment(), null);
var previousSpan = pendingSpan;
pendingSpan = SpeakerSampleSpan.Create(segment, maximumSampleDuration);
return (pendingSpan, previousSpan);
}
private static SampleScore Score(
TranscriptionSegment segment,
TimeSpan minimumUninterruptedSpeechDuration)
TimeSpan speechDuration,
TimeSpan minimumSampleSpeechDuration)
{
var durationSeconds = (segment.End - segment.Start).TotalSeconds;
if (durationSeconds < minimumUninterruptedSpeechDuration.TotalSeconds)
var durationSeconds = speechDuration.TotalSeconds;
if (durationSeconds < minimumSampleSpeechDuration.TotalSeconds)
{
return new SampleScore(false, 0, "speech duration is below the configured minimum", WordCount(segment.Text));
}
@@ -202,7 +215,7 @@ internal sealed class SpeakerAudioSampleCollector
return new SampleScore(false, 0, "word count is below the minimum useful sample length", words);
}
var durationScore = Math.Min(durationSeconds / Math.Max(1, minimumUninterruptedSpeechDuration.TotalSeconds), 2);
var durationScore = Math.Min(durationSeconds / Math.Max(1, minimumSampleSpeechDuration.TotalSeconds), 2);
var wordScore = Math.Min(words / 60.0, 1);
var sentenceBonus = segment.Text.TrimEnd().EndsWith('.') ||
segment.Text.TrimEnd().EndsWith('?') ||
@@ -221,30 +234,4 @@ internal sealed class SpeakerAudioSampleCollector
private sealed record SampleScore(bool Accepted, double Value, string? Reason, int WordCount);
private sealed record PendingSpanReset(string PreviousSpeaker, TimeSpan PreviousEnd, TimeSpan Gap);
private sealed record PendingSpeakerSpan(
string Speaker,
TimeSpan Start,
TimeSpan End,
IReadOnlyList<string> TextParts)
{
public PendingSpeakerSpan Extend(TranscriptionSegment segment)
{
return this with
{
End = segment.End > End ? segment.End : End,
TextParts = TextParts.Append(segment.Text).ToList()
};
}
public TranscriptionSegment ToSegment()
{
return new TranscriptionSegment(
Start,
End,
Speaker,
string.Join(' ', TextParts.Where(part => !string.IsNullOrWhiteSpace(part))));
}
}
}
@@ -49,6 +49,25 @@ public sealed class VaultTranscriptStore : ITranscriptStore
cancellationToken);
}
public Task AppendLinesAsync(
TranscriptSession session,
IReadOnlyList<string> lines,
CancellationToken cancellationToken)
{
if (lines.Count == 0)
{
return Task.CompletedTask;
}
return WithFileGateAsync(
session.TranscriptPath,
() => File.AppendAllTextAsync(
session.TranscriptPath,
string.Concat(lines.Select(line => line + Environment.NewLine)),
cancellationToken),
cancellationToken);
}
public async Task ReplaceLineAsync(
TranscriptSession session,
TranscriptLineReference lineReference,
@@ -11,7 +11,7 @@ public sealed class WindowsMeetingInactivityPromptService : IMeetingInactivityPr
private const string NotificationGroup = "meeting-inactivity";
private readonly ConcurrentDictionary<string, PendingPrompt> pendingPrompts = new(StringComparer.OrdinalIgnoreCase);
private readonly ILogger<WindowsMeetingInactivityPromptService> logger;
private readonly object registrationGate = new();
private readonly object notificationGate = new();
private bool registered;
public WindowsMeetingInactivityPromptService(ILogger<WindowsMeetingInactivityPromptService> logger)
@@ -37,24 +37,27 @@ public sealed class WindowsMeetingInactivityPromptService : IMeetingInactivityPr
try
{
EnsureRegistered();
var promptId = Guid.NewGuid().ToString("N");
pendingPrompts[promptId] = new PendingPrompt(handleResponseAsync);
logger.LogInformation(
"Registered native Windows inactivity notification {PromptId} response callback at threshold {Threshold}",
promptId,
request.Threshold);
var notification = BuildNotification(promptId, request);
notification.Show(toast =>
lock (notificationGate)
{
toast.Group = NotificationGroup;
toast.Tag = promptId;
toast.ExpirationTime = MeetingToastExpirationPolicy.StopReminderExpiration(DateTimeOffset.Now);
});
logger.LogInformation(
"Displayed native Windows inactivity notification {PromptId} at threshold {Threshold}",
promptId,
request.Threshold);
EnsureRegistered();
var promptId = Guid.NewGuid().ToString("N");
pendingPrompts[promptId] = new PendingPrompt(handleResponseAsync);
logger.LogInformation(
"Registered native Windows inactivity notification {PromptId} response callback at threshold {Threshold}",
promptId,
request.Threshold);
var notification = BuildNotification(promptId, request);
notification.Show(toast =>
{
toast.Group = NotificationGroup;
toast.Tag = promptId;
toast.ExpirationTime = MeetingToastExpirationPolicy.StopReminderExpiration(DateTimeOffset.Now);
});
logger.LogInformation(
"Displayed native Windows inactivity notification {PromptId} at threshold {Threshold}",
promptId,
request.Threshold);
}
}
catch (Exception exception)
{
@@ -76,6 +79,36 @@ public sealed class WindowsMeetingInactivityPromptService : IMeetingInactivityPr
return Task.CompletedTask;
}
public Task DismissAllAsync(CancellationToken cancellationToken)
{
cancellationToken.ThrowIfCancellationRequested();
lock (notificationGate)
{
if (pendingPrompts.IsEmpty)
{
return Task.CompletedTask;
}
pendingPrompts.Clear();
if (!OperatingSystem.IsWindowsVersionAtLeast(10, 0, 17763))
{
return Task.CompletedTask;
}
try
{
ToastNotificationManagerCompat.History.RemoveGroup(NotificationGroup);
logger.LogInformation("Dismissed all native Windows inactivity notifications");
}
catch (Exception exception)
{
logger.LogWarning(exception, "Failed to dismiss native Windows inactivity notifications");
}
return Task.CompletedTask;
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
@@ -83,29 +116,28 @@ public sealed class WindowsMeetingInactivityPromptService : IMeetingInactivityPr
public void Dispose()
{
if (!registered)
lock (notificationGate)
{
return;
}
if (!registered)
{
return;
}
try
{
ToastNotificationManagerCompat.OnActivated -= OnNotificationInvoked;
}
catch (Exception exception)
{
logger.LogWarning(exception, "Failed to detach native Windows toast notification activation handler");
try
{
ToastNotificationManagerCompat.OnActivated -= OnNotificationInvoked;
registered = false;
}
catch (Exception exception)
{
logger.LogWarning(exception, "Failed to detach native Windows toast notification activation handler");
}
}
}
private void EnsureRegistered()
{
if (registered)
{
return;
}
lock (registrationGate)
lock (notificationGate)
{
if (registered)
{
@@ -122,29 +154,25 @@ public sealed class WindowsMeetingInactivityPromptService : IMeetingInactivityPr
string promptId,
MeetingInactivityPromptRequest request)
{
var yesButton = new ToastButton()
.SetContent("Yes")
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.AddArgument("response", "stop")
.SetBackgroundActivation();
var noButton = new ToastButton()
.SetContent("No")
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.AddArgument("response", "continue")
.SetBackgroundActivation();
return new ToastContentBuilder()
var builder = new ToastContentBuilder()
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.SetToastScenario(ToastScenario.Reminder)
.SetToastDuration(ToastDuration.Long)
.AddText("Stop meeting?")
.AddText($"No transcript text has arrived for {FormatDuration(request.InactivityDuration)}.")
.AddButton(yesButton)
.AddButton(noButton);
.AddText($"No transcript text has arrived for {FormatDuration(request.InactivityDuration)}.");
foreach (var action in MeetingInactivityPromptActions.All)
{
builder.AddButton(new ToastButton()
.SetContent(action.Content)
.AddArgument("source", NotificationSource)
.AddArgument("promptId", promptId)
.AddArgument("response", action.ResponseArgument)
.SetBackgroundActivation());
}
return builder;
}
private void OnNotificationInvoked(ToastNotificationActivatedEventArgsCompat args)
@@ -177,10 +205,9 @@ public sealed class WindowsMeetingInactivityPromptService : IMeetingInactivityPr
return;
}
var response = TryGetArgument(arguments, "response", out var responseValue) &&
string.Equals(responseValue, "stop", StringComparison.OrdinalIgnoreCase)
? MeetingInactivityPromptResponse.Stop
: MeetingInactivityPromptResponse.Continue;
var response = TryGetArgument(arguments, "response", out var responseValue)
? MeetingInactivityPromptActions.ParseResponse(responseValue)
: MeetingInactivityPromptResponse.Continue;
_ = Task.Run(async () =>
{
try
@@ -0,0 +1,96 @@
#if WINDOWS
using System.Collections.Concurrent;
using CommunityToolkit.WinUI.Notifications;
namespace MeetingAssistant.Recording;
public sealed class WindowsMicrophoneSwitchPromptService : IMicrophoneSwitchPromptService, IDisposable
{
private const string Source = "meeting-microphone-switch";
private readonly ConcurrentDictionary<string, TaskCompletionSource<bool>> pending = new();
private readonly ILogger<WindowsMicrophoneSwitchPromptService> logger;
private readonly object gate = new();
private bool registered;
public WindowsMicrophoneSwitchPromptService(ILogger<WindowsMicrophoneSwitchPromptService> logger)
{
this.logger = logger;
}
public async Task<bool> ConfirmSwitchAsync(MicrophoneDevice microphone, CancellationToken cancellationToken)
{
if (!OperatingSystem.IsWindowsVersionAtLeast(10, 0, 17763)) return false;
cancellationToken.ThrowIfCancellationRequested();
var id = Guid.NewGuid().ToString("N");
var response = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
try
{
lock (gate)
{
if (!registered)
{
ToastNotificationManagerCompat.OnActivated += OnActivated;
registered = true;
}
pending[id] = response;
new ToastContentBuilder()
.AddArgument("source", Source)
.AddArgument("promptId", id)
.SetToastScenario(ToastScenario.Reminder)
.SetToastDuration(ToastDuration.Long)
.AddText("New microphone available")
.AddText($"Switch this meeting to {microphone.Name}?")
.AddButton(new ToastButton().SetContent("Yes")
.AddArgument("source", Source).AddArgument("promptId", id)
.AddArgument("response", "yes").SetBackgroundActivation())
.AddButton(new ToastButton().SetContent("No")
.AddArgument("source", Source).AddArgument("promptId", id)
.AddArgument("response", "no").SetBackgroundActivation())
.Show(toast =>
{
toast.Tag = id;
toast.Group = Source;
toast.ExpirationTime = DateTimeOffset.Now.AddMinutes(1);
});
}
logger.LogInformation("Offered newly available microphone {MicrophoneName}", microphone.Name);
return await response.Task.WaitAsync(cancellationToken);
}
finally
{
pending.TryRemove(id, out _);
try
{
ToastNotificationManagerCompat.History.Remove(id, Source);
}
catch (Exception exception)
{
logger.LogDebug(exception, "Could not remove microphone notification {PromptId}", id);
}
}
}
private void OnActivated(ToastNotificationActivatedEventArgsCompat args)
{
var arguments = NotificationActivationArguments.Parse(args.Argument);
if (arguments.GetValueOrDefault("source") != Source ||
!arguments.TryGetValue("promptId", out var id) ||
!pending.TryRemove(id, out var response)) return;
response.TrySetResult(arguments.GetValueOrDefault("response") == "yes");
}
public void Dispose()
{
lock (gate)
{
if (registered) ToastNotificationManagerCompat.OnActivated -= OnActivated;
registered = false;
foreach (var response in pending.Values) response.TrySetResult(false);
pending.Clear();
}
}
}
#endif
@@ -190,7 +190,7 @@ public sealed class PyannoteSpeakerIdentityMatchValidator : ISpeakerIdentityMatc
return [];
}
return await finalizer.FinalizeAsync(
return await finalizer.FinalizeEnabledAsync(
wavPath,
[new TranscriptionSegment(TimeSpan.Zero, duration, "Unknown", "speaker identity validation sample")],
options.Diarization,
@@ -0,0 +1,631 @@
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Transcription;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Speakers;
public sealed class ResemblyzerSpeakerIdentificationService : ISpeakerIdentificationService
{
private readonly IDbContextFactory<SpeakerIdentityDbContext> dbContextFactory;
private readonly ISpeakerSnippetExtractor snippetExtractor;
private readonly IResemblyzerVoiceEncoder encoder;
private readonly ResemblyzerVoiceClusterMatcher clusterMatcher;
private readonly ResemblyzerVoiceVectorOutlierPruner outlierPruner;
private readonly SpeakerIdentificationOptions options;
private readonly ResemblyzerSpeakerRecognitionOptions resemblyzerOptions;
private readonly ILogger<ResemblyzerSpeakerIdentificationService> logger;
public ResemblyzerSpeakerIdentificationService(
IDbContextFactory<SpeakerIdentityDbContext> dbContextFactory,
ISpeakerSnippetExtractor snippetExtractor,
IResemblyzerVoiceEncoder encoder,
ResemblyzerVoiceClusterMatcher clusterMatcher,
ResemblyzerVoiceVectorOutlierPruner outlierPruner,
IOptions<MeetingAssistantOptions> options,
ILogger<ResemblyzerSpeakerIdentificationService> logger)
{
this.dbContextFactory = dbContextFactory;
this.snippetExtractor = snippetExtractor;
this.encoder = encoder;
this.clusterMatcher = clusterMatcher;
this.outlierPruner = outlierPruner;
this.options = options.Value.SpeakerIdentification;
resemblyzerOptions = this.options.Resemblyzer;
this.logger = logger;
}
public Task<SpeakerIdentificationResult> IdentifyKnownSpeakersAsync(
SpeakerIdentificationRequest request,
CancellationToken cancellationToken)
{
return ProcessTranscriptAsync(request, final: false, allowAudioFallback: false, cancellationToken);
}
public Task<SpeakerIdentificationResult> IdentifyFinishedSpeakersAsync(
SpeakerIdentificationRequest request,
CancellationToken cancellationToken)
{
return ProcessTranscriptAsync(request, final: false, allowAudioFallback: true, cancellationToken);
}
public Task<SpeakerIdentificationResult> ProcessFinishedTranscriptAsync(
SpeakerIdentificationRequest request,
CancellationToken cancellationToken)
{
return ProcessTranscriptAsync(request, final: true, allowAudioFallback: true, cancellationToken);
}
public async Task ApplySpeakerOverrideAsync(
SpeakerIdentificationRequest request,
string sourceSpeaker,
string targetSpeaker,
CancellationToken cancellationToken)
{
if (!options.Enabled ||
string.IsNullOrWhiteSpace(sourceSpeaker) ||
string.IsNullOrWhiteSpace(targetSpeaker) ||
string.Equals(sourceSpeaker, targetSpeaker, StringComparison.OrdinalIgnoreCase))
{
return;
}
var sourceLabel = sourceSpeaker.Trim();
var targetName = targetSpeaker.Trim();
await using var context = await dbContextFactory.CreateDbContextAsync(cancellationToken);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, cancellationToken);
var identities = await LoadIdentities(context).ToListAsync(cancellationToken);
var target = identities
.Where(identity => SpeakerIdentityNaming.GetAcceptedNames(identity).Contains(targetName))
.OrderBy(identity => string.Equals(identity.CanonicalName, targetName, StringComparison.OrdinalIgnoreCase) ? 0 : 1)
.ThenBy(identity => identity.Id)
.FirstOrDefault();
var reference = CreateReference(request.MeetingNote, DateTimeOffset.UtcNow);
var sourceCandidate = identities
.Where(identity => string.IsNullOrWhiteSpace(identity.CanonicalName))
.Where(identity => identity.References.Any(existing => SpeakerIdentityReferences.IsSame(existing, reference)))
.Where(identity => SpeakerIdentityNaming.GetAcceptedNames(identity).Contains(targetName))
.OrderBy(identity => identity.Id)
.FirstOrDefault();
var vectors = await ResolveAvailableVectorsAsync(request, sourceLabel, cancellationToken);
if (target is null && sourceCandidate is null && vectors.Count == 0)
{
logger.LogWarning(
"Skipping Resemblyzer speaker override from {SourceSpeaker} to {TargetSpeaker} because no source evidence was available",
sourceLabel,
targetName);
return;
}
var now = DateTimeOffset.UtcNow;
if (target is null)
{
target = sourceCandidate ?? new SpeakerIdentity
{
CreatedAt = now
};
if (target.Id == 0)
{
context.SpeakerIdentities.Add(target);
}
}
else if (sourceCandidate is not null && sourceCandidate.Id != target.Id)
{
SpeakerIdentityMerger.MergeIntoAndPrune(
target,
sourceCandidate,
options.MaxSnippetsPerSpeaker,
resemblyzerOptions.MaxVectorsPerIdentity,
outlierPruner);
context.SpeakerIdentities.Remove(sourceCandidate);
}
target.CanonicalName = targetName;
SpeakerIdentityNaming.SetCandidates(target, [targetName]);
SpeakerIdentityReferences.AddIfMissing(target, reference, now);
outlierPruner.AddAndPrune(
target,
vectors,
now);
target.UpdatedAt = now;
await context.SaveChangesAsync(cancellationToken);
await SpeakerIdentityTranscriptAudit.AppendIdentifiedAsync(
target.References,
sourceLabel,
targetName,
cancellationToken);
}
public async Task DeleteSpeakerIdentityAsync(
string identity,
CancellationToken cancellationToken)
{
if (!options.Enabled || string.IsNullOrWhiteSpace(identity))
{
return;
}
await using var context = await dbContextFactory.CreateDbContextAsync(cancellationToken);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, cancellationToken);
var identities = await LoadIdentities(context).ToListAsync(cancellationToken);
var target = identities.FirstOrDefault(candidate =>
SpeakerIdentityNaming.GetAcceptedNames(candidate).Contains(identity.Trim()));
if (target is null)
{
return;
}
context.SpeakerIdentities.Remove(target);
await context.SaveChangesAsync(cancellationToken);
}
private async Task<SpeakerIdentificationResult> ProcessTranscriptAsync(
SpeakerIdentificationRequest request,
bool final,
bool allowAudioFallback,
CancellationToken cancellationToken)
{
if (!options.Enabled || request.Segments.Count == 0)
{
return EmptyResult(request.Segments);
}
await using var context = await dbContextFactory.CreateDbContextAsync(cancellationToken);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, cancellationToken);
var attendees = SpeakerIdentityNaming.NormalizeAttendees(request.MeetingNote.Frontmatter.Attendees);
var knownMappings = request.KnownSpeakerMappings ??
new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
var knownLabels = knownMappings.Keys.ToHashSet(StringComparer.OrdinalIgnoreCase);
if (final && knownMappings.Count > 0)
{
await PersistMappedSpeakerEvidenceAsync(
context,
request,
knownMappings,
cancellationToken);
}
var identifiedNames = knownMappings.Values
.Where(name => !string.IsNullOrWhiteSpace(name))
.Select(name => name.Trim())
.ToHashSet(StringComparer.OrdinalIgnoreCase);
foreach (var segmentSpeaker in request.Segments
.Select(segment => segment.Speaker)
.Where(speaker => !string.IsNullOrWhiteSpace(speaker) && !SpeakerIdentityNaming.IsDiarizedSpeakerLabel(speaker)))
{
identifiedNames.Add(segmentSpeaker.Trim());
}
var mappings = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
var attendeeMatches = new List<SpeakerIdentityAttendeeMatch>();
var pendingAudits = new List<PendingIdentificationAudit>();
var matchedAcceptedNames = identifiedNames.ToHashSet(StringComparer.OrdinalIgnoreCase);
var unmatchedSpeakers = new List<(string Speaker, IReadOnlyList<float[]> Vectors)>();
foreach (var speaker in request.Segments
.Select(segment => segment.Speaker)
.Where(speaker => !string.IsNullOrWhiteSpace(speaker))
.Distinct(StringComparer.OrdinalIgnoreCase))
{
if (knownLabels.Contains(speaker) || identifiedNames.Contains(speaker) || !SpeakerIdentityNaming.IsDiarizedSpeakerLabel(speaker))
{
continue;
}
var vectors = await ResolveAutomaticVectorsAsync(
request,
speaker,
allowAudioFallback,
cancellationToken);
if (vectors.Count < resemblyzerOptions.RequiredVectorsPerSpeaker)
{
logger.LogInformation(
"Resemblyzer matching waits for more vectors for {Speaker}: {VectorCount}/{RequiredVectorCount}",
speaker,
vectors.Count,
resemblyzerOptions.RequiredVectorsPerSpeaker);
continue;
}
var (identity, decision) = await FindMatchAsync(
context,
attendees,
identifiedNames,
vectors,
cancellationToken);
if (identity is null)
{
if (final && decision.Cohesion >= resemblyzerOptions.MinimumClusterCohesion)
{
unmatchedSpeakers.Add((speaker, vectors));
}
continue;
}
var now = DateTimeOffset.UtcNow;
var previousCanonicalName = identity.CanonicalName;
var previousReferenceCount = identity.References.Count;
outlierPruner.AddAndPrune(
identity,
vectors,
now);
SpeakerIdentityReferences.AddIfMissing(
identity,
CreateReference(request.MeetingNote, now),
now);
if (identity.References.Count != previousReferenceCount)
{
identity.UpdatedAt = now;
}
if (final)
{
UpdateMatchedIdentity(identity, attendees);
if (string.IsNullOrWhiteSpace(previousCanonicalName) &&
!string.IsNullOrWhiteSpace(identity.CanonicalName))
{
pendingAudits.Add(new PendingIdentificationAudit(
identity,
speaker,
identity.CanonicalName));
}
foreach (var acceptedName in SpeakerIdentityNaming.GetAcceptedNames(identity))
{
matchedAcceptedNames.Add(acceptedName);
}
}
var displayName = identity.GetDisplayName();
if (!string.IsNullOrWhiteSpace(displayName))
{
mappings[speaker] = displayName;
identifiedNames.Add(displayName);
foreach (var acceptedName in SpeakerIdentityNaming.GetAcceptedNames(identity))
{
identifiedNames.Add(acceptedName);
}
attendeeMatches.Add(new SpeakerIdentityAttendeeMatch(
displayName,
SpeakerIdentityNaming.GetAcceptedNames(identity).ToList()));
}
}
if (final)
{
LearnUnmatchedSpeakers(
context,
request.MeetingNote,
attendees,
matchedAcceptedNames,
unmatchedSpeakers,
pendingAudits);
}
await context.SaveChangesAsync(cancellationToken);
await AppendAuditsAsync(pendingAudits, cancellationToken);
var relabeled = request.Segments
.Select(segment => mappings.TryGetValue(segment.Speaker, out var name)
? segment with { Speaker = name }
: segment)
.ToList();
return new SpeakerIdentificationResult(relabeled, mappings, attendeeMatches);
}
private async Task<(SpeakerIdentity? Identity, ResemblyzerVoiceClusterMatchResult Decision)> FindMatchAsync(
SpeakerIdentityDbContext context,
IReadOnlyList<string> attendees,
IReadOnlySet<string> identifiedNames,
IReadOnlyList<float[]> queryVectors,
CancellationToken cancellationToken)
{
var activeCutoff = DateTimeOffset.UtcNow - options.MatchIdentityActiveAge;
var identities = await LoadIdentities(context)
.OrderByDescending(identity => identity.References.Count)
.ThenBy(identity => identity.Id)
.ToListAsync(cancellationToken);
var candidates = identities
.Select(identity => new
{
Identity = identity,
IsAttendee = SpeakerIdentityNaming.MatchesAttendees(identity, attendees),
IsActive = identity.UpdatedAt >= activeCutoff
})
.Where(candidate => candidate.IsAttendee || candidate.IsActive)
.Where(candidate => !SpeakerIdentityNaming.MatchesAnyAcceptedName(candidate.Identity, identifiedNames))
.Where(candidate => candidate.Identity.VoiceVectors.Any(vector =>
string.Equals(vector.ModelId, resemblyzerOptions.ModelId, StringComparison.Ordinal)))
.OrderByDescending(candidate => candidate.IsAttendee)
.ThenByDescending(candidate => candidate.Identity.ReferenceCount)
.ThenBy(candidate => candidate.Identity.Id)
.Take(Math.Max(1, options.MaxMatchCandidates))
.Select(candidate => new ResemblyzerVoiceVectorCandidate(
candidate.Identity.Id,
SpeakerVoiceVectors.DecodeCompatible(
candidate.Identity,
resemblyzerOptions.ModelId,
logger)))
.Where(candidate => candidate.Vectors.Count > 0)
.ToList();
var match = clusterMatcher.Match(queryVectors, candidates);
return (
match.IdentityId is { } identityId
? identities.Single(identity => identity.Id == identityId)
: null,
match);
}
private void LearnUnmatchedSpeakers(
SpeakerIdentityDbContext context,
MeetingNote meetingNote,
IReadOnlyList<string> attendees,
IReadOnlySet<string> matchedAcceptedNames,
IReadOnlyList<(string Speaker, IReadOnlyList<float[]> Vectors)> unmatchedSpeakers,
ICollection<PendingIdentificationAudit> pendingAudits)
{
var candidates = attendees
.Except(matchedAcceptedNames, StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.ToList();
if (candidates.Count == 0)
{
return;
}
foreach (var (speaker, vectors) in unmatchedSpeakers)
{
var now = DateTimeOffset.UtcNow;
var identity = new SpeakerIdentity
{
CanonicalName = candidates.Count == 1 ? candidates[0] : null,
CreatedAt = now,
UpdatedAt = now,
CandidateNames = candidates
.Select(name => new SpeakerCandidateName { Name = name })
.ToList(),
References = [CreateReference(meetingNote, now)]
};
outlierPruner.AddAndPrune(
identity,
vectors,
now);
context.SpeakerIdentities.Add(identity);
logger.LogInformation(
"Created Resemblyzer identity candidate for {Speaker} with {VectorCount} vector(s) and candidates {Candidates}",
speaker,
identity.VoiceVectors.Count,
string.Join(", ", candidates));
if (!string.IsNullOrWhiteSpace(identity.CanonicalName))
{
pendingAudits.Add(new PendingIdentificationAudit(
identity,
speaker,
identity.CanonicalName));
}
}
}
private async Task AppendAuditsAsync(
IEnumerable<PendingIdentificationAudit> pendingAudits,
CancellationToken cancellationToken)
{
foreach (var audit in pendingAudits)
{
try
{
await SpeakerIdentityTranscriptAudit.AppendIdentifiedAsync(
audit.Identity.References,
audit.Speaker,
audit.Name,
cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
logger.LogError(
exception,
"Resemblyzer identity {IdentityId} was saved, but its transcript identification audit could not be written",
audit.Identity.Id);
}
}
}
private async Task<IReadOnlyList<float[]>> ResolveAutomaticVectorsAsync(
SpeakerIdentificationRequest request,
string speaker,
bool allowAudioFallback,
CancellationToken cancellationToken)
{
var requiredCount = resemblyzerOptions.RequiredVectorsPerSpeaker;
var samples = await ResolveWavSamplesAsync(
request,
speaker,
resemblyzerOptions.MaxVectorsPerIdentity,
allowAudioFallback,
cancellationToken);
if (samples.Count < requiredCount)
{
return [];
}
return await encoder.EncodeAsync(samples, cancellationToken);
}
private async Task PersistMappedSpeakerEvidenceAsync(
SpeakerIdentityDbContext context,
SpeakerIdentificationRequest request,
IReadOnlyDictionary<string, string> knownMappings,
CancellationToken cancellationToken)
{
var identities = await LoadIdentities(context).ToListAsync(cancellationToken);
foreach (var (speaker, mappedName) in knownMappings)
{
var identity = identities
.Where(candidate => SpeakerIdentityNaming.GetAcceptedNames(candidate).Contains(mappedName))
.OrderBy(candidate =>
string.Equals(candidate.CanonicalName, mappedName, StringComparison.OrdinalIgnoreCase) ? 0 : 1)
.ThenBy(candidate => candidate.Id)
.FirstOrDefault();
if (identity is null)
{
logger.LogWarning(
"Could not retain final Resemblyzer evidence for mapped speaker {Speaker}: identity {MappedName} was not found",
speaker,
mappedName);
continue;
}
var vectors = await ResolveAvailableVectorsAsync(request, speaker, cancellationToken);
if (vectors.Count == 0)
{
continue;
}
var now = DateTimeOffset.UtcNow;
var previousReferenceCount = identity.References.Count;
var vectorUpdate = outlierPruner.AddAndPrune(
identity,
vectors,
now);
SpeakerIdentityReferences.AddIfMissing(
identity,
CreateReference(request.MeetingNote, now),
now);
if (identity.References.Count != previousReferenceCount)
{
identity.UpdatedAt = now;
}
logger.LogInformation(
"Retained {AddedVectorCount} new final Resemblyzer vector(s) for mapped speaker {Speaker} as identity {IdentityId}",
vectorUpdate.AddedCount,
speaker,
identity.Id);
}
}
private async Task<IReadOnlyList<float[]>> ResolveAvailableVectorsAsync(
SpeakerIdentificationRequest request,
string speaker,
CancellationToken cancellationToken)
{
var samples = await ResolveWavSamplesAsync(
request,
speaker,
resemblyzerOptions.MaxVectorsPerIdentity,
allowAudioFallback: true,
cancellationToken);
if (samples.Count == 0)
{
return [];
}
return await encoder.EncodeAsync(samples, cancellationToken);
}
private async Task<IReadOnlyList<byte[]>> ResolveWavSamplesAsync(
SpeakerIdentificationRequest request,
string speaker,
int maxSamples,
bool allowAudioFallback,
CancellationToken cancellationToken)
{
var suppliedSamples = request.Samples?
.Where(sample => string.Equals(sample.Speaker, speaker, StringComparison.OrdinalIgnoreCase))
.Where(sample => sample.WavBytes.Length > 0)
.OrderByDescending(sample => sample.Score)
.Take(maxSamples)
.ToList()
?? [];
var wavSamples = suppliedSamples
.Select(sample => sample.WavBytes)
.ToList();
if (!allowAudioFallback || wavSamples.Count >= maxSamples)
{
return wavSamples;
}
var spans = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpans(
request.Segments,
speaker,
options.MinimumSampleSpeechDuration,
options.MaximumSampleDuration,
request.Segments.Count);
foreach (var span in spans.Where(span => !OverlapsSuppliedSample(span, suppliedSamples)))
{
var wavBytes = await snippetExtractor.ExtractSnippetAsync(
request.AudioPath,
span,
cancellationToken);
if (wavBytes.Length > 0)
{
wavSamples.Add(wavBytes);
}
if (wavSamples.Count >= maxSamples)
{
break;
}
}
logger.LogInformation(
"Resolved {SampleCount}/{RequestedSampleCount} Resemblyzer WAV samples for {Speaker}: {SuppliedSampleCount} supplied, {ExtractedSampleCount} extracted from completed audio",
wavSamples.Count,
maxSamples,
speaker,
suppliedSamples.Count,
wavSamples.Count - suppliedSamples.Count);
return wavSamples;
}
private static bool OverlapsSuppliedSample(
IReadOnlyList<TranscriptionSegment> span,
IReadOnlyList<SpeakerAudioSample> suppliedSamples)
{
if (span.Count == 0)
{
return false;
}
var start = span[0].Start;
var end = span[^1].End;
return suppliedSamples.Any(sample => sample.Segment.Start < end && sample.Segment.End > start);
}
private static IQueryable<SpeakerIdentity> LoadIdentities(SpeakerIdentityDbContext context)
{
return context.SpeakerIdentities
.AsSplitQuery()
.Include(identity => identity.Aliases)
.Include(identity => identity.CandidateNames)
.Include(identity => identity.Snippets)
.Include(identity => identity.VoiceVectors)
.Include(identity => identity.References);
}
private void UpdateMatchedIdentity(SpeakerIdentity identity, IReadOnlyList<string> attendees)
{
identity.UpdatedAt = DateTimeOffset.UtcNow;
if (!string.IsNullOrWhiteSpace(identity.CanonicalName) || attendees.Count == 0)
{
return;
}
SpeakerIdentityNaming.UpdateCandidateNames(identity, attendees);
}
private static SpeakerIdentityReference CreateReference(MeetingNote meetingNote, DateTimeOffset timestamp)
{
return SpeakerIdentityReferences.Create(meetingNote.Path, meetingNote.Frontmatter.Transcript, timestamp);
}
private static SpeakerIdentificationResult EmptyResult(IReadOnlyList<TranscriptionSegment> segments)
{
return new SpeakerIdentificationResult(segments, new Dictionary<string, string>());
}
private sealed record PendingIdentificationAudit(
SpeakerIdentity Identity,
string Speaker,
string Name);
}
@@ -0,0 +1,172 @@
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Speakers;
public sealed class ResemblyzerSpeakerIdentityMergeService : ISpeakerIdentityMergeService
{
private readonly IDbContextFactory<SpeakerIdentityDbContext> dbContextFactory;
private readonly ResemblyzerVoiceClusterMatcher matcher;
private readonly ResemblyzerVoiceVectorOutlierPruner outlierPruner;
private readonly SpeakerIdentificationOptions options;
private readonly ResemblyzerSpeakerRecognitionOptions resemblyzerOptions;
private readonly ILogger<ResemblyzerSpeakerIdentityMergeService> logger;
public ResemblyzerSpeakerIdentityMergeService(
IDbContextFactory<SpeakerIdentityDbContext> dbContextFactory,
ResemblyzerVoiceClusterMatcher matcher,
ResemblyzerVoiceVectorOutlierPruner outlierPruner,
IOptions<MeetingAssistantOptions> options,
ILogger<ResemblyzerSpeakerIdentityMergeService> logger)
{
this.dbContextFactory = dbContextFactory;
this.matcher = matcher;
this.outlierPruner = outlierPruner;
this.options = options.Value.SpeakerIdentification;
resemblyzerOptions = this.options.Resemblyzer;
this.logger = logger;
}
public async Task<SpeakerIdentityMergeResult> MergeRecentIdentitiesAsync(
TimeSpan? recentIdentityAge,
CancellationToken cancellationToken)
{
await using var context = await dbContextFactory.CreateDbContextAsync(cancellationToken);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, cancellationToken);
var identities = await context.SpeakerIdentities
.AsSplitQuery()
.Include(identity => identity.Aliases)
.Include(identity => identity.CandidateNames)
.Include(identity => identity.Snippets)
.Include(identity => identity.VoiceVectors)
.Include(identity => identity.References)
.OrderByDescending(identity => identity.References.Count)
.ThenBy(identity => identity.Id)
.ToListAsync(cancellationToken);
var cutoff = DateTimeOffset.UtcNow - (recentIdentityAge ?? options.MergeRecentIdentityAge);
var recentIds = identities
.Where(identity => identity.CreatedAt >= cutoff)
.Select(identity => identity.Id)
.ToList();
var required = resemblyzerOptions.RequiredVectorsPerSpeaker;
var attempts = 0;
var mergedPairs = 0;
var pendingAudits = new List<PendingMergeAudit>();
foreach (var sourceId in recentIds)
{
var source = identities.SingleOrDefault(identity => identity.Id == sourceId);
if (source is null)
{
continue;
}
var sourceVectors = SpeakerVoiceVectors.DecodeCompatible(
source,
resemblyzerOptions.ModelId,
logger)
.Take(required * 2)
.ToList();
if (sourceVectors.Count < required * 2)
{
logger.LogInformation(
"Skipping Resemblyzer merge source identity {SourceIdentityId}: {VectorCount}/{RequiredVectorCount} compatible vectors",
source.Id,
sourceVectors.Count,
required * 2);
continue;
}
var candidates = identities
.Where(identity => identity.Id != source.Id)
.Where(identity => identity.VoiceVectors.Any(vector =>
string.Equals(vector.ModelId, resemblyzerOptions.ModelId, StringComparison.Ordinal)))
.Take(Math.Max(1, options.MaxMatchCandidates))
.Select(identity => new ResemblyzerVoiceVectorCandidate(
identity.Id,
SpeakerVoiceVectors.DecodeCompatible(
identity,
resemblyzerOptions.ModelId,
logger)))
.Where(candidate => candidate.Vectors.Count > 0)
.ToList();
if (candidates.Count == 0)
{
continue;
}
attempts++;
var first = matcher.Match(sourceVectors.Take(required).ToList(), candidates);
if (!first.Accepted || first.IdentityId is not { } targetId)
{
continue;
}
attempts++;
var second = matcher.Match(sourceVectors.Skip(required).Take(required).ToList(), candidates);
if (!second.Accepted || second.IdentityId != targetId)
{
logger.LogInformation(
"Rejected Resemblyzer merge for source identity {SourceIdentityId}: disjoint clusters selected {FirstIdentityId} and {SecondIdentityId}",
source.Id,
first.IdentityId,
second.IdentityId);
continue;
}
var target = identities.SingleOrDefault(identity => identity.Id == targetId);
if (target is null)
{
continue;
}
var targetName = target.GetDisplayName() ?? $"identity-{target.Id}";
var sourceName = source.GetDisplayName() ?? $"identity-{source.Id}";
SpeakerIdentityMerger.MergeIntoAndPrune(
target,
source,
options.MaxSnippetsPerSpeaker,
resemblyzerOptions.MaxVectorsPerIdentity,
outlierPruner);
pendingAudits.Add(new PendingMergeAudit(
target,
targetName,
sourceName));
context.SpeakerIdentities.Remove(source);
identities.Remove(source);
mergedPairs++;
}
await context.SaveChangesAsync(cancellationToken);
foreach (var audit in pendingAudits)
{
try
{
await SpeakerIdentityTranscriptAudit.AppendMergedAsync(
audit.Identity.References,
audit.TargetName,
audit.SourceName,
cancellationToken);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
logger.LogError(
exception,
"Resemblyzer identity merge was saved for target {IdentityId}, but its transcript audit could not be written",
audit.Identity.Id);
}
}
return new SpeakerIdentityMergeResult(
recentIds.Count,
identities.Count,
attempts,
mergedPairs);
}
private sealed record PendingMergeAudit(
SpeakerIdentity Identity,
string TargetName,
string SourceName);
}
@@ -0,0 +1,204 @@
namespace MeetingAssistant.Speakers;
public sealed record ResemblyzerVoiceVectorCandidate(
int IdentityId,
IReadOnlyList<float[]> Vectors);
public sealed record ResemblyzerVoiceClusterMatchResult(
int? IdentityId,
string Reason,
double Cohesion,
double? BestSimilarity,
double? RunnerUpSimilarity)
{
public bool Accepted => IdentityId.HasValue;
}
public sealed class ResemblyzerVoiceClusterMatcher
{
private readonly ResemblyzerSpeakerRecognitionOptions options;
private readonly ILogger<ResemblyzerVoiceClusterMatcher> logger;
public ResemblyzerVoiceClusterMatcher(
ResemblyzerSpeakerRecognitionOptions options,
ILogger<ResemblyzerVoiceClusterMatcher> logger)
{
this.options = options;
this.logger = logger;
}
public ResemblyzerVoiceClusterMatchResult Match(
IReadOnlyList<float[]> queryVectors,
IReadOnlyList<ResemblyzerVoiceVectorCandidate> candidates)
{
var requiredCount = options.RequiredVectorsPerSpeaker;
if (queryVectors.Count < requiredCount)
{
return Reject(
$"insufficient query vectors ({queryVectors.Count}/{requiredCount})",
cohesion: 0,
bestSimilarity: null,
runnerUpSimilarity: null);
}
var normalizedQuery = queryVectors
.Take(requiredCount)
.Select(vector => SpeakerVoiceVectors.Normalize(vector, "Voice vector"))
.ToList();
var cohesion = MeanPairwiseCosine(normalizedQuery);
if (cohesion < options.MinimumClusterCohesion)
{
return Reject(
$"query cohesion {cohesion:F4} is below {options.MinimumClusterCohesion:F4}",
cohesion,
bestSimilarity: null,
runnerUpSimilarity: null);
}
var scored = candidates
.Select(candidate => ScoreCandidate(normalizedQuery, candidate))
.Where(candidate => candidate is not null)
.Select(candidate => candidate!.Value)
.OrderByDescending(candidate => candidate.Similarity)
.ThenBy(candidate => candidate.IdentityId)
.ToList();
if (scored.Count == 0)
{
return Reject("no compatible identity vectors were available", cohesion, null, null);
}
var best = scored[0];
var runnerUp = scored.Count > 1 ? scored[1].Similarity : (double?)null;
if (best.Similarity < options.MinimumIdentitySimilarity)
{
return Reject(
$"best similarity {best.Similarity:F4} is below {options.MinimumIdentitySimilarity:F4}",
cohesion,
best.Similarity,
runnerUp);
}
if (runnerUp is { } runnerUpSimilarity &&
best.Similarity - runnerUpSimilarity < options.MinimumSimilarityMargin)
{
return Reject(
$"similarity margin {best.Similarity - runnerUpSimilarity:F4} is below {options.MinimumSimilarityMargin:F4}",
cohesion,
best.Similarity,
runnerUpSimilarity);
}
logger.LogInformation(
"Resemblyzer cluster accepted identity {IdentityId}: cohesion {Cohesion:F4}, similarity {Similarity:F4}, runner-up {RunnerUpSimilarity}",
best.IdentityId,
cohesion,
best.Similarity,
runnerUp);
return new ResemblyzerVoiceClusterMatchResult(
best.IdentityId,
"accepted",
cohesion,
best.Similarity,
runnerUp);
}
private ResemblyzerVoiceClusterMatchResult Reject(
string reason,
double cohesion,
double? bestSimilarity,
double? runnerUpSimilarity)
{
logger.LogInformation(
"Resemblyzer cluster rejected: {Reason}; cohesion {Cohesion:F4}, best {BestSimilarity}, runner-up {RunnerUpSimilarity}",
reason,
cohesion,
bestSimilarity,
runnerUpSimilarity);
return new ResemblyzerVoiceClusterMatchResult(
null,
reason,
cohesion,
bestSimilarity,
runnerUpSimilarity);
}
private (int IdentityId, double Similarity)? ScoreCandidate(
IReadOnlyList<float[]> normalizedQuery,
ResemblyzerVoiceVectorCandidate candidate)
{
if (candidate.Vectors.Count == 0)
{
return null;
}
try
{
var normalizedCandidate = candidate.Vectors
.Select(vector => SpeakerVoiceVectors.Normalize(vector, "Voice vector"))
.ToList();
var centroid = SpeakerVoiceVectors.Normalize(Centroid(normalizedCandidate), "Voice-vector centroid");
var similarities = normalizedQuery
.Select(vector => SpeakerVoiceVectors.Cosine(vector, centroid))
.Order()
.ToList();
return (candidate.IdentityId, Median(similarities));
}
catch (InvalidDataException exception)
{
logger.LogWarning(
exception,
"Skipping invalid Resemblyzer evidence for identity {IdentityId}",
candidate.IdentityId);
return null;
}
}
private static float[] Centroid(IReadOnlyList<float[]> vectors)
{
var centroid = new float[ResemblyzerVectorContract.Dimensions];
foreach (var vector in vectors)
{
for (var index = 0; index < centroid.Length; index++)
{
centroid[index] += vector[index];
}
}
for (var index = 0; index < centroid.Length; index++)
{
centroid[index] /= vectors.Count;
}
return centroid;
}
private static double MeanPairwiseCosine(IReadOnlyList<float[]> vectors)
{
if (vectors.Count < 2)
{
return 1;
}
double total = 0;
var pairs = 0;
for (var first = 0; first < vectors.Count - 1; first++)
{
for (var second = first + 1; second < vectors.Count; second++)
{
total += SpeakerVoiceVectors.Cosine(vectors[first], vectors[second]);
pairs++;
}
}
return total / pairs;
}
private static double Median(IReadOnlyList<double> sortedValues)
{
var middle = sortedValues.Count / 2;
return sortedValues.Count % 2 == 0
? (sortedValues[middle - 1] + sortedValues[middle]) / 2
: sortedValues[middle];
}
}
@@ -0,0 +1,208 @@
namespace MeetingAssistant.Speakers;
public sealed record ResemblyzerVoiceVectorUpdateResult(
int AddedCount,
int RemovedCount)
{
public bool Changed => AddedCount > 0 || RemovedCount > 0;
}
public sealed class ResemblyzerVoiceVectorOutlierPruner
{
private readonly ResemblyzerSpeakerRecognitionOptions options;
private readonly ILogger<ResemblyzerVoiceVectorOutlierPruner> logger;
public ResemblyzerVoiceVectorOutlierPruner(
ResemblyzerSpeakerRecognitionOptions options,
ILogger<ResemblyzerVoiceVectorOutlierPruner> logger)
{
this.options = options;
this.logger = logger;
}
public int Prune(SpeakerIdentity identity)
{
var compatible = new List<DecodedSpeakerVoiceVector>();
foreach (var entry in SpeakerVoiceVectors.DecodeCompatibleEntries(
identity,
options.ModelId,
logger))
{
try
{
compatible.Add(new DecodedSpeakerVoiceVector(
entry.Stored,
SpeakerVoiceVectors.Normalize(
entry.Vector,
$"Stored voice vector {entry.Stored.Id}")));
}
catch (InvalidDataException exception)
{
logger.LogWarning(
exception,
"Preserving invalid stored voice vector {VectorId} while pruning identity {IdentityId}",
entry.Stored.Id,
identity.Id);
}
}
var minimumVectors = options.OutlierPruningMinimumVectors;
if (compatible.Count < minimumVectors)
{
return 0;
}
var clusters = FindDensityClusters(compatible.Select(item => item.Vector).ToList());
var ranked = clusters
.GroupBy(cluster => cluster)
.Where(group => group.Key > 0)
.Select(group => new { Id = group.Key, Count = group.Count() })
.OrderByDescending(group => group.Count)
.ThenBy(group => group.Id)
.ToList();
if (ranked.Count == 0)
{
return Skip(identity, "no dense cluster was found");
}
if (ranked.Count > 1 && ranked[0].Count == ranked[1].Count)
{
return Skip(identity, "no uniquely largest dense cluster was found");
}
var dominant = ranked[0];
var ratio = (double)dominant.Count / compatible.Count;
if (ratio < options.OutlierPruningMinimumClusterRatio)
{
return Skip(
identity,
$"largest dense cluster ratio {ratio:F4} is below {options.OutlierPruningMinimumClusterRatio:F4}");
}
var removed = compatible
.Where((_, index) => clusters[index] != dominant.Id)
.Select(item => item.Stored)
.ToList();
foreach (var vector in removed)
{
identity.VoiceVectors.Remove(vector);
}
if (removed.Count > 0)
{
identity.UpdatedAt = DateTimeOffset.UtcNow;
}
logger.LogInformation(
"Pruned {RemovedCount} Resemblyzer voice-vector outliers from identity {IdentityId}; retained dominant cluster of {RetainedCount}/{CompatibleCount} vectors",
removed.Count,
identity.Id,
dominant.Count,
compatible.Count);
return removed.Count;
}
public ResemblyzerVoiceVectorUpdateResult AddAndPrune(
SpeakerIdentity identity,
IReadOnlyList<float[]> vectors,
DateTimeOffset createdAt)
{
var removedBeforeAdding = Prune(identity);
var added = SpeakerVoiceVectors.AddDistinct(
identity,
vectors,
options.ModelId,
options.MaxVectorsPerIdentity,
createdAt);
var removedAfterAdding = Prune(identity);
var result = new ResemblyzerVoiceVectorUpdateResult(
added,
removedBeforeAdding + removedAfterAdding);
if (result.Changed)
{
identity.UpdatedAt = createdAt;
}
return result;
}
private int[] FindDensityClusters(IReadOnlyList<float[]> vectors)
{
var neighbors = Enumerable.Range(0, vectors.Count)
.Select(index => Enumerable.Range(0, vectors.Count)
.Where(candidate => SpeakerVoiceVectors.Cosine(vectors[index], vectors[candidate]) >= options.OutlierPruningNeighborSimilarity)
.ToList())
.ToList();
var minimumNeighbors = options.OutlierPruningMinimumNeighbors;
var labels = new int[vectors.Count];
var clusterId = 0;
for (var point = 0; point < vectors.Count; point++)
{
if (labels[point] != 0)
{
continue;
}
if (neighbors[point].Count < minimumNeighbors)
{
labels[point] = -1;
continue;
}
clusterId++;
ExpandCluster(point, clusterId, neighbors, labels, minimumNeighbors);
}
return labels;
}
private static void ExpandCluster(
int seed,
int clusterId,
IReadOnlyList<List<int>> neighbors,
int[] labels,
int minimumNeighbors)
{
labels[seed] = clusterId;
var pending = new Queue<int>(neighbors[seed]);
var queued = neighbors[seed].ToHashSet();
while (pending.TryDequeue(out var point))
{
if (labels[point] == -1)
{
labels[point] = clusterId;
}
if (labels[point] != 0)
{
continue;
}
labels[point] = clusterId;
if (neighbors[point].Count < minimumNeighbors)
{
continue;
}
foreach (var neighbor in neighbors[point])
{
if (queued.Add(neighbor))
{
pending.Enqueue(neighbor);
}
}
}
}
private int Skip(
SpeakerIdentity identity,
string reason)
{
logger.LogInformation(
"Skipped Resemblyzer voice-vector pruning for identity {IdentityId}: {Reason}",
identity.Id,
reason);
return 0;
}
}
@@ -0,0 +1,54 @@
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Speakers;
public sealed class ResemblyzerWarmupHostedService : BackgroundService
{
private readonly IResemblyzerVoiceEncoder encoder;
private readonly bool speakerIdentificationEnabled;
private readonly ResemblyzerSpeakerRecognitionOptions options;
private readonly ILogger<ResemblyzerWarmupHostedService> logger;
public ResemblyzerWarmupHostedService(
IResemblyzerVoiceEncoder encoder,
IOptions<MeetingAssistantOptions> options,
ILogger<ResemblyzerWarmupHostedService> logger)
{
this.encoder = encoder;
speakerIdentificationEnabled = options.Value.SpeakerIdentification.Enabled;
this.options = options.Value.SpeakerIdentification.Resemblyzer;
this.logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
if (!speakerIdentificationEnabled || !options.Enabled)
{
return;
}
await Task.Yield();
try
{
logger.LogInformation(
"Starting Resemblyzer warm-up in runtime folder {RuntimeFolder} for model {ModelId}",
VaultPath.Resolve(options.RuntimeFolder),
options.ModelId);
await encoder.WarmUpAsync(stoppingToken);
logger.LogInformation(
"Finished Resemblyzer warm-up in runtime folder {RuntimeFolder} for model {ModelId}",
VaultPath.Resolve(options.RuntimeFolder),
options.ModelId);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
logger.LogInformation("Resemblyzer warm-up was cancelled during application shutdown");
}
catch (Exception exception)
{
logger.LogError(
exception,
"Resemblyzer warm-up failed in runtime folder {RuntimeFolder}; voice encoding can still retry on demand",
VaultPath.Resolve(options.RuntimeFolder));
}
}
}
+36 -1
View File
@@ -21,6 +21,8 @@ public sealed class SpeakerIdentity
public List<SpeakerSnippet> Snippets { get; set; } = [];
public List<SpeakerVoiceVector> VoiceVectors { get; set; } = [];
public List<SpeakerIdentityReference> References { get; set; } = [];
[NotMapped]
@@ -93,7 +95,7 @@ public static class SpeakerIdentityReferences
string.IsNullOrWhiteSpace(reference.TranscriptPath);
}
private static bool IsSame(SpeakerIdentityReference first, SpeakerIdentityReference second)
public static bool IsSame(SpeakerIdentityReference first, SpeakerIdentityReference second)
{
return string.Equals(first.MeetingNotePath, second.MeetingNotePath, StringComparison.OrdinalIgnoreCase) &&
string.Equals(first.TranscriptPath, second.TranscriptPath, StringComparison.OrdinalIgnoreCase);
@@ -124,6 +126,25 @@ public sealed class SpeakerSnippet
public DateTimeOffset CreatedAt { get; set; }
}
public sealed class SpeakerVoiceVector
{
public int Id { get; set; }
public int SpeakerIdentityId { get; set; }
public SpeakerIdentity? SpeakerIdentity { get; set; }
public string ModelId { get; set; } = "";
public int Dimensions { get; set; }
public byte[] VectorBytes { get; set; } = [];
public string Fingerprint { get; set; } = "";
public DateTimeOffset CreatedAt { get; set; }
}
public sealed class SpeakerAlias
{
public int Id { get; set; }
@@ -150,6 +171,8 @@ public sealed class SpeakerIdentityDbContext : DbContext
public DbSet<SpeakerSnippet> SpeakerSnippets => Set<SpeakerSnippet>();
public DbSet<SpeakerVoiceVector> SpeakerVoiceVectors => Set<SpeakerVoiceVector>();
public DbSet<SpeakerIdentityReference> SpeakerIdentityReferences => Set<SpeakerIdentityReference>();
protected override void OnModelCreating(ModelBuilder modelBuilder)
@@ -175,6 +198,11 @@ public sealed class SpeakerIdentityDbContext : DbContext
.HasForeignKey(snippet => snippet.SpeakerIdentityId)
.OnDelete(DeleteBehavior.Cascade);
entity.HasMany(identity => identity.VoiceVectors)
.WithOne(vector => vector.SpeakerIdentity)
.HasForeignKey(vector => vector.SpeakerIdentityId)
.OnDelete(DeleteBehavior.Cascade);
entity.HasMany(identity => identity.References)
.WithOne(reference => reference.SpeakerIdentity)
.HasForeignKey(reference => reference.SpeakerIdentityId)
@@ -189,6 +217,13 @@ public sealed class SpeakerIdentityDbContext : DbContext
.HasIndex(alias => new { alias.SpeakerIdentityId, alias.Name })
.IsUnique();
modelBuilder.Entity<SpeakerVoiceVector>()
.HasIndex(vector => new { vector.SpeakerIdentityId, vector.Fingerprint })
.IsUnique();
modelBuilder.Entity<SpeakerVoiceVector>()
.HasIndex(vector => new { vector.SpeakerIdentityId, vector.ModelId });
modelBuilder.Entity<SpeakerIdentityReference>()
.HasIndex(reference => new { reference.SpeakerIdentityId, reference.MeetingNotePath, reference.TranscriptPath })
.IsUnique();
@@ -20,17 +20,20 @@ public sealed class SpeakerIdentityMergeService : ISpeakerIdentityMergeService
{
private readonly IDbContextFactory<SpeakerIdentityDbContext> dbContextFactory;
private readonly ISpeakerIdentityMatcher matcher;
private readonly ResemblyzerVoiceVectorOutlierPruner outlierPruner;
private readonly SpeakerIdentificationOptions options;
private readonly ILogger<SpeakerIdentityMergeService> logger;
public SpeakerIdentityMergeService(
IDbContextFactory<SpeakerIdentityDbContext> dbContextFactory,
ISpeakerIdentityMatcher matcher,
ResemblyzerVoiceVectorOutlierPruner outlierPruner,
IOptions<MeetingAssistantOptions> options,
ILogger<SpeakerIdentityMergeService> logger)
{
this.dbContextFactory = dbContextFactory;
this.matcher = matcher;
this.outlierPruner = outlierPruner;
this.options = options.Value.SpeakerIdentification;
this.logger = logger;
}
@@ -44,9 +47,11 @@ public sealed class SpeakerIdentityMergeService : ISpeakerIdentityMergeService
var cutoff = DateTimeOffset.UtcNow - (recentIdentityAge ?? options.MergeRecentIdentityAge);
var identities = await context.SpeakerIdentities
.AsSplitQuery()
.Include(identity => identity.Aliases)
.Include(identity => identity.CandidateNames)
.Include(identity => identity.Snippets)
.Include(identity => identity.VoiceVectors)
.Include(identity => identity.References)
.OrderByDescending(identity => identity.References.Count)
.ThenBy(identity => identity.Id)
@@ -154,10 +159,12 @@ public sealed class SpeakerIdentityMergeService : ISpeakerIdentityMergeService
var targetName = target.GetDisplayName() ?? $"identity-{target.Id}";
var sourceName = source.GetDisplayName() ?? $"identity-{source.Id}";
SpeakerIdentityMerger.MergeInto(
SpeakerIdentityMerger.MergeIntoAndPrune(
target,
source,
options.MaxSnippetsPerSpeaker);
options.MaxSnippetsPerSpeaker,
options.Resemblyzer.MaxVectorsPerIdentity,
outlierPruner);
logger.LogInformation(
"Speaker identity merge diagnostics merging source identity {SourceIdentityId} ({SourceName}) into target identity {TargetIdentityId} ({TargetName})",
source.Id,
@@ -2,10 +2,24 @@ namespace MeetingAssistant.Speakers;
internal static class SpeakerIdentityMerger
{
public static void MergeIntoAndPrune(
SpeakerIdentity target,
SpeakerIdentity source,
int maxSnippets,
int maxVoiceVectors,
ResemblyzerVoiceVectorOutlierPruner outlierPruner)
{
outlierPruner.Prune(target);
outlierPruner.Prune(source);
MergeInto(target, source, maxSnippets, maxVoiceVectors);
outlierPruner.Prune(target);
}
public static void MergeInto(
SpeakerIdentity target,
SpeakerIdentity source,
int maxSnippets)
int maxSnippets,
int maxVoiceVectors = int.MaxValue)
{
AddAlias(target, source.CanonicalName);
foreach (var alias in source.Aliases)
@@ -35,6 +49,26 @@ internal static class SpeakerIdentityMerger
.ToList();
target.Snippets.Clear();
target.Snippets.AddRange(retainedSnippets);
var retainedVectors = target.VoiceVectors
.Concat(source.VoiceVectors)
.GroupBy(vector => vector.Fingerprint, StringComparer.Ordinal)
.Select(group => group.OrderByDescending(vector => vector.CreatedAt).First())
.OrderByDescending(vector => vector.CreatedAt)
.ThenBy(vector => vector.Fingerprint, StringComparer.Ordinal)
.Take(Math.Max(1, maxVoiceVectors))
.Select(vector => new SpeakerVoiceVector
{
SpeakerIdentity = target,
ModelId = vector.ModelId,
Dimensions = vector.Dimensions,
VectorBytes = vector.VectorBytes.ToArray(),
Fingerprint = vector.Fingerprint,
CreatedAt = vector.CreatedAt
})
.ToList();
target.VoiceVectors.Clear();
target.VoiceVectors.AddRange(retainedVectors);
target.UpdatedAt = DateTimeOffset.UtcNow;
}
@@ -0,0 +1,103 @@
using MeetingAssistant.MeetingNotes;
namespace MeetingAssistant.Speakers;
internal static class SpeakerIdentityNaming
{
public static IReadOnlySet<string> GetAcceptedNames(SpeakerIdentity identity)
{
return new[] { identity.CanonicalName }
.Concat(identity.Aliases.Select(alias => alias.Name))
.Concat(identity.CandidateNames.Select(candidate => candidate.Name))
.Where(name => !string.IsNullOrWhiteSpace(name))
.Select(name => name!.Trim())
.ToHashSet(StringComparer.OrdinalIgnoreCase);
}
public static bool MatchesAnyAcceptedName(
SpeakerIdentity identity,
IReadOnlySet<string> names)
{
return GetAcceptedNames(identity).Any(names.Contains);
}
public static bool MatchesAttendees(
SpeakerIdentity identity,
IReadOnlyList<string> attendees)
{
var attendeeSet = attendees.ToHashSet(StringComparer.OrdinalIgnoreCase);
return GetAcceptedNames(identity).Any(attendeeSet.Contains);
}
public static IReadOnlyList<string> NormalizeAttendees(IEnumerable<string> attendees)
{
return attendees
.Select(MeetingAttendeeNames.NormalizeDisplayName)
.Where(attendee => !string.IsNullOrWhiteSpace(attendee))
.Distinct(StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.ToList();
}
public static bool IsDiarizedSpeakerLabel(string speaker)
{
var normalized = speaker.Trim();
return normalized.Equals("Unknown", StringComparison.OrdinalIgnoreCase) ||
normalized.StartsWith("Guest", StringComparison.OrdinalIgnoreCase) ||
normalized.StartsWith("Speaker", StringComparison.OrdinalIgnoreCase);
}
public static bool UpdateCandidateNames(
SpeakerIdentity identity,
IReadOnlyList<string> attendees)
{
if (!string.IsNullOrWhiteSpace(identity.CanonicalName) || attendees.Count == 0)
{
return false;
}
var currentCandidates = identity.CandidateNames
.Select(candidate => candidate.Name)
.ToHashSet(StringComparer.OrdinalIgnoreCase);
var fallbackAliasCandidate = currentCandidates.Count == 1 ? currentCandidates.Single() : null;
var aliasToCandidate = identity.Aliases
.Where(alias => !string.IsNullOrWhiteSpace(alias.Name))
.SelectMany(alias => currentCandidates.Select(candidate => new { Alias = alias.Name, Candidate = candidate }))
.Where(pair => string.Equals(pair.Alias, pair.Candidate, StringComparison.OrdinalIgnoreCase) ||
pair.Alias.Contains(pair.Candidate, StringComparison.OrdinalIgnoreCase) ||
pair.Candidate.Contains(pair.Alias, StringComparison.OrdinalIgnoreCase))
.ToDictionary(pair => pair.Alias, pair => pair.Candidate, StringComparer.OrdinalIgnoreCase);
var intersection = attendees
.Select(attendee => currentCandidates.Contains(attendee)
? attendee
: aliasToCandidate.GetValueOrDefault(attendee) ??
(identity.Aliases.Any(alias => string.Equals(alias.Name, attendee, StringComparison.OrdinalIgnoreCase))
? fallbackAliasCandidate
: null))
.Where(candidate => !string.IsNullOrWhiteSpace(candidate))
.Select(candidate => candidate!)
.Distinct(StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.ToList();
var resetToAttendees = intersection.Count == 0;
SetCandidates(identity, resetToAttendees ? attendees : intersection);
if (intersection.Count == 1)
{
identity.CanonicalName = intersection[0];
}
return resetToAttendees;
}
public static void SetCandidates(
SpeakerIdentity identity,
IEnumerable<string> candidates)
{
identity.CandidateNames.Clear();
identity.CandidateNames.AddRange(candidates
.Where(candidate => !string.IsNullOrWhiteSpace(candidate))
.Distinct(StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.Select(candidate => new SpeakerCandidateName { Name = candidate }));
}
}
@@ -46,6 +46,33 @@ internal static class SpeakerIdentitySchema
ON "SpeakerIdentityReferences" ("SpeakerIdentityId", "MeetingNotePath", "TranscriptPath");
""",
cancellationToken);
await context.Database.ExecuteSqlRawAsync(
"""
CREATE TABLE IF NOT EXISTS "SpeakerVoiceVectors" (
"Id" INTEGER NOT NULL CONSTRAINT "PK_SpeakerVoiceVectors" PRIMARY KEY AUTOINCREMENT,
"SpeakerIdentityId" INTEGER NOT NULL,
"ModelId" TEXT NOT NULL,
"Dimensions" INTEGER NOT NULL,
"VectorBytes" BLOB NOT NULL,
"Fingerprint" TEXT NOT NULL,
"CreatedAt" TEXT NOT NULL,
CONSTRAINT "FK_SpeakerVoiceVectors_SpeakerIdentities_SpeakerIdentityId"
FOREIGN KEY ("SpeakerIdentityId") REFERENCES "SpeakerIdentities" ("Id") ON DELETE CASCADE
);
""",
cancellationToken);
await context.Database.ExecuteSqlRawAsync(
"""
CREATE UNIQUE INDEX IF NOT EXISTS "IX_SpeakerVoiceVectors_SpeakerIdentityId_Fingerprint"
ON "SpeakerVoiceVectors" ("SpeakerIdentityId", "Fingerprint");
""",
cancellationToken);
await context.Database.ExecuteSqlRawAsync(
"""
CREATE INDEX IF NOT EXISTS "IX_SpeakerVoiceVectors_SpeakerIdentityId_ModelId"
ON "SpeakerVoiceVectors" ("SpeakerIdentityId", "ModelId");
""",
cancellationToken);
}
private static async Task EnsureSpeakerIdentityTimestampColumnsAsync(
@@ -143,7 +143,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
target.CanonicalName = targetName;
target.UpdatedAt = now;
ResetCandidates(target, [targetName]);
SpeakerIdentityNaming.SetCandidates(target, [targetName]);
AddMeetingReference(target, meetingReference);
var snippetAdded = AddSnippetIfNeeded(target, snippet);
await context.SaveChangesAsync(cancellationToken);
@@ -207,7 +207,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
await using var context = await dbContextFactory.CreateDbContextAsync(cancellationToken);
await SpeakerIdentitySchema.EnsureCreatedOrUpdatedAsync(context, cancellationToken);
var attendees = NormalizeAttendees(request.MeetingNote.Frontmatter.Attendees);
var attendees = SpeakerIdentityNaming.NormalizeAttendees(request.MeetingNote.Frontmatter.Attendees);
var meetingReference = CreateReference(request.MeetingNote, DateTimeOffset.UtcNow);
var speakerMappings = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
var attendeeMatches = new List<SpeakerIdentityAttendeeMatch>();
@@ -222,7 +222,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
.ToHashSet(StringComparer.OrdinalIgnoreCase);
foreach (var speaker in request.Segments
.Select(segment => segment.Speaker)
.Where(speaker => !string.IsNullOrWhiteSpace(speaker) && !IsDiarizedSpeakerLabel(speaker)))
.Where(speaker => !string.IsNullOrWhiteSpace(speaker) && !SpeakerIdentityNaming.IsDiarizedSpeakerLabel(speaker)))
{
alreadyIdentifiedNames.Add(speaker.Trim());
}
@@ -360,7 +360,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
cancellationToken);
}
foreach (var acceptedName in GetAcceptedNames(identity))
foreach (var acceptedName in SpeakerIdentityNaming.GetAcceptedNames(identity))
{
matchedAcceptedNames.Add(acceptedName);
}
@@ -371,14 +371,14 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
{
speakerMappings[speaker] = speakerName;
alreadyIdentifiedNames.Add(speakerName);
foreach (var acceptedName in GetAcceptedNames(identity))
foreach (var acceptedName in SpeakerIdentityNaming.GetAcceptedNames(identity))
{
alreadyIdentifiedNames.Add(acceptedName);
}
attendeeMatches.Add(new SpeakerIdentityAttendeeMatch(
speakerName,
GetAcceptedNames(identity).ToList()));
SpeakerIdentityNaming.GetAcceptedNames(identity).ToList()));
}
}
@@ -440,7 +440,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
.Select(identity => new
{
Identity = identity,
IsAttendee = MatchesAttendees(identity, attendees),
IsAttendee = SpeakerIdentityNaming.MatchesAttendees(identity, attendees),
IsActive = identity.UpdatedAt >= activeCutoff
})
.Where(candidate => candidate.IsAttendee || candidate.IsActive)
@@ -541,11 +541,11 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
string operation,
CancellationToken cancellationToken)
{
var span = SpeakerSampleSpanSelector.SelectBestContinuousSpan(
var span = SpeakerSampleSpanSelector.SelectBestSameSpeakerSpan(
request.Segments,
speaker,
options.MaximumSampleSegmentGap,
options.MinimumSampleSpeechDuration);
options.MinimumSampleSpeechDuration,
options.MaximumSampleDuration);
logger.LogInformation(
"{Operation} extracting fallback sample for {Speaker}: selected {SegmentCount} segment(s), span {SpanDuration}, minimum {MinimumDuration}",
operation,
@@ -569,7 +569,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
.Include(identity => identity.References)
.ToListAsync(cancellationToken);
return identities
.Where(identity => GetAcceptedNames(identity).Contains(name))
.Where(identity => SpeakerIdentityNaming.GetAcceptedNames(identity).Contains(name))
.OrderBy(identity => string.Equals(identity.CanonicalName, name, StringComparison.OrdinalIgnoreCase) ? 0 : 1)
.ThenBy(identity => identity.Id)
.FirstOrDefault();
@@ -589,7 +589,7 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
.Where(identity => string.IsNullOrWhiteSpace(identity.CanonicalName))
.ToListAsync(cancellationToken);
return candidates
.Where(identity => identity.References.Any(existing => IsSameReference(existing, reference)))
.Where(identity => identity.References.Any(existing => SpeakerIdentityReferences.IsSame(existing, reference)))
.Where(identity => identity.CandidateNames.Any(candidate =>
string.Equals(candidate.Name, targetName, StringComparison.OrdinalIgnoreCase)) ||
identity.Aliases.Any(alias =>
@@ -646,27 +646,11 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
identity.Aliases.Add(new SpeakerAlias { Name = alias.Trim() });
}
private static bool IsSameReference(
SpeakerIdentityReference first,
SpeakerIdentityReference second)
{
return string.Equals(first.MeetingNotePath, second.MeetingNotePath, StringComparison.OrdinalIgnoreCase) &&
string.Equals(first.TranscriptPath, second.TranscriptPath, StringComparison.OrdinalIgnoreCase);
}
private static bool MatchesAcceptedNames(
SpeakerIdentity identity,
IReadOnlySet<string> names)
{
return GetAcceptedNames(identity).Any(names.Contains);
}
private static bool MatchesAttendees(
SpeakerIdentity identity,
IReadOnlyList<string> attendees)
{
var attendeeSet = attendees.ToHashSet(StringComparer.OrdinalIgnoreCase);
return GetAcceptedNames(identity).Any(attendeeSet.Contains);
return SpeakerIdentityNaming.MatchesAnyAcceptedName(identity, names);
}
private static Task<SpeakerIdentity?> LoadIdentityAsync(
@@ -694,48 +678,25 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
var currentCandidates = identity.CandidateNames
.Select(candidate => candidate.Name)
.ToHashSet(StringComparer.OrdinalIgnoreCase);
var fallbackAliasCandidate = currentCandidates.Count == 1 ? currentCandidates.Single() : null;
var aliasToCandidate = identity.Aliases
.Where(alias => !string.IsNullOrWhiteSpace(alias.Name))
.SelectMany(alias => currentCandidates.Select(candidate => new { Alias = alias.Name, Candidate = candidate }))
.Where(pair => string.Equals(pair.Alias, pair.Candidate, StringComparison.OrdinalIgnoreCase) ||
pair.Alias.Contains(pair.Candidate, StringComparison.OrdinalIgnoreCase) ||
pair.Candidate.Contains(pair.Alias, StringComparison.OrdinalIgnoreCase))
.ToDictionary(pair => pair.Alias, pair => pair.Candidate, StringComparer.OrdinalIgnoreCase);
var intersection = attendees
.Select(attendee => currentCandidates.Contains(attendee)
? attendee
: aliasToCandidate.GetValueOrDefault(attendee) ??
(identity.Aliases.Any(alias => string.Equals(alias.Name, attendee, StringComparison.OrdinalIgnoreCase))
? fallbackAliasCandidate
: null))
.Where(candidate => !string.IsNullOrWhiteSpace(candidate))
.Select(candidate => candidate!)
.Distinct(StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.ToList();
if (intersection.Count == 0)
var resetToAttendees = SpeakerIdentityNaming.UpdateCandidateNames(identity, attendees);
if (resetToAttendees)
{
logger.LogInformation(
"Speaker identity candidate elimination for identity {IdentityId} had empty intersection; resetting candidates to attendees {Attendees} and replacing oldest snippet",
identity.Id,
FormatNames(attendees));
ResetCandidates(identity, attendees);
ReplaceOldestSnippet(identity, snippet);
return;
}
logger.LogInformation(
"Speaker identity candidate elimination for identity {IdentityId}: candidates {CurrentCandidates}, attendees {Attendees}, intersection {Intersection}",
"Speaker identity candidate elimination for identity {IdentityId}: candidates {CurrentCandidates}, attendees {Attendees}, remaining {RemainingCandidates}",
identity.Id,
FormatNames(currentCandidates),
FormatNames(attendees),
FormatNames(intersection));
ResetCandidates(identity, intersection);
if (intersection.Count == 1)
FormatNames(identity.CandidateNames.Select(candidate => candidate.Name)));
if (!string.IsNullOrWhiteSpace(identity.CanonicalName))
{
identity.CanonicalName = intersection[0];
logger.LogInformation(
"Speaker identity candidate elimination promoted identity {IdentityId} to canonical name {CanonicalName}",
identity.Id,
@@ -872,51 +833,6 @@ public sealed class SpeakerIdentityService : ISpeakerIdentificationService
}
}
private static void ResetCandidates(SpeakerIdentity identity, IReadOnlyList<string> candidates)
{
identity.CandidateNames.Clear();
identity.CandidateNames.AddRange(candidates
.Distinct(StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.Select(candidate => new SpeakerCandidateName { Name = candidate }));
}
private static IReadOnlySet<string> GetAcceptedNames(SpeakerIdentity identity)
{
return new[]
{
identity.CanonicalName
}
.Concat(identity.Aliases.Select(alias => alias.Name))
.Concat(identity.CandidateNames.Select(candidate => candidate.Name))
.Where(name => !string.IsNullOrWhiteSpace(name))
.Select(name => name!.Trim())
.ToHashSet(StringComparer.OrdinalIgnoreCase);
}
private static IReadOnlyList<string> NormalizeAttendees(IEnumerable<string> attendees)
{
return attendees
.Select(NormalizeAttendee)
.Where(attendee => !string.IsNullOrWhiteSpace(attendee))
.Distinct(StringComparer.OrdinalIgnoreCase)
.Order(StringComparer.OrdinalIgnoreCase)
.ToList();
}
private static string NormalizeAttendee(string attendee)
{
return MeetingAttendeeNames.NormalizeDisplayName(attendee);
}
private static bool IsDiarizedSpeakerLabel(string speaker)
{
var normalized = speaker.Trim();
return normalized.Equals("Unknown", StringComparison.OrdinalIgnoreCase) ||
normalized.StartsWith("Guest", StringComparison.OrdinalIgnoreCase) ||
normalized.StartsWith("Speaker", StringComparison.OrdinalIgnoreCase);
}
private static SpeakerIdentityReference CreateReference(MeetingNote meetingNote, DateTimeOffset timestamp)
{
return SpeakerIdentityReferences.Create(
@@ -0,0 +1,154 @@
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Speakers;
public sealed record SpeakerSampleCollectionPolicy(
int MinimumRetainedSamples,
bool RequireNonOverlappingSamples)
{
public static SpeakerSampleCollectionPolicy ExistingBackend { get; } = new(1, false);
public static SpeakerSampleCollectionPolicy IndependentVectors(int maximumVectors)
{
return new SpeakerSampleCollectionPolicy(Math.Max(1, maximumVectors), true);
}
public int ResolveRetainedSampleLimit(int configuredLimit)
{
return Math.Max(Math.Max(1, configuredLimit), MinimumRetainedSamples);
}
}
internal static class SpeakerSampleDurationConfiguration
{
public static void ValidateOrThrow(
SpeakerIdentificationOptions options,
string context)
{
ValidateOrThrow(
options.MinimumSampleSpeechDuration,
options.MaximumSampleDuration,
context);
}
public static void ValidateOrThrow(
TimeSpan minimumDuration,
TimeSpan maximumDuration,
string context)
{
var error = GetError(minimumDuration, maximumDuration);
if (error is not null)
{
throw new InvalidOperationException($"{context}: {error}");
}
}
public static string? GetError(
TimeSpan minimumDuration,
TimeSpan maximumDuration)
{
if (minimumDuration < TimeSpan.Zero)
{
return "SpeakerIdentification:MinimumSampleSpeechDuration must not be negative.";
}
if (maximumDuration <= TimeSpan.Zero)
{
return "SpeakerIdentification:MaximumSampleDuration must be greater than zero.";
}
return minimumDuration > maximumDuration
? "SpeakerIdentification:MinimumSampleSpeechDuration must not exceed SpeakerIdentification:MaximumSampleDuration."
: null;
}
}
internal static class ResemblyzerSpeakerRecognitionConfiguration
{
public static void ValidateOrThrow(
ResemblyzerSpeakerRecognitionOptions options,
string context)
{
var error = GetError(options);
if (error is not null)
{
throw new InvalidOperationException($"{context}: {error}");
}
}
public static string? GetError(ResemblyzerSpeakerRecognitionOptions options)
{
if (options.RequiredVectorsPerSpeaker < 1)
{
return "SpeakerIdentification:Resemblyzer:RequiredVectorsPerSpeaker must be at least one.";
}
if (options.MaxVectorsPerIdentity < options.RequiredVectorsPerSpeaker)
{
return "SpeakerIdentification:Resemblyzer:MaxVectorsPerIdentity must not be less than RequiredVectorsPerSpeaker.";
}
if (options.OutlierPruningMinimumVectors < options.RequiredVectorsPerSpeaker)
{
return "SpeakerIdentification:Resemblyzer:OutlierPruningMinimumVectors must not be less than RequiredVectorsPerSpeaker.";
}
if (options.OutlierPruningMinimumVectors > options.MaxVectorsPerIdentity)
{
return "SpeakerIdentification:Resemblyzer:OutlierPruningMinimumVectors must not exceed MaxVectorsPerIdentity.";
}
if (options.OutlierPruningMinimumNeighbors < 1 ||
options.OutlierPruningMinimumNeighbors > options.OutlierPruningMinimumVectors)
{
return "SpeakerIdentification:Resemblyzer:OutlierPruningMinimumNeighbors must be between one and OutlierPruningMinimumVectors.";
}
if (!IsCosine(options.OutlierPruningNeighborSimilarity))
{
return "SpeakerIdentification:Resemblyzer:OutlierPruningNeighborSimilarity must be finite and between -1 and 1.";
}
if (!double.IsFinite(options.OutlierPruningMinimumClusterRatio) ||
options.OutlierPruningMinimumClusterRatio is <= 0 or > 1)
{
return "SpeakerIdentification:Resemblyzer:OutlierPruningMinimumClusterRatio must be finite, greater than zero, and at most one.";
}
if (!IsCosine(options.MinimumClusterCohesion))
{
return "SpeakerIdentification:Resemblyzer:MinimumClusterCohesion must be finite and between -1 and 1.";
}
if (!IsCosine(options.MinimumIdentitySimilarity))
{
return "SpeakerIdentification:Resemblyzer:MinimumIdentitySimilarity must be finite and between -1 and 1.";
}
return !double.IsFinite(options.MinimumSimilarityMargin) ||
options.MinimumSimilarityMargin is < 0 or > 2
? "SpeakerIdentification:Resemblyzer:MinimumSimilarityMargin must be finite and between 0 and 2."
: null;
}
private static bool IsCosine(double value)
{
return double.IsFinite(value) && value is >= -1 and <= 1;
}
}
internal sealed class MeetingAssistantSpeakerSampleOptionsValidator
: IValidateOptions<MeetingAssistantOptions>
{
public ValidateOptionsResult Validate(string? name, MeetingAssistantOptions options)
{
var error = SpeakerSampleDurationConfiguration.GetError(
options.SpeakerIdentification.MinimumSampleSpeechDuration,
options.SpeakerIdentification.MaximumSampleDuration) ??
ResemblyzerSpeakerRecognitionConfiguration.GetError(
options.SpeakerIdentification.Resemblyzer);
return error is null
? ValidateOptionsResult.Success
: ValidateOptionsResult.Fail(error);
}
}
@@ -0,0 +1,82 @@
using MeetingAssistant.Transcription;
namespace MeetingAssistant.Speakers;
internal sealed record SpeakerSampleSpan(
string Speaker,
TimeSpan Start,
TimeSpan End,
TimeSpan SpeechDuration,
TimeSpan MaximumDuration,
IReadOnlyList<TranscriptionSegment> Segments)
{
public static SpeakerSampleSpan? Create(
TranscriptionSegment segment,
TimeSpan maximumDuration)
{
if (maximumDuration <= TimeSpan.Zero)
{
throw new ArgumentOutOfRangeException(
nameof(maximumDuration),
maximumDuration,
"Speaker sample maximum duration must be greater than zero.");
}
var boundedEnd = Min(segment.End, segment.Start + maximumDuration);
return boundedEnd > segment.Start
? new SpeakerSampleSpan(
segment.Speaker,
segment.Start,
boundedEnd,
boundedEnd - segment.Start,
maximumDuration,
[segment with { End = boundedEnd }])
: null;
}
public bool TryExtend(
TranscriptionSegment segment,
out SpeakerSampleSpan extended)
{
extended = this;
if (!string.Equals(Speaker, segment.Speaker, StringComparison.OrdinalIgnoreCase) ||
segment.Start >= Start + MaximumDuration)
{
return false;
}
var boundedEnd = Min(segment.End, Start + MaximumDuration);
if (boundedEnd <= segment.Start)
{
return false;
}
var uncoveredStart = segment.Start > End ? segment.Start : End;
var additionalSpeechDuration = boundedEnd > uncoveredStart
? boundedEnd - uncoveredStart
: TimeSpan.Zero;
extended = this with
{
End = boundedEnd > End ? boundedEnd : End,
SpeechDuration = SpeechDuration + additionalSpeechDuration,
Segments = Segments.Append(segment with { End = boundedEnd }).ToList()
};
return true;
}
public TranscriptionSegment ToSegment()
{
return new TranscriptionSegment(
Start,
End,
Speaker,
string.Join(' ', Segments
.Select(segment => segment.Text)
.Where(text => !string.IsNullOrWhiteSpace(text))));
}
private static TimeSpan Min(TimeSpan left, TimeSpan right)
{
return left < right ? left : right;
}
}
@@ -4,56 +4,84 @@ namespace MeetingAssistant.Speakers;
internal static class SpeakerSampleSpanSelector
{
public static bool CanExtend(
string currentSpeaker,
TimeSpan currentEnd,
TranscriptionSegment nextSegment,
TimeSpan maximumSegmentGap)
{
return string.Equals(currentSpeaker, nextSegment.Speaker, StringComparison.OrdinalIgnoreCase) &&
nextSegment.Start - currentEnd <= maximumSegmentGap;
}
public static IReadOnlyList<TranscriptionSegment> SelectBestContinuousSpan(
public static IReadOnlyList<TranscriptionSegment> SelectBestSameSpeakerSpan(
IReadOnlyList<TranscriptionSegment> segments,
string speaker,
TimeSpan maximumSegmentGap,
TimeSpan minimumDuration)
TimeSpan minimumDuration,
TimeSpan maximumSampleDuration)
{
var best = new List<TranscriptionSegment>();
var current = new List<TranscriptionSegment>();
SpeakerSampleSpan? best = null;
SpeakerSampleSpan? current = null;
foreach (var segment in segments.OrderBy(segment => segment.Start))
{
if (current.Count == 0)
if (!IsSpeaker(segment, speaker))
{
if (IsSpeaker(segment, speaker))
{
current.Add(segment);
best = LongerSpan(current, best);
}
current = null;
continue;
}
if (!CanExtend(speaker, current[^1].End, segment, maximumSegmentGap))
current = ExtendOrStart(current, segment, maximumSampleDuration);
if (current is not null &&
(best is null || current.SpeechDuration > best.SpeechDuration))
{
current.Clear();
if (!IsSpeaker(segment, speaker))
{
continue;
}
best = current;
}
current.Add(segment);
best = LongerSpan(current, best);
}
return SpanDuration(best) >= minimumDuration
? best
return best is not null && best.SpeechDuration >= minimumDuration
? best.Segments
: [];
}
public static IReadOnlyList<IReadOnlyList<TranscriptionSegment>> SelectBestSameSpeakerSpans(
IReadOnlyList<TranscriptionSegment> segments,
string speaker,
TimeSpan minimumDuration,
TimeSpan maximumSampleDuration,
int maxSpans)
{
if (maxSpans <= 0)
{
return [];
}
var completed = new List<SpeakerSampleSpan>();
SpeakerSampleSpan? current = null;
TimeSpan? lastCompletedEnd = null;
foreach (var segment in segments.OrderBy(segment => segment.Start))
{
if (!IsSpeaker(segment, speaker))
{
current = null;
continue;
}
if (lastCompletedEnd is { } end && segment.Start < end)
{
current = null;
continue;
}
current = ExtendOrStart(current, segment, maximumSampleDuration);
if (current is null || current.SpeechDuration < minimumDuration)
{
continue;
}
completed.Add(current);
lastCompletedEnd = current.End;
current = null;
}
return completed
.OrderByDescending(span => span.SpeechDuration)
.ThenBy(span => span.Start)
.Take(maxSpans)
.OrderBy(span => span.Start)
.Select(span => span.Segments)
.ToList();
}
public static TimeSpan SpanDuration(IReadOnlyList<TranscriptionSegment> segments)
{
return segments.Count == 0
@@ -61,19 +89,20 @@ internal static class SpeakerSampleSpanSelector
: segments[^1].End - segments[0].Start;
}
private static SpeakerSampleSpan? ExtendOrStart(
SpeakerSampleSpan? current,
TranscriptionSegment segment,
TimeSpan maximumSampleDuration)
{
return current is not null && current.TryExtend(segment, out var extended)
? extended
: SpeakerSampleSpan.Create(segment, maximumSampleDuration);
}
private static bool IsSpeaker(
TranscriptionSegment segment,
string speaker)
{
return string.Equals(segment.Speaker, speaker, StringComparison.OrdinalIgnoreCase);
}
private static List<TranscriptionSegment> LongerSpan(
List<TranscriptionSegment> current,
List<TranscriptionSegment> best)
{
return SpanDuration(current) > SpanDuration(best)
? current.ToList()
: best;
}
}
@@ -0,0 +1,191 @@
using System.Buffers.Binary;
using System.Security.Cryptography;
using System.Text;
namespace MeetingAssistant.Speakers;
internal sealed record DecodedSpeakerVoiceVector(
SpeakerVoiceVector Stored,
float[] Vector);
internal static class ResemblyzerVectorContract
{
public const int Dimensions = 256;
}
internal static class SpeakerVoiceVectors
{
public static float[] Decode(SpeakerVoiceVector stored)
{
if (stored.Dimensions != ResemblyzerVectorContract.Dimensions ||
stored.VectorBytes.Length != stored.Dimensions * sizeof(float))
{
throw new InvalidDataException(
$"Stored voice vector {stored.Id} has invalid dimension or byte length metadata.");
}
var vector = new float[stored.Dimensions];
for (var index = 0; index < vector.Length; index++)
{
vector[index] = BinaryPrimitives.ReadSingleLittleEndian(
stored.VectorBytes.AsSpan(index * sizeof(float), sizeof(float)));
if (!float.IsFinite(vector[index]))
{
throw new InvalidDataException($"Stored voice vector {stored.Id} contains a non-finite value.");
}
}
return vector;
}
public static int AddDistinct(
SpeakerIdentity identity,
IEnumerable<float[]> vectors,
string modelId,
int maximumCount,
DateTimeOffset createdAt)
{
var limit = Math.Max(1, maximumCount);
var fingerprints = identity.VoiceVectors
.Select(vector => vector.Fingerprint)
.ToHashSet(StringComparer.Ordinal);
var added = 0;
foreach (var vector in vectors)
{
if (identity.VoiceVectors.Count >= limit)
{
break;
}
var bytes = Encode(vector);
var fingerprint = Fingerprint(modelId, bytes);
if (!fingerprints.Add(fingerprint))
{
continue;
}
identity.VoiceVectors.Add(new SpeakerVoiceVector
{
ModelId = modelId,
Dimensions = vector.Length,
VectorBytes = bytes,
Fingerprint = fingerprint,
CreatedAt = createdAt
});
added++;
}
return added;
}
public static IReadOnlyList<float[]> DecodeCompatible(
SpeakerIdentity identity,
string modelId,
ILogger logger)
{
return DecodeCompatibleEntries(identity, modelId, logger)
.Select(entry => entry.Vector)
.ToList();
}
public static IReadOnlyList<DecodedSpeakerVoiceVector> DecodeCompatibleEntries(
SpeakerIdentity identity,
string modelId,
ILogger logger)
{
var vectors = new List<DecodedSpeakerVoiceVector>();
foreach (var stored in identity.VoiceVectors
.Where(vector => string.Equals(vector.ModelId, modelId, StringComparison.Ordinal))
.OrderBy(vector => vector.CreatedAt)
.ThenBy(vector => vector.Id))
{
try
{
vectors.Add(new DecodedSpeakerVoiceVector(stored, Decode(stored)));
}
catch (InvalidDataException exception)
{
logger.LogWarning(
exception,
"Ignoring invalid stored voice vector {VectorId} for identity {IdentityId}",
stored.Id,
identity.Id);
}
}
return vectors;
}
public static double Cosine(float[] first, float[] second)
{
double dot = 0;
for (var index = 0; index < first.Length; index++)
{
dot += first[index] * second[index];
}
return dot;
}
public static float[] Normalize(float[]? vector, string description)
{
if (vector is null || vector.Length != ResemblyzerVectorContract.Dimensions)
{
throw new InvalidDataException(
$"{description} has {vector?.Length ?? 0} dimensions; expected {ResemblyzerVectorContract.Dimensions}.");
}
double magnitudeSquared = 0;
foreach (var value in vector)
{
if (!float.IsFinite(value))
{
throw new InvalidDataException($"{description} contains a non-finite value.");
}
magnitudeSquared += value * value;
}
var magnitude = Math.Sqrt(magnitudeSquared);
if (magnitude <= double.Epsilon)
{
throw new InvalidDataException($"{description} has zero magnitude.");
}
return vector.Select(value => (float)(value / magnitude)).ToArray();
}
private static byte[] Encode(float[] vector)
{
if (vector.Length != ResemblyzerVectorContract.Dimensions)
{
throw new InvalidDataException(
$"Voice vector has {vector.Length} dimensions; expected {ResemblyzerVectorContract.Dimensions}.");
}
var bytes = new byte[vector.Length * sizeof(float)];
for (var index = 0; index < vector.Length; index++)
{
if (!float.IsFinite(vector[index]))
{
throw new InvalidDataException("Voice vector contains a non-finite value.");
}
BinaryPrimitives.WriteSingleLittleEndian(
bytes.AsSpan(index * sizeof(float), sizeof(float)),
vector[index]);
}
return bytes;
}
private static string Fingerprint(string modelId, byte[] vectorBytes)
{
var modelBytes = Encoding.UTF8.GetBytes(modelId);
var input = new byte[modelBytes.Length + 1 + vectorBytes.Length];
modelBytes.CopyTo(input, 0);
input[modelBytes.Length] = 0;
vectorBytes.CopyTo(input, modelBytes.Length + 1);
return Convert.ToHexString(SHA256.HashData(input));
}
}
@@ -0,0 +1,391 @@
using System.Globalization;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using MeetingAssistant.Transcription;
using Microsoft.Extensions.Options;
namespace MeetingAssistant.Speakers;
public interface IResemblyzerVoiceEncoder
{
Task<IReadOnlyList<float[]>> EncodeAsync(
IReadOnlyList<byte[]> wavSamples,
CancellationToken cancellationToken);
Task WarmUpAsync(CancellationToken cancellationToken);
}
public sealed partial class VenvResemblyzerVoiceEncoder : IResemblyzerVoiceEncoder
{
private const string EnvironmentSchemaVersion = "venv-v2";
private const string JsonStart = "__MEETING_ASSISTANT_RESEMBLYZER_JSON_START__";
private const string JsonEnd = "__MEETING_ASSISTANT_RESEMBLYZER_JSON_END__";
private const string NumpyBeforePython313Requirement = "numpy<2; python_version < '3.13'";
private const string NumpyFromPython313Requirement = "numpy>=2,<3; python_version >= '3.13'";
private const string LibrosaRequirement = "librosa>=0.9.1";
private const string ScipyRequirement = "scipy>=1.2.1";
private readonly SemaphoreSlim commandLock = new(1, 1);
private readonly ICommandRunner commandRunner;
private readonly ResemblyzerSpeakerRecognitionOptions options;
private readonly ILogger<VenvResemblyzerVoiceEncoder> logger;
private string? verifiedEnvironmentPythonPath;
public VenvResemblyzerVoiceEncoder(
ICommandRunner commandRunner,
IOptions<MeetingAssistantOptions> options,
ILogger<VenvResemblyzerVoiceEncoder> logger)
{
this.commandRunner = commandRunner;
this.options = options.Value.SpeakerIdentification.Resemblyzer;
this.logger = logger;
}
public async Task<IReadOnlyList<float[]>> EncodeAsync(
IReadOnlyList<byte[]> wavSamples,
CancellationToken cancellationToken)
{
if (wavSamples.Count == 0)
{
return [];
}
if (wavSamples.Any(sample => sample.Length == 0))
{
throw new InvalidDataException("Resemblyzer cannot encode an empty WAV sample.");
}
var runtimeFolder = VaultPath.Resolve(options.RuntimeFolder);
var inputFolder = Path.Combine(runtimeFolder, "input", Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(inputFolder);
try
{
for (var index = 0; index < wavSamples.Count; index++)
{
await File.WriteAllBytesAsync(
Path.Combine(inputFolder, $"{index:D4}.wav"),
wavSamples[index],
cancellationToken);
}
await commandLock.WaitAsync(cancellationToken);
try
{
return await RunEncodingAsync(inputFolder, wavSamples.Count, cancellationToken);
}
finally
{
commandLock.Release();
}
}
finally
{
try
{
Directory.Delete(inputFolder, recursive: true);
}
catch (DirectoryNotFoundException)
{
}
catch (IOException exception)
{
logger.LogWarning(exception, "Could not remove Resemblyzer temporary input folder {InputFolder}", inputFolder);
}
catch (UnauthorizedAccessException exception)
{
logger.LogWarning(exception, "Could not remove Resemblyzer temporary input folder {InputFolder}", inputFolder);
}
}
}
private async Task<IReadOnlyList<float[]>> RunEncodingAsync(
string inputFolder,
int expectedCount,
CancellationToken cancellationToken)
{
return await RunWithTimeoutAsync(async token =>
{
var pythonPath = await EnsureEnvironmentAsync(token);
var scriptPath = Path.Combine(VaultPath.Resolve(options.RuntimeFolder), "encode.py");
await File.WriteAllTextAsync(scriptPath, BuildEncodingScript(), token);
var result = await RunRequiredAsync(
pythonPath,
[scriptPath, Path.GetFullPath(inputFolder)],
"encoding",
token);
return ParseAndValidateVectors(result.StandardOutput, expectedCount);
}, cancellationToken);
}
public async Task WarmUpAsync(CancellationToken cancellationToken)
{
await commandLock.WaitAsync(cancellationToken);
try
{
await RunWithTimeoutAsync(async token =>
{
await EnsureEnvironmentAsync(token);
return true;
}, cancellationToken);
}
finally
{
commandLock.Release();
}
}
private async Task<string> EnsureEnvironmentAsync(CancellationToken cancellationToken)
{
ValidateDependencySettings();
var runtimeFolder = VaultPath.Resolve(options.RuntimeFolder);
var environmentFolder = Path.Combine(runtimeFolder, "venv", BuildEnvironmentKey());
var pythonPath = GetEnvironmentPythonPath(environmentFolder);
var readyPath = Path.Combine(environmentFolder, ".ready");
if (string.Equals(verifiedEnvironmentPythonPath, pythonPath, StringComparison.OrdinalIgnoreCase)
&& File.Exists(pythonPath)
&& File.Exists(readyPath))
{
return pythonPath;
}
if (File.Exists(pythonPath) && File.Exists(readyPath))
{
var verification = await commandRunner.RunAsync(
pythonPath,
WarmupArguments,
cancellationToken);
if (verification.ExitCode == 0)
{
verifiedEnvironmentPythonPath = pythonPath;
return pythonPath;
}
logger.LogWarning(
"Existing Resemblyzer environment verification failed with exit code {ExitCode}: {Error}. Reprovisioning it.",
verification.ExitCode,
verification.StandardError);
File.Delete(readyPath);
}
Directory.CreateDirectory(environmentFolder);
await RunRequiredAsync(
options.PythonCommand,
["-m", "venv", "--clear", environmentFolder],
"virtual environment creation",
cancellationToken);
await RunRequiredAsync(
pythonPath,
["-m", "pip", "install", "--upgrade", "pip"],
"pip upgrade",
cancellationToken);
await RunRequiredAsync(
pythonPath,
[
"-m", "pip", "install",
"--index-url", options.TorchIndexUrl,
TorchRequirement
],
"CPU PyTorch installation",
cancellationToken);
await RunRequiredAsync(
pythonPath,
[
"-m", "pip", "install",
.. RuntimeDependencyRequirements
],
"Resemblyzer dependency installation",
cancellationToken);
await RunRequiredAsync(
pythonPath,
[
"-m", "pip", "install", "--no-deps",
ResemblyzerRequirement
],
"Resemblyzer installation",
cancellationToken);
await RunRequiredAsync(
pythonPath,
WarmupArguments,
"environment verification",
cancellationToken);
await File.WriteAllTextAsync(readyPath, BuildEnvironmentKey(), cancellationToken);
verifiedEnvironmentPythonPath = pythonPath;
return pythonPath;
}
private async Task<CommandResult> RunRequiredAsync(
string fileName,
IReadOnlyList<string> arguments,
string operation,
CancellationToken cancellationToken)
{
var result = await commandRunner.RunAsync(fileName, arguments, cancellationToken);
ThrowIfFailed(result, operation);
return result;
}
private static string BuildEncodingScript()
{
return
"import json\n"
+ "import sys\n"
+ "import numpy as np\n"
+ "from pathlib import Path\n"
+ "from resemblyzer import VoiceEncoder, preprocess_wav\n"
+ "encoder = VoiceEncoder('cpu', verbose=False)\n"
+ "vectors = []\n"
+ "for path in sorted(Path(sys.argv[1]).glob('*.wav')):\n"
+ " wav = preprocess_wav(path)\n"
+ " vector = encoder.embed_utterance(wav)\n"
+ " vectors.append(np.asarray(vector, dtype=np.float32).tolist())\n"
+ $"print('{JsonStart}')\n"
+ "print(json.dumps(vectors, allow_nan=False))\n"
+ $"print('{JsonEnd}')\n";
}
private static IReadOnlyList<float[]> ParseAndValidateVectors(string output, int expectedCount)
{
var json = ExtractJson(output);
float[][]? vectors;
try
{
vectors = JsonSerializer.Deserialize<float[][]>(json);
}
catch (JsonException exception)
{
throw new InvalidDataException("Resemblyzer returned malformed vector JSON.", exception);
}
if (vectors is null || vectors.Length != expectedCount)
{
throw new InvalidDataException(
$"Resemblyzer returned {vectors?.Length ?? 0} vectors for {expectedCount} WAV samples.");
}
return vectors
.Select((vector, index) => SpeakerVoiceVectors.Normalize(vector, $"Resemblyzer vector {index}"))
.ToList();
}
private static string ExtractJson(string output)
{
var start = output.IndexOf(JsonStart, StringComparison.Ordinal);
if (start < 0)
{
throw new InvalidDataException("Resemblyzer output did not contain the JSON start marker.");
}
start += JsonStart.Length;
var end = output.IndexOf(JsonEnd, start, StringComparison.Ordinal);
if (end < 0)
{
throw new InvalidDataException("Resemblyzer output did not contain the JSON end marker.");
}
return output[start..end].Trim();
}
private string BuildEnvironmentKey()
{
var settings = string.Join('\n', EnvironmentManifest);
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(settings));
return Convert.ToHexString(hash)[..16].ToLowerInvariant();
}
private string TorchRequirement => $"torch=={options.TorchVersion}";
private string ResemblyzerRequirement => $"Resemblyzer=={options.PackageVersion}";
private static string[] WarmupArguments =>
[
"-c",
"from resemblyzer import VoiceEncoder; VoiceEncoder('cpu', verbose=False); print('Resemblyzer warm-up complete')"
];
private string[] RuntimeDependencyRequirements =>
[
NumpyBeforePython313Requirement,
NumpyFromPython313Requirement,
LibrosaRequirement,
ScipyRequirement,
$"webrtcvad-wheels=={options.WebRtcVadVersion}"
];
private IEnumerable<string> EnvironmentManifest =>
[
EnvironmentSchemaVersion,
options.PythonCommand,
options.TorchIndexUrl,
TorchRequirement,
.. RuntimeDependencyRequirements,
ResemblyzerRequirement
];
private static string GetEnvironmentPythonPath(string environmentFolder)
{
return OperatingSystem.IsWindows()
? Path.Combine(environmentFolder, "Scripts", "python.exe")
: Path.Combine(environmentFolder, "bin", "python");
}
private void ValidateDependencySettings()
{
if (string.IsNullOrWhiteSpace(options.PythonCommand))
{
throw new InvalidOperationException("The Resemblyzer Python command cannot be empty.");
}
ValidateVersion(options.PackageVersion, "Resemblyzer package");
ValidateVersion(options.TorchVersion, "PyTorch");
ValidateVersion(options.WebRtcVadVersion, "webrtcvad-wheels");
if (!Uri.TryCreate(options.TorchIndexUrl, UriKind.Absolute, out var indexUri)
|| indexUri.Scheme != Uri.UriSchemeHttps)
{
throw new InvalidOperationException(
$"Invalid Resemblyzer PyTorch index URL '{options.TorchIndexUrl}'.");
}
}
private static void ValidateVersion(string version, string dependency)
{
if (!PackageVersionPattern().IsMatch(version))
{
throw new InvalidOperationException($"Invalid {dependency} version '{version}'.");
}
}
private async Task<T> RunWithTimeoutAsync<T>(
Func<CancellationToken, Task<T>> operation,
CancellationToken cancellationToken)
{
using var timeoutSource = options.CommandTimeout > TimeSpan.Zero
? new CancellationTokenSource(options.CommandTimeout)
: null;
using var linkedSource = timeoutSource is null
? null
: CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeoutSource.Token);
try
{
return await operation(linkedSource?.Token ?? cancellationToken);
}
catch (OperationCanceledException) when (
!cancellationToken.IsCancellationRequested && timeoutSource?.IsCancellationRequested == true)
{
throw new TimeoutException(
$"Resemblyzer command timed out after {options.CommandTimeout.ToString(null, CultureInfo.InvariantCulture)}.");
}
}
private static void ThrowIfFailed(CommandResult result, string operation)
{
if (result.ExitCode != 0)
{
throw new InvalidOperationException(
$"Resemblyzer {operation} failed with exit code {result.ExitCode}: {result.StandardError}");
}
}
[GeneratedRegex("^[0-9A-Za-z.+-]+$", RegexOptions.CultureInvariant)]
private static partial Regex PackageVersionPattern();
}
@@ -18,6 +18,12 @@ public sealed class BoundMeetingProjectResolver
public string ProjectsRoot => VaultPath.Resolve(options.Vault, options.Vault.ProjectsFolder);
public List<BoundMeetingProject> GetAllProjects() => !Directory.Exists(ProjectsRoot) ? [] :
Directory.EnumerateDirectories(ProjectsRoot)
.Select(path => new BoundMeetingProject(Path.GetFileName(path), path))
.OrderBy(project => project.Name, StringComparer.OrdinalIgnoreCase)
.ToList();
public async Task<List<BoundMeetingProject>> GetBoundProjectsAsync(
MeetingSessionArtifacts artifacts,
CancellationToken cancellationToken = default)
@@ -33,10 +39,8 @@ public sealed class BoundMeetingProjectResolver
return [];
}
return Directory.EnumerateDirectories(ProjectsRoot)
.Select(path => new BoundMeetingProject(Path.GetFileName(path), path))
return GetAllProjects()
.Where(project => projectNames.Contains(project.Name))
.OrderBy(project => project.Name, StringComparer.OrdinalIgnoreCase)
.ToList();
}
@@ -7,7 +7,6 @@ using System.Text.Json.Nodes;
using Microsoft.Agents.AI.Compaction;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Logging;
using OpenAI;
using OpenAI.Responses;
namespace MeetingAssistant.Summary;
@@ -798,7 +797,7 @@ public sealed class LiteLlmResponsesChatClient : IChatClient
string apiKey,
AsyncLocal<string?> requestInitiator)
{
var options = new OpenAIClientOptions
var options = new ResponsesClientOptions
{
Endpoint = httpClient.BaseAddress
?? throw new InvalidOperationException("LiteLLM HTTP client requires a base address."),
@@ -20,12 +20,20 @@ public sealed class MeetingSummaryInstructionBuilder : IMeetingSummaryInstructio
Use override_speaker only when you are very certain that a transcript speaker label belongs to a named person, for example from user notes, OCR evidence with a matching timestamp, or very clear context cues. Provide the exact speaker label from the transcript and the replacement speaker name. If the replacement speaker already exists in the transcript, use merge=true only when you are certain both speaker labels are the same identity; otherwise do not merge them.
Use delete_identity only when you are certain that an existing speaker identity was wrongfully matched. Provide the exact identity name currently used in the transcript.
After writing the summary, update existing project files when the meeting produced durable project knowledge, decisions, next steps, or context.
Use list_projects first to see which projects are bound to this meeting. Use search, list_past_project_meetings, read_past_project_meeting_summary, and read_projectfile before changing existing project files. search includes both project files and past meeting summaries for the requested current-meeting project scope.
Use search, list_past_project_meetings, read_past_project_meeting_summary, and read_projectfile before changing existing project files. search includes both project files and past meeting summaries for the requested current-meeting project scope.
The summary note should contain concise sections for summary, decisions, open questions, and next steps.
If the assistant context contains cropped screenshot markdown links, include only the most relevant cropped screenshots in the summary by markdown-linking them near the related summary text. When embedding them, encode spaces in image-link targets as `%20`; never leave literal spaces in the link target. Do not include every cropped screenshot by default, and do not link uncropped screenshots unless no cropped version exists and the image is important.
Keep the output grounded in the source material and explicitly say when a section has no known items.
""";
private const string ProjectAndRequestInstructions = """
Project discovery and approval:
list_projects returns every configured project's id, displayname, and description; only projects in the current meeting note's projects frontmatter are accessible for reads, writes, search, and historical summaries. Catalog visibility does not grant file access.
Before finalizing the summary, use the catalog when meeting evidence suggests a missing project association. Call request_project_association with its exact id and a short evidence-based reason (one line, at most 100 characters). It waits for user approval for up to ten minutes. A denial or timeout grants no access; continue with the existing scope. Approval adds the project to the meeting and returns its current project instructions; follow them immediately.
For each associated project, maintain a root PROJECT.md with YAML frontmatter name (display name) and description (at most 256 characters). Describe the project's purpose and distinguishing context so future summarizers can select it. Read existing content first and use write_projectfile to create missing metadata or update it from grounded knowledge, preserving other fields and body notes. Do not invent facts or write unassociated projects.
If a workflow improvement would help, use request_workflow_change with a short one-line intention (at most 100 characters) and a detailed self-contained change prompt, preferably a scenario/specification with expected behavior. This returns requested immediately and does not mean approved or implemented. Continue summarizing; only user approval starts the separate settings agent. You cannot change workflows directly.
""";
private readonly MeetingAssistantOptions options;
private readonly BoundMeetingProjectResolver projectResolver;
@@ -50,6 +58,7 @@ public sealed class MeetingSummaryInstructionBuilder : IMeetingSummaryInstructio
var instructions = string.IsNullOrWhiteSpace(options.Agent.InitialPrompt)
? DefaultInitialPrompt
: options.Agent.InitialPrompt.Trim();
instructions = instructions.TrimEnd() + "\n\n" + ProjectAndRequestInstructions;
var projectInstructions = await BuildProjectInstructionsAsync(artifacts, options, cancellationToken);
return string.IsNullOrWhiteSpace(projectInstructions)
? instructions
@@ -60,44 +69,15 @@ public sealed class MeetingSummaryInstructionBuilder : IMeetingSummaryInstructio
MeetingSessionArtifacts artifacts,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
var projects = await GetBoundProjectsWithInstructionsAsync(artifacts, options, cancellationToken);
if (projects.Count == 0)
{
return "";
}
var blocks = projects.Select(project =>
$"# {project.Name}\n\n{project.Instructions.Trim()}");
return "---\nprojects:\n\n" + string.Join("\n\n", blocks);
}
private async Task<List<ProjectInstructions>> GetBoundProjectsWithInstructionsAsync(
MeetingSessionArtifacts artifacts,
MeetingAssistantOptions options,
CancellationToken cancellationToken)
{
var resolver = ReferenceEquals(options, this.options)
? projectResolver
: new BoundMeetingProjectResolver(options);
var projects = new List<ProjectInstructions>();
var blocks = new List<string>();
foreach (var project in await resolver.GetBoundProjectsAsync(artifacts, cancellationToken))
{
var agentsPath = Path.Combine(project.Path, "AGENTS.md");
if (!File.Exists(agentsPath))
{
continue;
}
var content = await File.ReadAllTextAsync(agentsPath, cancellationToken);
if (!string.IsNullOrWhiteSpace(content))
{
projects.Add(new ProjectInstructions(project.Name, content));
}
blocks.Add(await ProjectMetadata.BuildContextAsync(project, cancellationToken));
}
return projects;
return blocks.Count == 0 ? "" : "---\nprojects:\n\n" + string.Join("\n\n", blocks);
}
private sealed record ProjectInstructions(string Name, string Instructions);
}
+89 -17
View File
@@ -2,6 +2,9 @@ using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Speakers;
using MeetingAssistant.Transcription;
using MeetingAssistant.Workflow;
using System.Text.Json;
using MeetingAssistant.Notifications;
using YamlDotNet.RepresentationModel;
using YamlDotNet.Serialization;
namespace MeetingAssistant.Summary;
@@ -16,6 +19,8 @@ public sealed class MeetingSummaryTools
private readonly SummaryAgentWriteAudit? writeAudit;
private readonly IMeetingWorkflowEngine meetingWorkflowEngine;
private readonly BoundMeetingProjectResolver projectResolver;
private readonly AgentApprovalService? approvals;
private readonly WorkflowChangeRequestService? workflowRequests;
public MeetingSummaryTools(MeetingSessionArtifacts artifacts)
: this(artifacts, new MeetingAssistantOptions(), null)
@@ -27,7 +32,9 @@ public sealed class MeetingSummaryTools
MeetingAssistantOptions options,
IDictationWordStore? dictationWordStore = null,
SummaryAgentWriteAudit? writeAudit = null,
IMeetingWorkflowEngine? meetingWorkflowEngine = null)
IMeetingWorkflowEngine? meetingWorkflowEngine = null,
AgentApprovalService? approvals = null,
WorkflowChangeRequestService? workflowRequests = null)
{
this.artifacts = artifacts;
this.options = options;
@@ -35,6 +42,8 @@ public sealed class MeetingSummaryTools
this.writeAudit = writeAudit;
this.meetingWorkflowEngine = meetingWorkflowEngine ?? NoopMeetingWorkflowEngine.Instance;
projectResolver = new BoundMeetingProjectResolver(options);
this.approvals = approvals;
this.workflowRequests = workflowRequests;
}
public Task<string> ReadTranscript(int? @from = null, int? to = null, int? tail = null)
@@ -291,8 +300,71 @@ public sealed class MeetingSummaryTools
public async Task<string> ListProjects()
{
var projects = await GetBoundProjectsAsync();
return string.Join('\n', projects.Select(project => project.Name));
var projects = await Task.WhenAll(projectResolver.GetAllProjects().Select(project => ProjectMetadata.ReadAsync(project)));
return JsonSerializer.Serialize(projects.Select(project => new
{
id = project.Id, displayname = project.DisplayName, description = project.Description
}));
}
public async Task<string> RequestProjectAssociation(string project_id, string reason, CancellationToken cancellationToken = default)
{
if (string.IsNullOrWhiteSpace(reason) || reason.Length > 100 || reason.Contains('\n') || reason.Contains('\r'))
return "Refused: reason must be a nonempty single line of at most 100 characters.";
var project = projectResolver.GetAllProjects()
.FirstOrDefault(candidate => string.Equals(candidate.Name, project_id, StringComparison.OrdinalIgnoreCase));
if (project is null) return "Refused: unknown project ID.";
var associated = await projectResolver.ReadMeetingProjectNamesAsync(artifacts, cancellationToken);
if (!associated.Contains(project.Name))
{
var metadata = await ProjectMetadata.ReadAsync(project, cancellationToken);
if (approvals is null || !await approvals.ConfirmAsync("Project association requested",
$"Associate {metadata.DisplayName} ({metadata.Id}) with {Path.GetFileName(artifacts.MeetingNotePath)}?\n{reason}",
cancellationToken))
return "Denied: project association was not approved.";
cancellationToken.ThrowIfCancellationRequested();
if (!Directory.Exists(project.Path) || !File.Exists(artifacts.MeetingNotePath))
return "Refused: the project or meeting note no longer exists.";
await AddProjectAssociationAsync(project.Name, cancellationToken);
}
return "Approved: project is associated with this meeting.\n\n"
+ await ProjectMetadata.BuildContextAsync(project, cancellationToken);
}
public string RequestWorkflowChange(string intention, string detailed_prompt)
=> workflowRequests?.Request(intention, detailed_prompt, artifacts.MeetingNotePath)
?? "Refused: workflow change requests are unavailable.";
private async Task AddProjectAssociationAsync(string projectId, CancellationToken cancellationToken)
{
var content = await File.ReadAllTextAsync(artifacts.MeetingNotePath, cancellationToken);
var document = MarkdownDocumentParser.SplitRequired(content,
"Meeting note requires frontmatter.", "Meeting note frontmatter is not closed.");
var yaml = new YamlStream();
yaml.Load(new StringReader(document.Frontmatter));
var mapping = (YamlMappingNode)yaml.Documents.Single().RootNode;
mapping.Children.TryGetValue(new YamlScalarNode("projects"), out var existing);
var projects = existing as YamlSequenceNode ?? new YamlSequenceNode();
if (existing is YamlScalarNode scalar && !string.IsNullOrWhiteSpace(scalar.Value))
projects.Add(new YamlScalarNode(scalar.Value));
if (!projects.Children.OfType<YamlScalarNode>().Any(value => string.Equals(value.Value, projectId, StringComparison.OrdinalIgnoreCase)))
projects.Add(new YamlScalarNode(projectId));
mapping.Children[new YamlScalarNode("projects")] = projects;
using var writer = new StringWriter();
yaml.Save(writer, assignAnchors: false);
var pendingPath = artifacts.MeetingNotePath + "." + Guid.NewGuid().ToString("N") + ".tmp";
try
{
await File.WriteAllTextAsync(pendingPath, "---\n" + writer + "---\n\n" + document.Body, cancellationToken);
cancellationToken.ThrowIfCancellationRequested();
File.Replace(pendingPath, artifacts.MeetingNotePath, destinationBackupFileName: null);
}
finally
{
File.Delete(pendingPath);
}
}
public async Task<string> ListProjectFiles(string project)
@@ -375,10 +447,18 @@ public sealed class MeetingSummaryTools
return "Refused: supply either both from and to for replacement, insert for insertion, or no line arguments for append; set replace_file=true only for whole-file replacement.";
}
var target = ResolveExistingProjectFilePath(project, path);
var target = await ResolveBoundProjectFileTargetAsync(project, path);
if (target is null)
{
return "Refused: project does not exist or the path escapes the project folder.";
return "Refused: project is not assigned to this meeting, does not exist, or the path escapes the project folder.";
}
var existingContent = File.Exists(target.Path) ? await File.ReadAllTextAsync(target.Path) : "";
var updatedContent = AgentFileToolContent.ApplyLineEdit(existingContent, content, editMode);
if (string.Equals(Path.GetRelativePath(target.Project.Path, target.Path), "PROJECT.md", StringComparison.OrdinalIgnoreCase)
&& !ProjectMetadata.IsValidContent(updatedContent))
{
return "Refused: PROJECT.md requires YAML frontmatter name and description; description must be at most 256 characters.";
}
Directory.CreateDirectory(Path.GetDirectoryName(target.Path)!);
@@ -386,11 +466,11 @@ public sealed class MeetingSummaryTools
{
await writeAudit.CaptureFileWriteAsync(
target.Path,
() => AgentFileToolContent.WriteFileContentAsync(target.Path, content, editMode));
() => File.WriteAllTextAsync(target.Path, updatedContent));
}
else
{
await AgentFileToolContent.WriteFileContentAsync(target.Path, content, editMode);
await File.WriteAllTextAsync(target.Path, updatedContent);
}
return $"{target.Project.Name}/{AgentFileToolContent.ToToolPath(path)}";
@@ -589,7 +669,7 @@ public sealed class MeetingSummaryTools
return AgentFileToolContent.IsWithinDirectory(projectRoot, fullPath) ? fullPath : null;
}
private ProjectFileTarget? ResolveExistingProjectFilePath(string project, string path)
private async Task<ProjectFileTarget?> ResolveBoundProjectFileTargetAsync(string project, string path)
{
if (string.IsNullOrWhiteSpace(project) ||
string.IsNullOrWhiteSpace(path) ||
@@ -598,15 +678,7 @@ public sealed class MeetingSummaryTools
return null;
}
var projectsRoot = GetProjectsRoot();
if (!Directory.Exists(projectsRoot))
{
return null;
}
var projectFolder = Directory.EnumerateDirectories(projectsRoot)
.Select(candidate => new BoundMeetingProject(Path.GetFileName(candidate), candidate))
.FirstOrDefault(candidate => string.Equals(candidate.Name, project, StringComparison.OrdinalIgnoreCase));
var projectFolder = await ResolveBoundProjectAsync(project);
if (projectFolder is null)
{
return null;
@@ -1,4 +1,5 @@
using MeetingAssistant.MeetingNotes;
using MeetingAssistant.Notifications;
using MeetingAssistant.Transcription;
using MeetingAssistant.Workflow;
using Microsoft.Agents.AI;
@@ -19,6 +20,8 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
private readonly IMeetingSummaryInstructionBuilder instructionBuilder;
private readonly IDictationWordStore dictationWordStore;
private readonly IMeetingWorkflowEngine meetingWorkflowEngine;
private readonly AgentApprovalService approvals;
private readonly WorkflowChangeRequestService workflowRequests;
public OpenAiMeetingSummaryAgentPipeline(
IOptions<MeetingAssistantOptions> options,
@@ -28,7 +31,9 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
IMeetingSummaryFailureWriter failureWriter,
IMeetingSummaryInstructionBuilder instructionBuilder,
IDictationWordStore dictationWordStore,
IMeetingWorkflowEngine meetingWorkflowEngine)
IMeetingWorkflowEngine meetingWorkflowEngine,
AgentApprovalService approvals,
WorkflowChangeRequestService workflowRequests)
{
this.options = options.Value;
this.loggerFactory = loggerFactory;
@@ -38,6 +43,8 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
this.instructionBuilder = instructionBuilder;
this.dictationWordStore = dictationWordStore;
this.meetingWorkflowEngine = meetingWorkflowEngine;
this.approvals = approvals;
this.workflowRequests = workflowRequests;
}
public async Task<MeetingSummaryRunResult> RunAsync(
@@ -60,7 +67,9 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
options,
dictationWordStore,
writeAudit,
meetingWorkflowEngine);
meetingWorkflowEngine,
approvals,
workflowRequests);
var tools = CreateTools(meetingTools);
var instructions = await instructionBuilder.BuildAsync(artifacts, options, cancellationToken);
using var compactionSummaryClient = LiteLlmResponsesChatClient.Create(
@@ -80,7 +89,11 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
.AsBuilder()
.UseFunctionInvocation(
loggerFactory,
client => client.FunctionInvoker = FunctionInvocationGuard.InvokeAsync)
client =>
{
client.FunctionInvoker = FunctionInvocationGuard.InvokeAsync;
client.AllowConcurrentInvocation = false;
})
.Build()
.AsAIAgent(
instructions,
@@ -176,7 +189,15 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
AIFunctionFactory.Create(
tools.ListProjects,
"list_projects",
"List the configured project folders bound to the current meeting note frontmatter."),
"List all configured projects with id, displayname and description. Catalog visibility does not grant access; project files require current meeting association."),
AIFunctionFactory.Create(
tools.RequestProjectAssociation,
"request_project_association",
"Request user approval to associate a project_id with this meeting. reason must be one line and at most 100 characters. Waits up to ten minutes; approval adds the project and returns its AGENTS.md if present."),
AIFunctionFactory.Create(
tools.RequestWorkflowChange,
"request_workflow_change",
"Request a workflow change with a one-line intention (maximum 100 characters) and a detailed_prompt describing the desired scenario/specification. Returns requested immediately; user approval separately opens and starts the settings agent."),
AIFunctionFactory.Create(
tools.ListProjectFiles,
"list_projectfiles",
@@ -188,7 +209,7 @@ public sealed class OpenAiMeetingSummaryAgentPipeline : IMeetingSummaryPipeline
AIFunctionFactory.Create(
tools.WriteProjectFile,
"write_projectfile",
"Write a file inside an existing project folder. With no line arguments, append or create the file. Set replace_file=true only when intentionally replacing the whole file. With from and to, replace that inclusive 1-based line range. With insert, insert content at that 1-based line position."),
"Write a file inside a project associated with the current meeting. Root PROJECT.md requires YAML name and description (maximum 256 characters). With no line arguments, append or create. Set replace_file=true only for intentional whole-file replacement. With from and to, replace that inclusive 1-based line range; with insert, insert at that line."),
AIFunctionFactory.Create(
tools.ListPastProjectMeetings,
"list_past_project_meetings",
@@ -0,0 +1,55 @@
using MeetingAssistant.MeetingNotes;
using YamlDotNet.Core;
using YamlDotNet.Serialization;
namespace MeetingAssistant.Summary;
public sealed record ProjectMetadata(string Id, string DisplayName, string Description)
{
public static async Task<string> BuildContextAsync(BoundMeetingProject project, CancellationToken cancellationToken = default)
{
var metadata = await ReadAsync(project, cancellationToken);
var context = $"# {metadata.Id}\nDisplay name: {metadata.DisplayName}\nDescription: {metadata.Description}";
var agentsPath = Path.Combine(project.Path, "AGENTS.md");
if (File.Exists(agentsPath))
{
var instructions = await File.ReadAllTextAsync(agentsPath, cancellationToken);
if (!string.IsNullOrWhiteSpace(instructions)) context += "\n\n" + instructions.Trim();
}
return context;
}
public static bool IsValidContent(string content) => Parse(content) is not null;
public static async Task<ProjectMetadata> ReadAsync(BoundMeetingProject project, CancellationToken cancellationToken = default)
{
var path = Path.Combine(project.Path, "PROJECT.md");
var metadata = File.Exists(path) ? Parse(await File.ReadAllTextAsync(path, cancellationToken)) : null;
return new(project.Name,
string.IsNullOrWhiteSpace(metadata?.Name) ? project.Name : metadata.Name.Trim(),
metadata?.Description?.Trim() ?? "");
}
private static Frontmatter? Parse(string content)
{
try
{
var document = MarkdownDocumentParser.SplitOptional(content);
if (!document.HasFrontmatter) return null;
var metadata = FrontmatterYamlDeserializer.Create().Deserialize<Frontmatter>(document.Frontmatter);
return string.IsNullOrWhiteSpace(metadata?.Name) || metadata.Description is null || metadata.Description.Length > 256
? null
: metadata;
}
catch (YamlException) { return null; }
catch (InvalidDataException) { return null; }
}
private sealed class Frontmatter
{
[YamlMember(Alias = "name")]
public string? Name { get; set; }
[YamlMember(Alias = "description")]
public string? Description { get; set; }
}
}
@@ -0,0 +1,70 @@
using MeetingAssistant.Notifications;
using MeetingAssistant.Workflow;
using System.Threading.Channels;
namespace MeetingAssistant.Summary;
public sealed class WorkflowChangeRequestService(
AgentApprovalService approvals,
IWorkflowRulesEditorWindowService editor,
ILogger<WorkflowChangeRequestService> logger) : BackgroundService
{
private readonly Channel<RequestDetails> requests = Channel.CreateUnbounded<RequestDetails>(
new UnboundedChannelOptions { SingleReader = true });
public string Request(string intention, string detailedPrompt, string meetingNotePath)
{
if (string.IsNullOrWhiteSpace(intention) || intention.Length > 100 || intention.Contains('\r') || intention.Contains('\n'))
return "Refused: intention must be a nonempty single line of at most 100 characters.";
if (string.IsNullOrWhiteSpace(detailedPrompt))
return "Refused: supply a detailed change prompt, preferably a scenario or specification.";
var request = new RequestDetails(Guid.NewGuid().ToString("N"), intention, detailedPrompt, meetingNotePath);
return requests.Writer.TryWrite(request) ? $"Requested: {request.Id}. User approval is pending."
: "Refused: application is stopping.";
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var pending = new List<Task>();
try
{
await foreach (var request in requests.Reader.ReadAllAsync(stoppingToken))
{
pending.RemoveAll(task => task.IsCompleted);
pending.Add(HandleAsync(request, stoppingToken));
}
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) { }
finally
{
await Task.WhenAll(pending);
}
}
private async Task HandleAsync(RequestDetails request, CancellationToken cancellationToken)
{
try
{
if (await approvals.ConfirmAsync($"Workflow change requested: {request.Intention}",
$"The summarizer for {Path.GetFileName(request.MeetingNotePath)} requests a workflow change. Open the settings agent to apply it?",
cancellationToken))
{
cancellationToken.ThrowIfCancellationRequested();
editor.ShowWithPrompt(request.DetailedPrompt);
}
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { }
catch (Exception exception)
{
logger.LogWarning(exception, "Workflow request {RequestId} could not be handled", request.Id);
}
}
public override Task StopAsync(CancellationToken cancellationToken)
{
requests.Writer.TryComplete();
return base.StopAsync(cancellationToken);
}
private sealed record RequestDetails(string Id, string Intention, string DetailedPrompt, string MeetingNotePath);
}
@@ -9,6 +9,8 @@ public enum MeetingTaskbarAction
OpenSubmenu,
StartRecording,
StopRecording,
PauseTranscription,
UnpauseTranscription,
AbortRecording,
SwitchProfile,
SelectMicrophone,
@@ -51,6 +53,15 @@ public static class MeetingTaskbarMenuBuilder
}
var secondaryControls = new List<MeetingTaskbarMenuItem>();
if (status.IsRecording)
{
secondaryControls.Add(new MeetingTaskbarMenuItem(
status.IsPaused ? "Unpause transcription" : "Pause transcription",
status.IsPaused
? MeetingTaskbarAction.UnpauseTranscription
: MeetingTaskbarAction.PauseTranscription));
}
if (microphones is { Count: > 0 })
{
secondaryControls.Add(BuildMicrophoneMenu(microphones, currentMicrophoneDeviceId));
@@ -249,6 +249,12 @@ public sealed class UnoTaskbarIconService : IHostedService, IDisposable
case MeetingTaskbarAction.StopRecording:
await coordinator.StopAsync(CancellationToken.None);
break;
case MeetingTaskbarAction.PauseTranscription:
await coordinator.SetTranscriptionPausedAsync(true, CancellationToken.None);
break;
case MeetingTaskbarAction.UnpauseTranscription:
await coordinator.SetTranscriptionPausedAsync(false, CancellationToken.None);
break;
case MeetingTaskbarAction.AbortRecording:
await coordinator.AbortAsync(CancellationToken.None);
break;
@@ -32,14 +32,13 @@ public sealed class PyannoteDiarizationWarmupHostedService : IHostedService
public async Task StopAsync(CancellationToken cancellationToken)
{
var cancellation = startupCancellation;
var cancellation = Interlocked.Exchange(ref startupCancellation, null);
var task = startupTask;
if (cancellation is null || task is null)
{
return;
}
startupCancellation = null;
startupTask = null;
await cancellation.CancelAsync();
try
@@ -96,14 +95,18 @@ public sealed class PyannoteDiarizationWarmupHostedService : IHostedService
}
}
private IEnumerable<PyannoteDiarizationOptions> GetEnabledDiarizationOptions()
private IEnumerable<PyannoteRuntimeOptions> GetEnabledDiarizationOptions()
{
return launchProfiles.GetProfiles()
.SelectMany(profile => GetEnabledDiarizationOptions(profile.Options))
var transcriptionRuntimes = launchProfiles.GetProfiles()
.SelectMany(profile => GetEnabledTranscriptionDiarizationOptions(profile.Options));
var speakerValidationRuntimes = GetEnabledSpeakerValidationDiarizationOptions(
launchProfiles.GetRequiredProfile(null).Options);
return transcriptionRuntimes
.Concat(speakerValidationRuntimes)
.DistinctBy(CreateWarmUpKey);
}
private static IEnumerable<PyannoteDiarizationOptions> GetEnabledDiarizationOptions(
private static IEnumerable<PyannoteRuntimeOptions> GetEnabledTranscriptionDiarizationOptions(
MeetingAssistantOptions options)
{
if (options.Recording.TranscriptionProvider.Equals("whisper-local", StringComparison.OrdinalIgnoreCase) &&
@@ -111,15 +114,19 @@ public sealed class PyannoteDiarizationWarmupHostedService : IHostedService
{
yield return options.WhisperLocal.Diarization;
}
}
if (options.SpeakerIdentification.PyannoteValidation.Enabled &&
options.SpeakerIdentification.PyannoteValidation.Diarization.Enabled)
private static IEnumerable<PyannoteRuntimeOptions> GetEnabledSpeakerValidationDiarizationOptions(
MeetingAssistantOptions options)
{
if (!options.SpeakerIdentification.Resemblyzer.Enabled &&
options.SpeakerIdentification.PyannoteValidation.Enabled)
{
yield return options.SpeakerIdentification.PyannoteValidation.Diarization;
}
}
private static string CreateWarmUpKey(PyannoteDiarizationOptions diarization)
private static string CreateWarmUpKey(PyannoteRuntimeOptions diarization)
{
return string.Join(
'\u001f',
@@ -28,7 +28,12 @@ public sealed class PyannoteTranscriptFinalizer
SpeechRecognitionPipelineOptions pipelineOptions,
CancellationToken cancellationToken)
{
return await FinalizeAsync(
if (!options.WhisperLocal.Diarization.Enabled)
{
return [];
}
return await FinalizeEnabledAsync(
audioPath,
liveSegments,
options.WhisperLocal.Diarization,
@@ -36,14 +41,14 @@ public sealed class PyannoteTranscriptFinalizer
cancellationToken);
}
public async Task<IReadOnlyList<TranscriptionSegment>> FinalizeAsync(
internal async Task<IReadOnlyList<TranscriptionSegment>> FinalizeEnabledAsync(
string audioPath,
IReadOnlyList<TranscriptionSegment> liveSegments,
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
SpeechRecognitionPipelineOptions pipelineOptions,
CancellationToken cancellationToken)
{
if (!diarization.Enabled || liveSegments.Count == 0)
if (liveSegments.Count == 0)
{
return [];
}
@@ -83,14 +88,9 @@ public sealed class PyannoteTranscriptFinalizer
}
public async Task WarmUpAsync(
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
CancellationToken cancellationToken)
{
if (!diarization.Enabled)
{
return;
}
var token = ResolveToken(diarization);
if (string.IsNullOrWhiteSpace(token))
{
@@ -137,7 +137,7 @@ public sealed class PyannoteTranscriptFinalizer
private async Task<CommandResult> RunDiarizationAsync(
string fullAudioPath,
string token,
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
SpeechRecognitionPipelineOptions pipelineOptions,
CancellationToken cancellationToken)
{
@@ -171,7 +171,7 @@ public sealed class PyannoteTranscriptFinalizer
}
private async Task EnsureDockerImageAsync(
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
string modelsFolder,
CancellationToken cancellationToken)
{
@@ -198,7 +198,7 @@ public sealed class PyannoteTranscriptFinalizer
}
}
private static string BuildDockerfile(PyannoteDiarizationOptions diarization)
private static string BuildDockerfile(PyannoteRuntimeOptions diarization)
{
return
$"FROM {diarization.BaseImage}\n"
@@ -211,7 +211,7 @@ public sealed class PyannoteTranscriptFinalizer
}
private string[] BuildDockerArguments(
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
string fullAudioPath,
string modelsFolder,
SpeechRecognitionPipelineOptions pipelineOptions)
@@ -242,7 +242,7 @@ public sealed class PyannoteTranscriptFinalizer
}
private string[] BuildWarmUpDockerArguments(
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
string modelsFolder)
{
return
@@ -269,7 +269,7 @@ public sealed class PyannoteTranscriptFinalizer
}
private static string BuildPythonCommand(
PyannoteDiarizationOptions diarization,
PyannoteRuntimeOptions diarization,
SpeechRecognitionPipelineOptions pipelineOptions)
{
var model = diarization.Model;
@@ -296,7 +296,7 @@ public sealed class PyannoteTranscriptFinalizer
+ "python /tmp/meeting_assistant_pyannote.py";
}
private static string BuildWarmUpPythonCommand(PyannoteDiarizationOptions diarization)
private static string BuildWarmUpPythonCommand(PyannoteRuntimeOptions diarization)
{
var model = diarization.Model;
return
@@ -475,7 +475,7 @@ public sealed class PyannoteTranscriptFinalizer
return turns;
}
private static string ResolveToken(PyannoteDiarizationOptions options)
private static string ResolveToken(PyannoteRuntimeOptions options)
{
if (!string.IsNullOrWhiteSpace(options.Token))
{
@@ -3,6 +3,7 @@ namespace MeetingAssistant.Workflow;
public interface IWorkflowRulesEditorWindowService
{
void Show();
void ShowWithPrompt(string initialPrompt);
}
public sealed class NoopWorkflowRulesEditorWindowService : IWorkflowRulesEditorWindowService
@@ -18,4 +19,6 @@ public sealed class NoopWorkflowRulesEditorWindowService : IWorkflowRulesEditorW
{
logger.LogInformation("Workflow rules editor UI is only available on Windows");
}
public void ShowWithPrompt(string initialPrompt) => Show();
}
@@ -24,6 +24,7 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
private readonly IMeetingMetadataProvider meetingMetadataProvider;
private readonly ILaunchProfileOptionsProvider launchProfiles;
private readonly ISpeakerIdentityMergeService identityMergeService;
private readonly ResemblyzerVoiceVectorOutlierPruner outlierPruner;
private readonly AsrDiagnosticService asrDiagnosticService;
private readonly IConfiguration configuration;
@@ -38,6 +39,7 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
IMeetingMetadataProvider meetingMetadataProvider,
ILaunchProfileOptionsProvider launchProfiles,
ISpeakerIdentityMergeService identityMergeService,
ResemblyzerVoiceVectorOutlierPruner outlierPruner,
AsrDiagnosticService asrDiagnosticService,
IConfiguration configuration)
{
@@ -51,6 +53,7 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
this.meetingMetadataProvider = meetingMetadataProvider;
this.launchProfiles = launchProfiles;
this.identityMergeService = identityMergeService;
this.outlierPruner = outlierPruner;
this.asrDiagnosticService = asrDiagnosticService;
this.configuration = configuration;
}
@@ -78,7 +81,8 @@ public sealed class WorkflowRulesEditorChatPipeline : IWorkflowRulesEditorChatPi
identityMergeService,
asrDiagnosticService,
configuration,
logger: logger);
logger: logger,
outlierPruner: outlierPruner);
var messages = conversation
.Select(ToChatMessage)
.Append(new ChatMessage(ChatRole.User, userMessage.Trim()))
@@ -25,9 +25,11 @@ public sealed class WorkflowRulesEditorChatViewModel
public event EventHandler? Changed;
public async Task SendAsync(CancellationToken cancellationToken = default)
public Task SendAsync(CancellationToken cancellationToken = default) => SendPromptAsync(Draft, cancellationToken);
public async Task SendPromptAsync(string prompt, CancellationToken cancellationToken = default)
{
var prompt = Draft.Trim();
prompt = prompt.Trim();
if (string.IsNullOrWhiteSpace(prompt) || IsThinking)
{
return;
@@ -8,6 +8,7 @@ public interface IWorkflowRulesEditorSamplePlaybackQueue
Task<string> QueueAsync(int sampleId, byte[] wavBytes, CancellationToken cancellationToken = default);
}
#if WINDOWS
public sealed class WorkflowRulesEditorSamplePlaybackQueue : IWorkflowRulesEditorSamplePlaybackQueue, IDisposable
{
private readonly Channel<QueuedSample> channel = Channel.CreateUnbounded<QueuedSample>();
@@ -98,3 +99,13 @@ public sealed class WorkflowRulesEditorSamplePlaybackQueue : IWorkflowRulesEdito
private sealed record QueuedSample(int SampleId, byte[] WavBytes);
}
#else
public sealed class UnavailableWorkflowRulesEditorSamplePlaybackQueue : IWorkflowRulesEditorSamplePlaybackQueue
{
public Task<string> QueueAsync(int sampleId, byte[] wavBytes, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
return Task.FromResult("Speaker sample playback requires the Windows build of Meeting Assistant.");
}
}
#endif
@@ -10,6 +10,7 @@ using MeetingAssistant.Transcription;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using YamlDotNet.Core;
using YamlDotNet.Serialization;
@@ -42,6 +43,7 @@ public sealed class WorkflowRulesEditorTools
private readonly IMeetingMetadataProvider? meetingMetadataProvider;
private readonly ILaunchProfileOptionsProvider? launchProfiles;
private readonly ISpeakerIdentityMergeService? identityMergeService;
private readonly ResemblyzerVoiceVectorOutlierPruner outlierPruner;
private readonly AsrDiagnosticService? asrDiagnosticService;
private readonly IConfiguration? configuration;
private readonly ILogger? logger;
@@ -61,7 +63,8 @@ public sealed class WorkflowRulesEditorTools
string? logDirectory = null,
string? specRootPath = null,
string? projectAgentsTemplatePath = null,
ILogger? logger = null)
ILogger? logger = null,
ResemblyzerVoiceVectorOutlierPruner? outlierPruner = null)
{
this.options = options;
rulesPath = WorkflowRulesPathResolver.Resolve(options.Automation.RulesPath);
@@ -82,6 +85,9 @@ public sealed class WorkflowRulesEditorTools
this.meetingMetadataProvider = meetingMetadataProvider;
this.launchProfiles = launchProfiles;
this.identityMergeService = identityMergeService;
this.outlierPruner = outlierPruner ?? new ResemblyzerVoiceVectorOutlierPruner(
speakerOptions.Resemblyzer,
NullLogger<ResemblyzerVoiceVectorOutlierPruner>.Instance);
this.asrDiagnosticService = asrDiagnosticService;
this.configuration = configuration;
this.logger = logger;
@@ -743,8 +749,8 @@ public sealed class WorkflowRulesEditorTools
string[]? candidateNames = null)
{
return Task.FromResult(
"Refused: speaker identities require at least one audio sample. " +
"Use a speaker override from an existing transcript speaker sample, or update/merge an existing sampled identity.");
"Refused: speaker identities require audio evidence (a WAV sample or voice vector). " +
"Use a speaker override from an existing transcript speaker, or update/merge an existing identity.");
}
public async Task<string> UpdateIdentity(
@@ -826,10 +832,12 @@ public sealed class WorkflowRulesEditorTools
return $"Could not find target {targetIdentityId} or source {sourceIdentityId}.";
}
SpeakerIdentityMerger.MergeInto(
SpeakerIdentityMerger.MergeIntoAndPrune(
target,
source,
speakerOptions.MaxSnippetsPerSpeaker);
speakerOptions.MaxSnippetsPerSpeaker,
speakerOptions.Resemblyzer.MaxVectorsPerIdentity,
outlierPruner);
context.SpeakerIdentities.Remove(source);
await context.SaveChangesAsync();
return ToJson(ToIdentityDetail(target));
@@ -1567,9 +1575,11 @@ public sealed class WorkflowRulesEditorTools
private static IQueryable<SpeakerIdentity> LoadIdentities(SpeakerIdentityDbContext context)
{
return context.SpeakerIdentities
.AsSplitQuery()
.Include(identity => identity.Aliases)
.Include(identity => identity.CandidateNames)
.Include(identity => identity.Snippets)
.Include(identity => identity.VoiceVectors)
.Include(identity => identity.References);
}
@@ -1591,6 +1601,7 @@ public sealed class WorkflowRulesEditorTools
identity.Aliases.Select(alias => alias.Name).Order(StringComparer.OrdinalIgnoreCase).ToArray(),
identity.CandidateNames.Select(candidate => candidate.Name).Order(StringComparer.OrdinalIgnoreCase).ToArray(),
identity.Snippets.Count,
identity.VoiceVectors.Count,
identity.References.Count,
identity.UpdatedAt);
}
@@ -1650,6 +1661,7 @@ public sealed class WorkflowRulesEditorTools
IReadOnlyList<string> Aliases,
IReadOnlyList<string> CandidateNames,
int SampleCount,
int VoiceVectorCount,
int ReferenceCount,
DateTimeOffset UpdatedAt);
@@ -37,6 +37,26 @@ internal sealed class WpfWorkflowRulesEditorWindowService : IWorkflowRulesEditor
dispatcher?.BeginInvoke(ShowOnUiThread);
}
public void ShowWithPrompt(string initialPrompt)
{
EnsureUiThread();
dispatcher?.BeginInvoke(new Action(async () =>
{
try
{
var viewModel = services.GetRequiredService<WorkflowRulesEditorChatViewModel>();
var requestedWindow = new WorkflowRulesEditorWpfWindow(viewModel, linkResolver, () => { });
requestedWindow.Show();
requestedWindow.Activate();
await viewModel.SendPromptAsync(initialPrompt);
}
catch (Exception exception)
{
logger.LogError(exception, "Could not start settings agent for an approved workflow request");
}
}));
}
private void EnsureUiThread()
{
lock (sync)
+24 -4
View File
@@ -36,6 +36,7 @@
"00:10:00"
],
"AutoStopAfter": "00:30:00",
"MaximumPauseDuration": "04:00:00",
"InferredEndPadding": "00:01:00",
"CheckInterval": "00:00:15"
}
@@ -119,18 +120,17 @@
"MaxMatchCandidates": 100,
"MatchIdentityActiveAge": "365.00:00:00",
"MaxSnippetsPerSpeaker": 3,
"MinimumSampleSpeechDuration": "00:00:30",
"MaximumSampleSegmentGap": "00:00:01",
"MinimumSampleSpeechDuration": "00:00:10",
"MaximumSampleDuration": "00:01:00",
"SilenceBetweenSnippetsSeconds": 1,
"LiveSampleBufferDuration": "00:10:00",
"MergeRecentIdentityAge": "14.00:00:00",
"MatchTimeout": "00:03:00",
"PyannoteValidation": {
"Enabled": true,
"Enabled": false,
"MinimumSingleSpeakerCoverage": 0.9,
"MinimumMatchingKnownSnippetRatio": 0.5,
"Diarization": {
"Enabled": false,
"DockerCommand": "docker",
"BaseImage": "python:3.11-slim",
"Image": "meeting-assistant-pyannote:local",
@@ -142,6 +142,26 @@
"TokenEnv": "HF_TOKEN",
"CommandTimeout": "01:00:00"
}
},
"Resemblyzer": {
"Enabled": false,
"RequiredVectorsPerSpeaker": 5,
"MaxVectorsPerIdentity": 1000,
"OutlierPruningMinimumVectors": 20,
"OutlierPruningNeighborSimilarity": 0.75,
"OutlierPruningMinimumNeighbors": 3,
"OutlierPruningMinimumClusterRatio": 0.60,
"MinimumClusterCohesion": 0.75,
"MinimumIdentitySimilarity": 0.75,
"MinimumSimilarityMargin": 0.05,
"ModelId": "resemblyzer-0.1.4-pretrained",
"PackageVersion": "0.1.4",
"PythonCommand": "python",
"TorchVersion": "2.14.0+cpu",
"TorchIndexUrl": "https://download.pytorch.org/whl/cpu",
"WebRtcVadVersion": "2.0.14",
"RuntimeFolder": "%LOCALAPPDATA%\\MeetingAssistant\\Resemblyzer",
"CommandTimeout": "00:15:00"
}
},
"Automation": {
+85 -77
View File
@@ -1,28 +1,32 @@
# Meeting Assistant
Meeting Assistant is Manuel's local .NET meeting capture and knowledge service. It runs on Windows and macOS, records microphone plus system audio, writes Obsidian meeting artifacts, transcribes with speaker attribution, enriches meetings from local context, and drives summary/project agents without depending on Teams, Zoom, or another meeting-platform API for the primary flow.
Meeting Assistant is Manuel's local .NET meeting capture and knowledge service. It runs on Windows and macOS, records microphone plus system audio, maintains meeting artifacts in an Obsidian vault, and runs transcription, speaker attribution, screenshot OCR, and agentic summarization without depending on a meeting-platform API for the primary flow.
## What This Repo Owns
## Scope And Runtime Boundary
This repository owns the application source, tests, OpenSpec requirements, configuration examples, local runtime documentation, and CI build/test workflow for Meeting Assistant. It does not own a homelab deployment stack, Traefik routing, Docker Compose deployment, or production image tag selection.
The app is intentionally local-first:
This repository owns the application source, tests, OpenSpec requirements, checked-in configuration, and CI validation. It does not own workstation startup automation, a homelab deployment stack, Traefik routing, Docker Compose, or a deployed image tag.
- Windows builds provide the tray icon, global hotkeys, NAudio capture, Outlook Classic COM enrichment, active-window screenshots, and notifications.
- Portable builds on macOS provide a native menu-bar icon and global hotkeys, and capture the default microphone through AVFoundation plus computer output through ScreenCaptureKit.
- Portable builds on other hosts keep the server/testable service surface but do not provide meeting audio capture.
- The normal runtime endpoint is local HTTP on port `5090`, with `/health` and `/recording/status` as the safe first checks.
- Behavior changes are OpenSpec-driven; current accepted requirements live under `openspec/specs`.
Accepted requirements live under `openspec/specs`. Relevant active changes under `openspec/changes` can describe implemented behavior that has not yet been folded into the accepted specs.
## Quick Start
For a development build:
The project requires the .NET 10 SDK. Build and test it with:
```powershell
dotnet restore MeetingAssistant.slnx
dotnet build MeetingAssistant.slnx
dotnet test MeetingAssistant.slnx
dotnet run --project MeetingAssistant
```
Do not start a second copy over the live workstation instance. Check the local control surface first:
```powershell
Invoke-RestMethod http://127.0.0.1:5090/health
Invoke-RestMethod http://127.0.0.1:5090/recording/status
```
On macOS, build the portable target on the Mac that will run it:
@@ -34,38 +38,32 @@ dotnet run --project MeetingAssistant/MeetingAssistant.csproj -f net10.0
The macOS build compiles bundled Swift helpers into `Native/macos-meeting-audio-capture`, `Native/macos-desktop-controls`, and `Native/macos-meeting-integrations`. On first use, allow **Microphone**, **Screen & System Audio Recording**, and **Calendar Full Access** under System Settings > Privacy & Security. The menu-bar icon and global hotkeys use the same configured bindings as Windows and call the local service surface.
For the durable local background instance, use the snippets restart helper:
For a foreground Windows development run when port `5090` is free:
```powershell
dotnet run --project MeetingAssistant --framework net10.0-windows10.0.19041.0
```
The durable workstation instance is published and replaced through automation owned by `Manuel/snippets`:
```powershell
powershell -ExecutionPolicy Bypass -File C:\Manuel\snippets\restart-meeting-assistant.ps1
```
The helper publishes `MeetingAssistant\MeetingAssistant.csproj` for `net10.0-windows10.0.19041.0` into timestamped folders below `C:\Manuel\meeting-assistant\tmp\meeting-assistant-runtime`. It stops the old port-`5090` instance only after publishing succeeds, starts the published `MeetingAssistant.exe`, waits for `/health`, and writes process logs plus the last PID to:
The helper publishes before stopping the old process, starts the timestamped Windows publish on loopback port `5090`, waits for `/health`, and records stdout, stderr, and the PID under `%LOCALAPPDATA%\MeetingAssistant\Logs`. Always confirm `/recording/status` is idle before invoking it.
```text
%LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.out.log
%LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.err.log
%LOCALAPPDATA%\MeetingAssistant\Logs\meeting-assistant.pid
```
Before restarting, always check the running process:
```powershell
Invoke-RestMethod http://127.0.0.1:5090/recording/status
```
Do not restart while a meeting is recording, transcribing, finalizing speaker recognition, waiting on OCR, or summarizing unless the user explicitly approves the interruption.
## Runtime Behavior
## Recording And Control
Recording can be controlled through global hotkeys, the Windows tray icon, the macOS menu-bar icon, or local HTTP endpoints. The default hotkeys are:
- `Ctrl+Alt+M`: toggle the default recording profile.
- `Ctrl+Alt+L`: toggle or switch to the configured `english` launch profile.
- `Ctrl+Alt+Z`: abort the active recording and delete that run's artifacts.
- `Ctrl+Alt+S`: capture the active window into the current meeting context.
- `Ctrl+Alt+L`: start or switch to the configured `english` profile.
- `Ctrl+Alt+Z`: abort the active run and delete its artifacts.
- `Ctrl+Alt+S`: capture the active window into the meeting context.
The main local endpoints are:
The Windows tray presents `Finish meeting` as the primary action during capture; microphone selection, cancel/discard, profile switching, and pause/unpause transcription remain separate controls. A paused meeting stays active and can still be finished or canceled normally. Windows `Exit` requires confirmation while any meeting is recording or processing. The macOS menu offers profile start actions while idle and `Stop meeting recording and transcribe` plus cancel/discard during capture; its `Exit` action currently exits directly.
The loopback HTTP surface has no application authentication, so port `5090` must remain a trusted loopback-only control surface. The main endpoints are:
- `GET /health`
- `GET /recording/status`
@@ -80,46 +78,60 @@ The main local endpoints are:
- `POST` or `GET /meetings/summary/retry`
- `POST` or `GET /meetings/screenshot-ocr/retry`
Stopping a recording stops new audio capture but lets that run continue transcription drain, speaker processing, meeting-note image OCR, screenshot OCR waits, summary generation, and final artifact updates. A later recording can start while an older stopped run is still finalizing; each run keeps isolated artifact paths and options.
Some generated retry links use `GET` while starting work.
Stopping capture lets buffered transcription, speaker work, meeting-note image OCR, screenshot OCR, and summary generation finish. Another meeting can start while an older stopped run finalizes; each run retains isolated options and artifact paths.
Pausing transcription keeps the active recognition pipeline and meeting session alive, including Azure conversation transcription and the run's speaker context. Captured audio is replaced with equal-length silence before it reaches the temporary WAV or transcription backend, so real audio from the paused interval is discarded while provider continuity and meeting-relative timing are preserved. Normal transcript-inactivity notifications and auto-stop are suppressed during pause, while a separate four-hour maximum continuous pause prevents a forgotten paused meeting from running indefinitely. The recording status response exposes pause state as `isPaused`.
During an active run, microphone creation failures and disconnects are retried every second with fresh endpoint selection. The meeting and system-loopback capture stay active, with microphone silence mixed in until capture resumes. This recovery does not cover a failed system-loopback source.
A newly available microphone during a meeting triggers a Windows notification with its name and Yes/No actions, including while transcription is paused. Yes switches the current microphone within the same meeting; No or no response keeps capture unchanged. Each device is offered once per meeting, and unanswered prompts expire after one minute.
Outlook enrichment selects an unambiguous current or imminent appointment. A scheduled prompt shown during an active recording can apply that exact appointment's title, eligible attendees, agenda, and scheduled end without interrupting capture; explicit prompt metadata wins over a slower background lookup.
## Data And Side Effects
Meeting Assistant writes durable meeting knowledge to the configured Obsidian vault:
Meeting Assistant writes meeting notes, transcripts, assistant context, summaries, and project knowledge into the configured Obsidian vault. Assistant context is persistent meeting-specific memory: the summarizer records problems and assumptions, and the interactive agent can read it and append later repairs and conclusions.
- meeting notes
- transcripts
- assistant context notes
- summary notes
- project knowledge files that agents may read or update
- optional dictation words used as speech-recognition phrase hints
Project discovery uses the `name` and short `description` in each project's `PROJECT.md`. The summarizer can request missing associations through a Yes/No notification that waits up to ten minutes; only approval adds the project and enables its file access. Workflow-change requests return immediately and, when approved, open a separate settings-agent conversation with the proposed change.
It also writes local runtime state outside the vault:
Agents are intentionally stateful. Depending on the invoked tools, they can change workflow rules and appsettings, create or update project and meeting files, change frontmatter, merge or delete speaker identities and samples, run diagnostics, and trigger transcription or summary work. Screenshot OCR can add recognized attendee names to the meeting note after workflow transformation; OCR of images already embedded in the meeting note does not add attendees or modify that note.
- `%LOCALAPPDATA%\MeetingAssistant\Recordings`: temporary mixed WAV files during active/finalizing runs; stale files are deleted at startup and completed runs delete their temporary audio.
- `%LOCALAPPDATA%\MeetingAssistant\SpeakerIdentity\speaker-identities.db`: local SQLite speaker identity database, aliases, meeting references, and bounded voice snippets.
- `%LOCALAPPDATA%\MeetingAssistant\FunASR\models`: persistent FunASR model and hotword cache when the managed FunASR backend is enabled.
- `%LOCALAPPDATA%\MeetingAssistant\Pyannote\models`: persistent pyannote/Hugging Face/torch cache when pyannote diarization or validation is enabled.
- `%TEMP%\MeetingAssistant\Logs\meeting-assistant.log`: application-owned rotating logs; the snippets starter separately captures stdout/stderr under `%LOCALAPPDATA%`.
Local runtime state outside the vault includes:
The repo intentionally ignores local rule files, local appsettings overrides, models, recordings, runtime caches, build output, and temporary publish folders.
- `%LOCALAPPDATA%\MeetingAssistant\Recordings`: mixed WAV files are normally deleted after completion, and unqueued stale files are deleted at startup. If an Azure stop cannot drain within `Recording:StopProcessingTimeout`, the WAV plus a JSON item under `offline-transcription-backlog` are retained and retried every minute. Both are removed only after successful replay, transcript finalization, and summary processing.
- `%LOCALAPPDATA%\MeetingAssistant\SpeakerIdentity\speaker-identities.db`: SQLite identities, aliases, meeting references, bounded WAV snippets for the existing matcher, and separate versioned voice vectors for the optional Resemblyzer matcher.
- `%LOCALAPPDATA%\MeetingAssistant\Resemblyzer`: content-versioned managed Python environments, the local encoder script, and temporary encoder inputs. Per-meeting WAV inputs are deleted after each encoding command.
- `%LOCALAPPDATA%\MeetingAssistant\FunASR\models` and `%LOCALAPPDATA%\MeetingAssistant\Pyannote\models`: persistent model, hotword, Hugging Face, and torch caches for optional local backends.
- `%TEMP%\MeetingAssistant\Logs\meeting-assistant.log`: application log with four rotated predecessors. Paths, transcript text, agent diagnostics, and provider errors can make these logs sensitive.
Abort is destructive: it removes the active run's note, transcript, context, summary, and linked screenshot attachments and skips summarization. A normal stop below `Recording:MinimumCompletedMeetingDuration`, or an otherwise content-empty stop, can also remove generated artifacts.
## External Data Boundaries
“Local” describes control and durable storage, not every processing step:
- The default `azure-speech` provider sends mixed meeting audio and dictation phrase hints to Azure AI Speech. Azure-backed speaker matching also sends selected voice audio.
- Summary, screenshot OCR, and interactive-agent requests go to the configured OpenAI-compatible Responses endpoint. They can include meeting/transcript/project text, screenshots, configuration, logs, and speaker samples when corresponding tools are used. The checked-in endpoint is a loopback proxy; its ultimate provider, data path, and retention policy are outside this repository.
- Outlook Classic access on Windows is local COM. EventKit access on macOS reads calendars synchronized into the Calendar app. Neither provides the primary capture path.
- A managed FunASR run pulls its configured image, removes any same-named container, starts a disposable container privileged by default, publishes the configured host port, and mounts the model/hotword cache. Pyannote may build a local image and start disposable containers with audio inputs mounted read-only; those paths require Docker Desktop or a compatible Docker CLI. The opt-in Resemblyzer speaker matcher instead provisions an isolated local Python venv with CPU-only dependencies. First use may download images, Python packages, or models from external registries.
## Configuration
Configuration is normal .NET configuration under `MeetingAssistant`; the checked-in `MeetingAssistant/appsettings.json` is the canonical example. Keep secret values out of source and use the configured environment-variable names instead.
`MeetingAssistant/appsettings.json` is the canonical configuration example. Keep secret values out of tracked JSON and local rule/config files out of source control.
Important settings:
The settings with the largest operational effect are:
- `Vault`: controls the Obsidian vault root and the relative folders for notes, transcripts, summaries, assistant context, project knowledge, and dictation words.
- `Recording:TranscriptionProvider`: selects `azure-speech`, `funasr`, or `whisper-local`.
- `Recording:MicrophoneDeviceId`: optionally pins the Windows microphone endpoint; blank follows the Windows default, and the tray icon can override it for later starts.
- `Recording:TemporaryRecordingsFolder`: controls temporary mixed WAV storage.
- `Recording:InactivitySafeguard`: prompts and then auto-stops forgotten silent recordings without aborting artifacts.
- `LaunchProfiles`: overlays named profile settings onto the default profile; profile hotkeys must be distinct.
- `Automation:RulesPath`: points to the local YAML workflow-rules file. The default `meeting-rules.local.yaml` is ignored by git.
- `CalendarRecordingPrompts`: enables Teams-start prompts from Outlook Classic on Windows or EventKit calendars on macOS.
- `Screenshots`: controls the capture hotkey, attachment folder, and optional OCR/vision model.
- `Agent`: configures the OpenAI-compatible summary/project agent endpoint, model, retries, output limits, and compaction.
- `WorkflowRulesEditor`: optionally overrides the agent settings for the tray-launched assistant.
- `Vault`: selects the durable vault and artifact/project locations.
- `Recording:TranscriptionProvider`: selects `azure-speech`, `funasr`, or `whisper-local`; the latter requires a local Whisper model file.
- `Recording:MicrophoneDeviceId`, mix gains, stop timeout, minimum duration, and temporary folder: control capture selection, audio, cleanup, and Azure backlog behavior. Microphone-device selection is Windows-only; macOS follows the system default input device.
- `Recording:InactivitySafeguard`: prompts and can auto-finish a run after no new transcript text; it is not an audio-silence detector.
- `LaunchProfiles`: overlay named recording/ASR/agent settings and require distinct hotkeys.
- `SpeakerIdentification:Resemblyzer:Enabled`: selects the local, managed-Python-venv vector matcher for the whole application; when disabled, the existing WAV/Azure path stays active. Five vectors unlock matching by default without capping retained evidence, and mature profiles use configurable fail-safe density clustering to remove likely mixed-speaker outliers.
- `Automation:RulesPath`: points to the local YAML workflow-rules file, normally ignored `meeting-rules.local.yaml`.
- `CalendarRecordingPrompts` and `Screenshots`: control Outlook prompts on Windows, EventKit prompts on macOS, capture, attachments, and configured OCR.
- `Agent` and `WorkflowRulesEditor`: select the Responses endpoint/model, streaming or non-streaming transport, reasoning, retry, output, and compaction behavior; the available tools are defined by the application.
Required or commonly used secrets:
@@ -149,37 +161,33 @@ The Windows tray and macOS menu-bar menus expose `Open agent`, which opens the `
At startup, Meeting Assistant detects whether the host is Windows or macOS and includes that immutable runtime context in the interactive settings/logs assistant instructions. On Windows the assistant uses Windows commands and concepts such as PowerShell, Windows paths, services, and Task Manager. On macOS it uses zsh, POSIX paths, launchd/LaunchAgents, and Activity Monitor. This guidance is also appended when a custom interactive-agent prompt is configured. macOS audio capture, menu-bar controls, global hotkeys, EventKit calendar enrichment/prompts, active-window screenshots, screenshot OCR, speaker identification, FunASR, local diarization, and the AppKit/WebKit workflow editor are available in the portable build. Outlook Classic COM and Windows toast notifications remain Windows-specific implementations.
The macOS publish output includes a signed `MeetingAssistant.app` bundle. Install that bundle in `/Applications` and launch the background service through its native executable so macOS microphone, calendar, and Screen/System Audio privacy grants are attributed to the stable Meeting Assistant application identity.
Detailed workflow syntax and extension guidance live in `docs/meeting-workflow-engine.md`.
## Development And CI
The repo builds against .NET 10 and targets both `net10.0` and `net10.0-windows10.0.19041.0` for the app. Tests target `net10.0`.
Behavior changes are OpenSpec-driven and test-first: update the relevant requirement/scenario, add a failing public behavior test, implement the smallest passing change, run focused tests and then the justified broader suite, and validate the active change with `openspec validate <change-id> --strict`. Documentation-only maintenance does not need a new OpenSpec change.
The Gitea workflow `.gitea/workflows/pr-push-build-and-test.yaml` runs on pull requests, pushes, and manual dispatch. It restores/builds the Windows target on an Ubuntu runner, installs Wine, downloads a matching Windows .NET SDK, and runs the test project through the Windows dotnet host under Wine.
The Gitea workflow runs for pull requests, pushes, and manual dispatch on the existing `ubuntu-latest` runners. One job explicitly builds the Windows desktop target, installs Wine plus a matching Windows .NET SDK, and runs the portable test project through the Windows host under Wine. Another job builds and tests `net10.0` on Ubuntu, including the managed macOS audio, calendar, screenshot, registration, and desktop-control behavior tests. Its `TZ=Europe/Berlin` setting also exercises the calendar daylight-saving regression. No additional runner labels or host devices are required.
Before behavior changes:
Ubuntu does not compile the Swift helpers or execute Apple frameworks. Tests requiring the native macOS environment report an explicit skip through `MacOsFact`; they must also be run on a supported Mac with `dotnet test MeetingAssistant.Tests/MeetingAssistant.Tests.csproj -f net10.0 -c Release -p:EnableWindowsTargeting=true`. That build compiles and signs the helpers, and the suite checks packaging, helper self-tests, and native image cropping. Real microphone/system-audio capture and privacy permissions still require the operational checks described above.
1. Read `AGENTS.md`, this README, `openspec/config.yaml`, and relevant specs under `openspec/specs`.
2. Add or update the OpenSpec requirement/scenario for the behavior.
3. Add a failing behavior test through the public surface.
4. Implement the smallest passing change.
5. Run the narrowest useful tests, then broader tests when the blast radius justifies it.
6. Run `openspec validate <change-id> --strict` for active spec changes.
[Docker-OSX](https://github.com/sickcodes/Docker-OSX) runs a macOS VM rather than providing a Wine-style compatibility layer. Its launcher supports software emulation with `KVM=accel=tcg`, so KVM is not an absolute requirement. A supported .NET 10 guest needs macOS 14 or later plus the Swift build tools. The documented `auto` build downloads a preinstalled guest disk through `IMAGE_URL`; its documented ready-made tags and disk downloads were unavailable when checked on 2026-10-03. No verified native guest bootstrap is owned by this repository. CI validates source; it does not publish or deploy the workstation application.
Documentation-only README maintenance does not need a new OpenSpec change.
The separate manual [native Recovery diagnostic](docs/macos-native-diagnostic.md) probes macOS startup and disk readiness through an unprivileged TCG guest on the existing Ubuntu Docker runner. It neither installs macOS nor runs application tests; its result is a prerequisite for a future native test job, not verification of macOS CI support.
## Operational Notes
## Operations And Limitations
- Treat the app as live user work. Check `/recording/status` before restarting, killing processes, deleting runtime files, or running scripts that might take over port `5090`.
- Abort is destructive for the active run: it removes that meeting's note, transcript, assistant context, summary if present, and linked screenshot attachments, and it skips summary generation.
- Too-short or empty normal stops can also delete generated artifacts instead of producing summaries.
- Summary and screenshot OCR retry links use `Api:PublicBaseUrl`, defaulting to `http://localhost:5090`.
- The workflow reload endpoint reloads configuration for future workflow reads, but a meeting run that already captured options may continue with those captured options.
- Public hostnames and homelab ingress are intentionally out of scope for this repo.
- Treat recording, transcription drain, speaker finalization, OCR, and summarization as live user work. Never restart, kill, or clean runtime files until `/recording/status` is idle unless interruption is explicitly intended.
- Summary and screenshot retry links use `Api:PublicBaseUrl`, which defaults to `http://localhost:5090`.
- Workstation startup is external to this repository. On `NA-EXC765X84` as of 2026-08-11, the installed Windows Startup shortcut invokes the restart helper at sign-in, so it republishes and replaces the process; it differs from the start-only shortcut tracked in `Manuel/snippets`, and the reason for that local deviation is not documented.
- No public hostname, homelab ingress, always-on deployment, or remote-service availability contract is owned here.
## More Documentation
- `docs/meeting-assistant-configuration.md`: full configuration reference.
- `docs/meeting-workflow-engine.md`: workflow rules engine reference.
- `docs/meeting-assistant-configuration.md`: complete configuration and backend behavior.
- `docs/meeting-workflow-engine.md`: workflow triggers, conditions, templates, steps, and editor behavior.
- `openspec/specs`: accepted behavioral requirements.
- `openspec/changes/archive`: archived change proposals and designs behind the accepted specs.
- `openspec/changes`: active change specs and designs, including implemented work not yet archived.
- `openspec/changes/archive`: historical proposals and decisions behind accepted behavior.
@@ -0,0 +1,44 @@
# Native macOS guest on existing Docker infrastructure — 2026-10-03
Result: real macOS Recovery booted under software emulation, but the bounded experiment did not reach an installed guest or execute the native tests. This is not a passing CI result or a verified workflow recipe.
## Source and execution boundary
- Dockur source: [`16a5b470cdd601bae8b05b02d748d7edfb36c12e`](https://github.com/dockur/macos/tree/16a5b470cdd601bae8b05b02d748d7edfb36c12e).
- Local environment: existing ARM64 Docker Desktop, linux/amd64 container translation, x86_64 QEMU guest with TCG. This does not establish the runtime or resources of the upstream Ubuntu-x64 runner.
- Guest settings: `VERSION=14`, `MANUAL=N`, `KVM=N`, `NETWORK=slirp`, 4 GiB guest RAM, two vCPUs, 64 GiB sparse guest disk. Container memory was limited to 6 GiB.
- Container inspection: `Privileged=false`, `CapAdd=null`, `Devices=[]`, ordinary bridge network. Diagnostic ports were published only on localhost. No runner registration, host configuration, device passthrough or new secret was requested.
- Candidate archive: clean application/test source at `1164c26846686c1912fd1816cd06de80352e9504`, SHA-256 `30796a27c75792ab87d23dd4c4db991b426d9c55a0e7dc3d4dc45c41056846e1`. It was not executed in the guest.
## Observed boundary failure
The container built and downloaded Apple's Recovery image. Recovery mounted its installation-state share, found Sonoma's `startosinstall`, and passed the upstream unattended preflight. Native `sw_vers` and `uname` output identified:
```text
ProductVersion: 14.6.1
BuildVersion: 23G93
Darwin Kernel Version 23.6.0 ... RELEASE_X86_64 x86_64
```
The first target-selection attempt found no writable disk. Both virtio and SATA trials subsequently exposed `diskutil` failure: it could not use the DiskManagement framework. Its diagnostic listed unavailable DiskArbitration/single-user mode as a possible cause; that cause was not independently established. SATA also exposed an independent missing `io2` QEMU object, corrected through an ordinary QEMU argument without host capabilities.
The pinned Recovery patch replaces an early `rc.cdrom.sh` block, before normal daemon startup, with a blocking `exec launch.sh`. One final causal trial changed only this padded bootstrap to launch the installer in the background. Its old/new bootstrap SHA-256 values were `73fd171ea4c889f9946f04d6866047fedd417a69a241b3e84db469c13975d897` and `c5fc86f95e9c65dd7dc59536911c81c96928d4f9541b63121c22cad5690f3582`. The launch script also waited for actual `diskutil list physical` success before the unchanged writable-disk/size guards and captured diagnostics after mounting the share.
That trial still recorded 12 completed native `diskutil` failures and an incomplete thirteenth attempt:
```text
Unable to run because unable to use the DiskManagement framework.
Common reasons include, but are not limited to, the DiskArbitration
framework being unavailable due to being booted in single-user mode.
[exit 1]
```
There was no successful disk enumeration, selected installation target, target erase or `startosinstall` invocation. The background change did not establish a working bootstrap; the remaining daemon/framework cause is unresolved. This result does not prove that macOS under TCG on Ubuntu is impossible.
## End state and evidence
Final container: `c4d5b83f5199063df75d43236d610ef9b71c3b39c14b80a649b0e904d13e87c8`. It started at `10:46:24.987543461 UTC`; the authorized final boot boundary was start plus 16 minutes. It was stopped at `11:02:33.810090465 UTC`: `Running=false`, `ExitCode=143`, `OOMKilled=false`. No experiment container remained running.
Raw evidence is retained locally under `/private/tmp/meeting-assistant-macos-guest-proof-20261003/runs/c4d5b83f5199/`: `guest-disk-platform.log`, `physical-disks-ready.log`, `recovery-install.log`, their capture metadata, and `end-container-inspect.json`/`end-state.json`. Metadata identifies the full container ID, actual start time, capture time and log SHA-256; earlier flat logs are historical and were not treated as success in this run.
No guest SSH session, official .NET/Apple CLT bootstrap, Swift build or TRX result was obtained. The application's workflow, production runtime and tests were not changed by this experiment. Native Ubuntu CI remains open; local Mac qualification is recorded separately in [the CI evidence](macos-ubuntu-ci-completion.md).
@@ -0,0 +1,55 @@
# macOS PR validation on existing Ubuntu infrastructure — 2026-10-03
## Scope and identity
Source repository: `/Users/dh/Documents/DanielsVault/_ops/meeting-assistant`.
Delivery branch: `codex/macos-support` in `Daniel/meeting-assistant`; intended PR target: `Manuel/meeting-assistant:main`, PR #39.
Fixed change baseline: `bd35ebc4c81bedf80caabb82481169124775fb36` (already contains main `7b2bcd36310ae6434a4ee6eeb3a47b6d1d431df4`).
Implementation and qualification were isolated under `/private/tmp/meeting-assistant-macos-ci-20261003`; the original checkout was clean. Updating that original checkout after publication unexpectedly invoked its existing post-merge deployment hook and restarted the workstation service. The immediately checked recording endpoint reported idle (`state=0`, `isRecording=false`) and health reported OK. Future checkout updates disable Git hooks explicitly. This qualification did not intentionally start a new recording.
Review covers this CI/test/shutdown delta. Earlier macOS feature and main-sync evidence remains in the respective OpenSpec implementation receipts; this record is not a fresh review of all earlier branch changes.
The hashes below identify the substantive implementation published in `1164c26846686c1912fd1816cd06de80352e9504`. The evidence record and generated logs/TRX files are excluded to avoid self-referential hashes. Subsequent factual documentation corrections do not change the reviewed workflow, runtime or tests.
| Reviewed file | SHA-256 |
| --- | --- |
| `.gitea/workflows/pr-push-build-and-test.yaml` | `a6251c435c55967ca2babd58c136ba739ed7c18674e93fff96b477f7b7e1366e` |
| `MeetingAssistant.Tests/MacOsDesktopControlManifestTests.cs` | `1df35cf6071f3cdae7d06af8566d59bd2af6c8c5c8c6ca9e04c24197a391975e` |
| `MeetingAssistant.Tests/MacOsFactAttribute.cs` | `c756088c1efca9dfe7509a8597845d94e2076445a9f7a2ecec08cd41d333ba7e` |
| `MeetingAssistant.Tests/MacOsMeetingAudioSourceTests.cs` | `11c3e36bb31a8aa3b7542ff44367bb0e71ad68360e494fa219b77c0ced3e2456` |
| `MeetingAssistant.Tests/MacOsMeetingIntegrationTests.cs` | `51337c27d6126e8e37f4e3b268307830678b8365de51b0c9ce49557ac40160f4` |
| `MeetingAssistant.Tests/PyannoteDiarizationWarmupHostedServiceTests.cs` | `0d3494860ce044a2e73e7edba11017b910ca5c0207e2a79da8ff08bde2ff0dfe` |
| `MeetingAssistant/Transcription/PyannoteDiarizationWarmupHostedService.cs` | `bcabf6007c9c942845336c1dd958852a8d6dcc383225b4bb1ea97903ef2848e8` |
| `README.md` | `79e834ae2953ba690defd4ab10a24e08ae2ce2186ab689afc0ab607e579498e9` |
| `openspec/specs/meeting-transcription/spec.md` | `c77edffc0b6ea374e250df758a9be76cbc3ee1146422fc374901c02f216676f6` |
## Requirements and observed evidence
- Existing Ubuntu infrastructure: both workflow jobs select only `ubuntu-latest`. The native `macos:host` runner requirement is removed. No KVM device, privileged container, new secret, guest image, or runner registration is required by these jobs. YAML parsing, each job's shell syntax (`bash -n`), and `git diff --check` passed.
- Executable macOS managed coverage: the complete portable job's restore/build/test shell steps ran in an Ubuntu 24.04 amd64 `.NET 10` container with `TZ=Europe/Berlin`: **572 passed, 5 explicitly skipped, 0 failed, 577 total**. The skipped tests require a macOS environment and are not counted as passed. Three formerly guarded source/registration/approval tests now actually execute on Ubuntu. A separate `linux-x64` RID build passed the source identity test, proving lookup through the extra output-directory level. Raw Linux log: `/private/tmp/meeting-assistant-portable-final.log`; TRX: `/private/tmp/meeting-assistant-macos-ci-20261003/artifacts/tests/portable.trx`.
- Native macOS coverage: the same test/runtime source contents built and signed the Swift helpers on this Mac and passed **577/577**, no skips. Raw receipt: `/private/tmp/meeting-assistant-native-ci-20261003/artifacts/tests/macos-warmup-final-green.trx`. This is local qualification; Ubuntu does not compile or execute the Swift helpers. Earlier real capture → Azure transcription → summary evidence is recorded in `openspec/changes/add-macos-desktop-controls/main-sync-completion.md`; no new live recording was started here.
- Accurate Windows verification: `win-x64` RID alone does not select the application's Windows desktop TFM. CI now explicitly builds `net10.0-windows10.0.19041.0` through the existing Wine/Windows .NET SDK, then runs the portable tests under that Windows host. A fresh Windows DLL and fresh TRX are mandatory; a zero exit code without executed work is insufficient. Full Wine validation on the current candidate remains pending on the actual Ubuntu-x64 runner. The local Apple-Silicon Docker attempt hit `rosetta error: invalid gdt selector index 5`; it is not a passing Windows result. Raw attempt: `/private/tmp/meeting-assistant-wine-ci.log`.
- Safe warmup shutdown: two real macOS full-suite runs exposed `ObjectDisposedException` from `PyannoteDiarizationWarmupHostedService.StopAsync` at `CancelAsync`, in different endpoint fixtures. A public concurrent-stop reproducer with temporarily increased scheduling overlap produced the exact linked-source exception. Under identical instrumentation, changing only cancellation-source ownership to `Interlocked.Exchange` passed; after instrumentation removal all four lifecycle tests and both final suites passed. Raw red/green receipts: `warmup-overlap-linked-red.trx`, `warmup-overlap-instrumented-green.trx`, and `warmup-overlap-final-green.trx` under the native checkout's `artifacts/tests`. The permanent test coordinates real stop callers with bounded waits; a race test is scheduling-sensitive on the original code, so the instrumented counterexample and original full-suite failure remain part of the causal evidence. The existing single-stop test still proves pending model work observes cancellation.
- Test isolation: the macOS audio endpoint fixture supplies its own speech pipeline and no longer starts the unrelated external pyannote model warmup. Its actual mixing, recording-stop, and native-process-termination assertions remain. This fixture isolation is not presented as the production race fix.
- OpenSpec: CI/build/test plumbing needs no new change under `AGENTS.md`. The runtime shutdown bug is specified as an addendum to the original accepted pyannote warmup requirement in `openspec/specs/meeting-transcription/spec.md`; strict validation passed (`openspec validate meeting-transcription --type spec --strict`). No active change was archived.
## Standards coverage and sequential review
Sources: repository `AGENTS.md`, applicable specs, and `code-review` criteria.
DRY owns duplicated platform/test/CI knowledge; SOLID owns lifecycle resource ownership, dependency boundaries and test isolation; KISS owns bounded concurrency test clarity, command naming, accurate documentation, and remaining manual standards. Tests, source builds, OpenSpec validation, YAML/shell parsing and whitespace checks provide automated coverage.
The passes ran sequentially in fresh, independent reviewer contexts against the nine frozen hashes above. Each inspected the delta plus adjacent implementation and repository standards; none edited files or claimed remote CI success.
- DRY (`/root/review_dry`): clean; platform decisions are centralized in `MacOsFact`, and the two CI host contracts do not justify further indirection.
- SOLID (`/root/review_solid`): clean; atomic cancellation ownership, the public stop regression, and the exact fixture dependency removal preserve coherent responsibilities.
- KISS (`/root/review_kiss`): clean; bounded stop coordination, CI command names, platform boundaries and README instructions accurately describe the checks.
No review repair changed the frozen substantive identity. All required structural passes are complete; runtime evidence and remaining Windows/native limitations are recorded separately above.
The subsequent README/evidence corrections update only observed artifact availability, publication status, and the checkout-hook event. Structural review is `not-required` for this factual documentation delta; the workflow, runtime, tests and binding instructions are unchanged. A separate read-only factual check confirmed the revised `auto` description against Dockerfile.auto and checked the limits against the saved receipts. Whitespace and the final documentation diff were checked; existing test results remain applicable to the unchanged code.
## Delivery and remaining limits
State at publication: structural review complete; current-head remote verification pending. This is not a merge-readiness or main-merge receipt.
The user authorized preparing this branch for PR CI and merging into main after tests pass. Authenticated Gitea identity `Daniel` has write/admin on the fork and read-only access to `Manuel/meeting-assistant`; it cannot merge there. The earlier local main-sync commits and reviewed implementation were normally pushed to the existing fork branch as `1164c26846686c1912fd1816cd06de80352e9504`; PR #39's description was updated and read back against that exact head.
Upstream [run 4145](https://gitea.schweigert.cloud/Manuel/meeting-assistant/actions/runs/4145) was created for this head, with both Ubuntu jobs waiting and no runner assigned. Its live job view explicitly says `Need approval to run workflows for fork pull request.` The current account cannot approve upstream workflows. This is the existing fork-workflow approval gate, not a request for additional infrastructure. Main is not merged; remote Windows and portable checks have not executed for this head.
A native macOS guest/Swift CI run on Ubuntu is not implemented. Docker-OSX supports software emulation, but the documented installed-guest downloads and `auto` tags were unavailable when checked. An [isolated unprivileged Dockur/TCG experiment](macos-tcg-guest-feasibility.md) booted an actual macOS 14.6.1 x86_64 Recovery environment, but `diskutil` could not use the DiskManagement framework; the actual cause remains unresolved. The bounded trial ended without an installed guest or native tests, and its container was stopped. No unverified guest bootstrap was made a required PR check. Passing the two current Ubuntu jobs would still leave the user's requested native macOS coverage in Ubuntu CI unmet; the local Mac result does not close that requirement.

Some files were not shown because too many files have changed in this diff Show More